SlideShare ist ein Scribd-Unternehmen logo
1 von 14
FAQ on SnapDIff
March, 2016
ashwinwriter@gmail.com
What is SnapDiff?
SnapDiff is NetApp's proprietary Crawler or Indexer. You can simply call it a 'fast indexing
engine'. It basically takes two snapshots called base snapshot and diff snapshot as input and
returns a list of all files that have been added, deleted, modified and renamed between the
two snapshots.
What is SnapDiff API?
SnapDiff’sApplication Programming Interface (API) is the way third-party Backup Software
vendors can integrate their Backup Solutions to make use of this software.
For example:
1. IBM - TSM
2. CommVault [IntelliSnap]
3. Bacula
4. Symantec Netbackup
What is the need of SnapDIff ?
Traditionally, Backup of NAS Volumes are done either over the network (NFS/CIFS) via
Windows mount or NFS mount or by using Industry standard NDMP protocol.
In case of traditional file system backups – A file system tree walk is performed to identify the
files/directories that have changed since the last full back up using traditional file system crawler:
A file system crawler is a piece of software that walks down a file system tree and gathers
information about each subdirectory and file within that tree. This is both time and resource [CPU
& MEM] consuming process for host machine.
Even though NDMP is an incremental [either inode file walk or tree walk] snapshot based
backups it can still be painful depending upon the number of small files and depth of the directory
structure [in Millions].
Slow : During NDMP Incremental backup, the information about file attributes have to be sent to
the NDMP Media Agent [DMA]. This is then compared with the Index that the MA had made
during the full backup. The media agent then generates a list of files that need to be backed up
for the incremental backup which is single stream and can be time consuming depending upon
the number of small files changes for that Incremental iteration. This is only an indexing part,
data also needs to be copied out of the snapshots to external storage.
Dis-advantage of NDMP backups are:
a. For TERA & Peta bytes of storage - You can only do '9' Incremental for traditional NDMP
Backups and after that you need to once again face the nightmare of long backup windows for
Full NDMP dumps, sometimes running into weeks if not days.
Clustered Data ONTAP 8.3 onwards supports 32 levels of NDMP dump Backups for NetApp.
Level 0 = is a Full Backup.
Level 1 through Level 31 are Incremental Backups.
However, after '31' Incremental NDMP backups, nightmare of running a Full backup will be back.
Whereas, SnapDIff is incremental forever.
b. NDMP Dump is always streamed out of the storage to secondary storage media - Either Disk
or Tape. In other words, even if you have manageable small number of files and changed
blocks, they still need to be copied out to external media.
SnapDiff - Is a fast indexer/crawler, which quickly identifies the file and directory differences
between two Snapshots (Base, n-1 & Incremental, n). It does it by quickly looking at the block
numbers of the inode file's indirect and data blocks, instead of shooting through the inode file
sequentially which could contains million files, wasting time and resources. If a block(s) number
has not changed, then it does not need to be crawled. As a result, SnapDiff can identify just the
data blocks that have changed and crawl only those data (files/dir).
With Intellisnap option - You can choose to keep the Primary Snaps on the NetApp itself, that
means the Snapshot creation job is only '2' second task. Rest of the time is spent in 'Snap
Differencing' for the files/dir inside the snapshot. You can also defer the SnapDiff for later time
and even do the SnapDiff on the Secondary NetApp. If that was not enough, NetApp now
allows CommVault to do Live browse - Which enables admins to quickly mount the snapshot
volume to access data in order to restore it back.
Is SnapDIff API free to implement or is it licensed?
Only approved [Backup] partners are given accompanying documentation and guidelines
on using SnapDiff APIs. Please see this post for more information.
https://community.netapp.com/t5/Software-Development-Kit-SDK-and-API-
Discussions/snapdiff-API-documentation/td-p/42561
Can NDMP DUMP Backup use 'SnapDIff' API?
No. NDMP DUMP Backups are done via NDMP native UNIX 'dump' engine which is completely
different from the way the way SnapDIff engine works.
Is SnapDiff applicable to both NAS & SAN protocols?
No. SnapDiff is only applicable to NAS [CIFS (SMB)/NFS] environment.
What do you mean by 'maxdiff' or 'SnapDiffMax?
It is the default number [256] of file differences requested in each query.
What is the purpose of 'SnapDiffMax' additional setting on the Media Agent?
SnapDiffMax additional registry setting allows NAS iDA to increase or decrease the number of
files to be exported. However, before making any changes, it is recommended to contact
NetApp to verify that the file server configurations can support changes in the values of these
additional settings.
How is SnapDIff performance measured?
SnapDiff performance is not measured in throughput [MB/GB/sec], rather in terms of 'number
of files processed per hour', in more technical language it is called 'Diffs/hour'. Please note,
SnapDIff is not a backup tool, it is designed to speed up backup.
What factors governs the performance of the 'SnapDIff' engine?
Performance is not affected by the size of the NetApp volumes in terms of terabytes or
petabytes. Nor do the sizes of the files catalogued have any effect on performance. However,
SnapDIff performance can get degraded if the source volume contains many millions of files
spread over a “deep” (many levels) directory structure.
Which Ontap version should the filer be for better SnapDiff performance?
Starting with Data ONTAP 8.2 onwards – regardless of FAS physical architecture (Clustered
Data ONTAP or 7-mode) – SnapDiff’s performance has been greatly improved.
Can increasing the maxdiff registry value from default '256' items to '1024' or more affect
filer's performance?
Please note - SnapDiff is a low priority process on a NetApp storage system. Processes that
support file sharing, such as NFS and/or CIFS, as well as replication functions like SnapVault
and SnapMirror all run at a higher priority. If load on a storage system becomes a problem, there
are several ways of minimizing the load, such as limiting the number of volumes selected for
SnapDiff from a single sub client.
What are the requirements for using SnapDiff API with NetApp filer?
SnapDiff API requires: XML over HTTP[s] & Vol Lang to be 'UTF-8'.
The HTTP[S] connection is used only to transmit administrative data between the backup
software and the NetApp filer. The administrative session data includes information such as
filer credentials, snapshot information, and file names and attributes that are generated by the
snapshot differencing process. The HTTP[S] connection is not used to transmit normal file data
that is accessed on the filer by the client through file sharing protocols.
Ensure:
1. Http/https is enabled
2. Vol Language on the Filer is set to - UTF-8 [en_US.UTF-8]
3. create_ucode and convert_ucode flag is turned on - Especially if SnapDiff ZAPI is run on
volumes on with nfs-only compatible names.
4. Account used for running the SnapDIff on the Filer has this role capability - login-http-
admin,api-*
How do Backup applications interact with SnapDiff engine?
Backup applications makes use of following SnapDiff Zephyr API [ZAPI], in short XML messages
transmitted over HTTP(S)
snapdiff-iter-start
snapdiff-iter-next
snapdiff-iter-end
Is there a limit on the concurrent SnapDiff session to NetApp controller?
DataONTAP limits the number of concurrent SnapDiff sessions to 16 per controller. When
planning for NAS backups that perform indexing it is recommended to distribute the schedules so
that no more than 16 jobs are running concurrently on a single controller.
SnapDiff starting point can be traced with following lines in CVNasFileScan.log on the
Media Agent server:
3996 f0c 01/29 15:35:19 173477 ManageONTAP::GetVolLanguage Get volume language.
FileServer:[ONTAP7MODE] Volume:[CIFS]
3996 f0c 01/29 15:35:20 173477 SnapDiffStart FileServer:[ONTAP7MODE] Volume:[CIFS]
SnapOne:[SP_2_173476_31_1454081328] SnapTwo:[SP_2_173477_32_1454081692]
Format:[UTF8] Items In Each Iteration [256]
3996 f0c 01/29 15:35:24 173477 SnapDiffStart File-access-protocol support:[true] version
check:[true]
3996 f0c 01/29 15:35:24 173477 SnapDiffStart atime support:[true] version check:[true]
3996 f0c 01/29 15:35:25 173477 SnapDiffStart End-of-diff support: [true]
3996 f0c 01/29 15:35:26 173477 SnapDiffStart Successfully kicked off SnapDiff.
SnapDiff API module scans the Base & Diff snapshot and identifies changed files using:
1. Inode [No]
2. Change-type attribute: [Values could be - file_creation | inode_modification
| file_deletion ]
Steps to capture 'SnapDIff' process from CommVault side.
1. Open Process Manager on the Media Agent Server responsible for IntelliSnap NAS iDA
backups.
2. Go to Logging Tab | Select Module 'CVNasFileScan'
3. Increase the DbgLevel | LogFileSize | LogFileMaxVer to higher number as desired
In this example:
1. CIFS share is setup on NetApp NAS Volume [/vol/CIFS].
2. Three Files A.txt, B.txt & C.txt is copied to share.
Content of each file are:
A.txt - 1,2
B.txt - 1,2,3,4
c.txt - 1, 2, 3, 4, 5, 6
Steps to capture SnapDIff process:
1. Run 'Full' Backup of CIFS share /vo/CIFS using IntelliSnap - Example Job ID:173473
2. Open CVNasFileScan using GxTail on the Media Agent and trace the log for Job ID:173473
CVNasFIlescan log shows '3' Files [A, B, C] being identified for Backup with change-
type [file_creation]
SnapDiff Indexing via inode identification in snapshot:
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Changed file: Original Name
[/vol/CIFS/.snapshot/SP_2_173473_29_1454079783/Download/A.txt]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - inode: [99]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - change-type: [file_creation]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ftype: [1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - links: [1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - fattr: [0x1ff]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - [0]=3 bits:
UserId,groupId,sticky 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() -
[777]=Owner,Group,Other rwx bits
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - owner: [0]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - group: [0]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - size: [2]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - crtime: [1454079489]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - mtime: [1454079508]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ctime: [1454079508]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - atime: [1454079785]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ntacl-inode: [-1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - streamdir-inode: [-1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - dos-bits: [1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - nfsacl-inode: [-1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Modified file name
[/vol/CIFS/Download/A.txt]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - File Name sent to Indexing
[/vol/CIFS/Download/A.txt]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Changed file: Original Name
[/vol/CIFS/.snapshot/SP_2_173473_29_1454079783/Download/B.txt]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - inode: [100]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - change-type: [file_creation]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ftype: [1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - links: [1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - fattr: [0x1ff]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - [0]=3 bits:
UserId,groupId,sticky 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() -
[777]=Owner,Group,Other rwx bits
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - owner: [0]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - group: [0]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - size: [4]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - crtime: [1454079494]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - mtime: [1454079513]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ctime: [1454079513]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - atime: [1454079785]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ntacl-inode: [-1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - streamdir-inode: [-1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - dos-bits: [1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - nfsacl-inode: [-1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Modified file name
[/vol/CIFS/Download/B.txt]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - File Name sent to Indexing
[/vol/CIFS/Download/B.txt]
4968 1be4 01/29 15:03:46 173473 IDXCONNECT sendFileInfo: File
'/vol/CIFS/Download/B.txt', restart 0, offset 0 size 4 flags 0
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Changed file: Original Name
[/vol/CIFS/.snapshot/SP_2_173473_29_1454079783/Download/C.txt]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - inode: [103]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - change-type: [file_creation]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ftype: [1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - links: [1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - fattr: [0x1ff]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - [0]=3 bits:
UserId,groupId,sticky 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() -
[777]=Owner,Group,Other rwx bits
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - owner: [0]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - group: [0]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - size: [6]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - crtime: [1454079502]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - mtime: [1454079523]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ctime: [1454079523]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - atime: [1454079785]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ntacl-inode: [-1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - streamdir-inode: [-1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - dos-bits: [1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - nfsacl-inode: [-1]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Modified file name
[/vol/CIFS/Download/C.txt]
4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - File Name sent to Indexing
[/vol/CIFS/Download/C.txt]
As this is the first 'Full' IntelliSnap Job, both Full & Incremental snaps are same
FullSnap: [SP_2_173473_29_1454079783]
IncrSnap: [SP_2_173473_29_1454079783]
1. This time we went back to the CIFS shared directory and made following changes:
2. We left the text File 'A' & 'B' untouched.
3. We modified text File 'C' and added two additional integers - 7 & 8 [Previously it
had :123456]
We ran another SnapDIff Incremental Backup. As we can see below, SnapDiff quickly identified
the changed file in seconds, here in this example - we expect to see 'C' text file only, and
there it is. We do not see other two text files in the SnapDIff changed files as expected.
CVNasFIlescan log shows 'C.txt' being indentified as changed file with change-
type [inode_modification]
3560 1984 01/29 15:11:49 173475 VolFetchSnapDiff() - Started SnapDiff fetch for volume [CIFS]
3560 1984 01/29 15:11:49 173475 VolFetchSnapDiff() - Will Fetch the chaged files in queue
element [0]
3560 1764 01/29 15:11:51 173475 IndexSnapShot_SnapDiff() - Indexing files at queue element
[0] <--This indicates any new files, as there were no new files, the value returned here is 0.
3560 1984 01/29 15:11:59 173475 VolFetchSnapDiff() - Fetched the chaged files in queue
element [0]
3560 1984 01/29 15:11:59 173475 VolFetchSnapDiff() - Will Fetch the chaged files in queue
element [1] <---This indicates, we have one changed file.
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - Changed file: Original Name
[/vol/CIFS/.snapshot/SP_2_173475_30_1454080264/Download/C.txt]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - inode: [103]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - change-type:
[inode_modification]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - ftype: [1]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - links: [1]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - fattr: [0x1ff]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - [0]=3 bits:
UserId,groupId,sticky 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() -
[777]=Owner,Group,Other rwx bits
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - owner: [0]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - group: [0]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - size: [8]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - crtime: [1454079502]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - mtime: [1454080236]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - ctime: [1454080236]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - atime: [1454080236]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - ntacl-inode: [-1]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - streamdir-inode: [-1]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - dos-bits: [1]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - nfsacl-inode: [-1]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - Modified file name
[/vol/CIFS/Download/C.txt]
3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - File Name sent to Indexing
[/vol/CIFS/Download/C.txt]
Incremental SnapDiff:
SnapDiffStart FileServer:[ONTAP7MODE] Volume:[CIFS]
SnapOne:[SP_2_173473_29_1454079783] SnapTwo:[SP_2_173475_30_1454080264]
FullSnap: [SP_2_173473_29_1454079783]
IncrSnap: [SP_2_173475_30_1454080264]
1. This time we went back to the CIFS shared directory and made following changes:
2. Just touched the '3' Files | Opened them and closed them.
3. We basically 'changed' the access time of these Files.
We ran another SnapDiff Incremental Backup. As we can see below, SnapDIff did not see any
changed file, as we only touched the three files:
SnapDiffStart FileServer:[ONTAP7MODE] Volume:[CIFS]
SnapOne:[SP_2_173475_30_1454080264] SnapTwo:[SP_2_173476_31_1454081328]
FullSnap: [SP_2_173473_29_1454079783]
IncrSnap: [SP_2_173476_31_1454081328]
1. This time we went back to the CIFS shared directory:
2. We left the text File 'A' & 'B' untouched.
3. We delete 'C.txt' text file.
We ran another SnapDIff Incremental Backup. As we can see below, SnapDiff quickly identified the
changed file in seconds, here in this example - we expect to see 'C' text file only, and there it is. We
do not see other two text files in the SnapDIff changed files as expected.
CVNasFIlescan log shows 'C.txt' being indentified as changed file with change-type [file_deletion]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - Changed file: Original Name
[/vol/CIFS/.snapshot/SP_2_173476_31_1454081328/Download/C.txt] 3996 f0c 01/29 15:35:27
173477 IndexSnapShot_SnapDiff() - inode: [103]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - change-type: [file_deletion]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - ftype: [1]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - links: [1]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - fattr: [0x1ff]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - [0]=3 bits: UserId,groupId,sticky
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - [777]=Owner,Group,Other rwx bits
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - owner: [0]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - group: [0]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - size: [8]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - crtime: [1454079502]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - mtime: [1454080236]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - ctime: [1454080236]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - atime: [1454081329]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - ntacl-inode: [-1]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - streamdir-inode: [-1]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - dos-bits: [1]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - nfsacl-inode: [-1]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - Modified file name
[/vol/CIFS/Download/C.txt]
3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - Deleted File Name sent to
Indexing [/vol/CIFS/Download/C.txt]
Incremental SnapDiff:
SnapDiffStart FileServer:[ONTAP7MODE] Volume:[CIFS]
SnapOne:[SP_2_173476_31_1454081328] SnapTwo:[SP_2_173477_32_1454081692]
FullSnap: [SP_2_173473_29_1454079783]
IncrSnap: [SP_2_173477_32_1454081692]
Following example shows First Full [Base snapshot] and two Incremental backup
jobs: SnapDiffStart FileServer:[CMODE83.TEST.COM]
Volume:[/CMODE83_SVM_08/CMODE83_SVM_08_CIFS]
Full = SnapOne:[] SnapTwo:[SP_2_173538_83_1454447246] Format:[UTF8] Items In Each
Iteration [256]
Incremental = SnapOne:[SP_2_173538_83_1454447246]
SnapTwo:[SP_2_173539_85_1454448660] Format:[UTF8] Items In Each Iteration [256]
Incremental = SnapOne:[SP_2_173539_85_1454448660]
SnapTwo:[SP_2_173548_86_1454528619] Format:[UTF8] Items In Each Iteration [256]
Differencing between latest and the previous snapshot:
FullSnap: [SP_2_173538_83_1454447246] IncrSnap: [SP_2_173538_83_1454447246]
FullSnap: [SP_2_173538_83_1454447246] IncrSnap: [SP_2_173539_85_1454448660]
FullSnap: [SP_2_173538_83_1454447246] IncrSnap: [SP_2_173548_86_1454528619]
SnapDiff API XML interaction can be traced via wireshark on the Media Agent, following
screenshots shows SnapDiff APIs in action:
snapdiff-iter-start
snapdiff-iter-next
snapdiff-iter-end
SnapDiff API XML interaction captured on the NetApp filer side:
snapdiff-iter-start
snapdiff-iter-next
snapdiff-iter-next
snapdiff-iter-end
Troubleshooting SnapDiff failures:
SnapDiff failures are not easy to troubleshoot especially considering the fact that CommVault
makes use of the standard APIs, and all the functionality is basically carried out by SnapDIff on
the NetApp filer itself. However, we can certainly do some investigation from our side in order
to make sure the basic requirements for SnapDiff operations are met [see the FAQ section].
From a broad perspective, there are three components involved that can cause 'SnapDIff'
failures at various API stages such as snapdiff-iter-start, snapdiff-iter-next,snapdiff-iter-end?
Three components are:
1.SnapDiff engine
2.SnapDiff ZAPI
3.Underlying TCP/IP network that enables XML Data exchange over HTTP/HTTPS.
SnapDIff engine: Unless there is identified bug, SnapDiff should norrmally work fine, there are
some bugs identified for SnapDiff are mentioned towards the end of the document.
SnapDIff API: Like CommVault there are other Backup vendors who also uses the standard
SnapDIff APIs just like us, and unless there is some configuration issue, it should be fine.
Network: This is probably the main gray area which causes 90% of the SnapDIff errors.
According to NetApp, some of the main causes are:
1. Misconfigured ISL (interswitch link)
2. Jumbo frames misconfigured (set on some points but not all points in the network path)
Verify the vendor recommendations for MTU and tcp-adjust-mss settings
3. Host side NIC teaming (Such as HP network teaming) misconfigured
Audit logs on the NetApp filer side are quite handy for monitoring ZAPI calls:
The Audit logs are located in ‘log’ directoryfilernameETC$underlog)and ETCare$ share ( named ‘auditlog’,
‘auditlog.0’, ‘auditlog.1’,the etcageof. theThelog; theindex numbe bigger the number, the older is the log.
filer>options auditlog.enable on
Some APIs are considered read-only as these calls are only to obtain information from the filer.
By default, APIs considered read-only are not logged to the auditlog.
Logging can be manually enabled using the command:
filer>options auditlog.readonly_api.enable on
A packet trace between the controller and application host server may help diagnose the
network level issue, please see the following KB articles from NetApp on how to run the packet
traces:
Please note - SnapDiff API requires converting information retrieved from Filer in to XML and
‘UTF-8’ is the default character encoding for XML, and therefore, it is important that the 'Volume'
against which SnapDiff is run is set to en_US.UTF-8 encoding at the time of Volume creation.
**NetApp does not recommend changing the volume language after the 'files' have been
copied/written**
If you do turn the switches create_ucode and convert_ucode on, later, please ensure:
Enable the create_ucode and convert_ucode flag on the volume and do the following:
1) Create a Snapshot copy
2) On windows: Start | Run and type: 'attrib /s /d' command in the volume which is mounted in
Windows. This will convert all the names to unicode in the active file system.
3) Disable i2p and then re-enable i2p.
Recommended KB Articles from NetApp:
How to collect a network trace with pktt in Data ONTAP 7-Mode
https://kb.netapp.com/support/index?page=content&id=1010155&actp=LIST
How to capture packet traces on Data ONTAP 8 Cluster-Mode systems
https://kb.netapp.com/support/index?page=content&id=1011204
NetApp KB articles related to SnapDiff:
API error: Parsing error in results: Extra content at the end of the
document https://kb.netapp.com/support/index?page=content&id=2018206
NetApp BUGs:
http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=533349
Degraded Snapdiff ZAPI performance when running on volume with create_ucode and
convert_ucode flag off
http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=533349
Snapdiff ZAPI with cifs file-access-protocol fails with "character out of allowed range" xml parse
error for filenames with invalid/reserved unicode characters
http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=588627
ONTAP panic when API snapdiff or inodepath takes more than 6 secs to process a large link
count file.
http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=773781
Controller disruption while using snapdiff to run indexing jobs
ashwinwriter@gmail.com
March, 2016

Weitere ähnliche Inhalte

Was ist angesagt?

Building Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCCBuilding Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCC
Kernel TLV
 
Accelerating Ceph with RDMA and NVMe-oF
Accelerating Ceph with RDMA and NVMe-oFAccelerating Ceph with RDMA and NVMe-oF
Accelerating Ceph with RDMA and NVMe-oF
inside-BigData.com
 

Was ist angesagt? (20)

Lisa 2015-gluster fs-hands-on
Lisa 2015-gluster fs-hands-onLisa 2015-gluster fs-hands-on
Lisa 2015-gluster fs-hands-on
 
ページャ lessを使いこなす
ページャ lessを使いこなすページャ lessを使いこなす
ページャ lessを使いこなす
 
Linux Profiling at Netflix
Linux Profiling at NetflixLinux Profiling at Netflix
Linux Profiling at Netflix
 
Lisa 2015-gluster fs-introduction
Lisa 2015-gluster fs-introductionLisa 2015-gluster fs-introduction
Lisa 2015-gluster fs-introduction
 
USENIX LISA11 Tutorial: ZFS a
USENIX LISA11 Tutorial: ZFS a USENIX LISA11 Tutorial: ZFS a
USENIX LISA11 Tutorial: ZFS a
 
Understanding DPDK
Understanding DPDKUnderstanding DPDK
Understanding DPDK
 
Building Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCCBuilding Network Functions with eBPF & BCC
Building Network Functions with eBPF & BCC
 
Continguous Memory Allocator in the Linux Kernel
Continguous Memory Allocator in the Linux KernelContinguous Memory Allocator in the Linux Kernel
Continguous Memory Allocator in the Linux Kernel
 
From printk to QEMU: Xen/Linux Kernel debugging
From printk to QEMU: Xen/Linux Kernel debuggingFrom printk to QEMU: Xen/Linux Kernel debugging
From printk to QEMU: Xen/Linux Kernel debugging
 
The Linux Block Layer - Built for Fast Storage
The Linux Block Layer - Built for Fast StorageThe Linux Block Layer - Built for Fast Storage
The Linux Block Layer - Built for Fast Storage
 
Denoで動くReactフレームワークAleph.jsでポートフォリオサイトをリプレイスした話
Denoで動くReactフレームワークAleph.jsでポートフォリオサイトをリプレイスした話Denoで動くReactフレームワークAleph.jsでポートフォリオサイトをリプレイスした話
Denoで動くReactフレームワークAleph.jsでポートフォリオサイトをリプレイスした話
 
지금 핫한 Real-time In-memory Stream Processing 이야기
지금 핫한 Real-time In-memory Stream Processing 이야기지금 핫한 Real-time In-memory Stream Processing 이야기
지금 핫한 Real-time In-memory Stream Processing 이야기
 
JavaOne 2015 Java Mixed-Mode Flame Graphs
JavaOne 2015 Java Mixed-Mode Flame GraphsJavaOne 2015 Java Mixed-Mode Flame Graphs
JavaOne 2015 Java Mixed-Mode Flame Graphs
 
Linux kernel debugging
Linux kernel debuggingLinux kernel debugging
Linux kernel debugging
 
Greenplum and Kafka: Real-time Streaming to Greenplum - Greenplum Summit 2019
Greenplum and Kafka: Real-time Streaming to Greenplum - Greenplum Summit 2019Greenplum and Kafka: Real-time Streaming to Greenplum - Greenplum Summit 2019
Greenplum and Kafka: Real-time Streaming to Greenplum - Greenplum Summit 2019
 
Kernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at NetflixKernel Recipes 2017: Using Linux perf at Netflix
Kernel Recipes 2017: Using Linux perf at Netflix
 
Kernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime Ripard
Kernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime RipardKernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime Ripard
Kernel Recipes 2017 - An introduction to the Linux DRM subsystem - Maxime Ripard
 
Accelerating Ceph with RDMA and NVMe-oF
Accelerating Ceph with RDMA and NVMe-oFAccelerating Ceph with RDMA and NVMe-oF
Accelerating Ceph with RDMA and NVMe-oF
 
Qemu device prototyping
Qemu device prototypingQemu device prototyping
Qemu device prototyping
 
Scaling Apache Storm - Strata + Hadoop World 2014
Scaling Apache Storm - Strata + Hadoop World 2014Scaling Apache Storm - Strata + Hadoop World 2014
Scaling Apache Storm - Strata + Hadoop World 2014
 

Ähnlich wie SnapDiff

Snap protect se_presentation_v3.0
Snap protect se_presentation_v3.0Snap protect se_presentation_v3.0
Snap protect se_presentation_v3.0
Mikis Eminov
 
It04 roshan basnet
It04 roshan basnetIt04 roshan basnet
It04 roshan basnet
rosu555
 
Tier 2 net app baseline design standard revised nov 2011
Tier 2 net app baseline design standard   revised nov 2011Tier 2 net app baseline design standard   revised nov 2011
Tier 2 net app baseline design standard revised nov 2011
Accenture
 

Ähnlich wie SnapDiff (20)

SnapDiff performance issue
SnapDiff performance issueSnapDiff performance issue
SnapDiff performance issue
 
Snap protect se_presentation_v3.0
Snap protect se_presentation_v3.0Snap protect se_presentation_v3.0
Snap protect se_presentation_v3.0
 
Testing Delphix: easy data virtualization
Testing Delphix: easy data virtualizationTesting Delphix: easy data virtualization
Testing Delphix: easy data virtualization
 
Ubuntu Core 技术详解
Ubuntu Core 技术详解Ubuntu Core 技术详解
Ubuntu Core 技术详解
 
Adaptive backup & Recovery partner enablement
Adaptive backup & Recovery partner enablementAdaptive backup & Recovery partner enablement
Adaptive backup & Recovery partner enablement
 
AIX Advanced Administration Knowledge Share
AIX Advanced Administration Knowledge ShareAIX Advanced Administration Knowledge Share
AIX Advanced Administration Knowledge Share
 
OSSV [Open System SnapVault]
OSSV [Open System SnapVault]OSSV [Open System SnapVault]
OSSV [Open System SnapVault]
 
JP Morgan Remote to Core Implementation
JP Morgan Remote to Core ImplementationJP Morgan Remote to Core Implementation
JP Morgan Remote to Core Implementation
 
Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Server Deep Dives (DrupalCon Amsterdam)Linux Server Deep Dives (DrupalCon Amsterdam)
Linux Server Deep Dives (DrupalCon Amsterdam)
 
Backtrack Manual Part4
Backtrack Manual Part4Backtrack Manual Part4
Backtrack Manual Part4
 
NetApp against ransomware
NetApp against ransomwareNetApp against ransomware
NetApp against ransomware
 
It04 roshan basnet
It04 roshan basnetIt04 roshan basnet
It04 roshan basnet
 
TekTape Manual
TekTape ManualTekTape Manual
TekTape Manual
 
Practical virtual network functions with Snabb (8th SDN Workshop)
Practical virtual network functions with Snabb (8th SDN Workshop)Practical virtual network functions with Snabb (8th SDN Workshop)
Practical virtual network functions with Snabb (8th SDN Workshop)
 
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,GrafanaPrometheus - Intro, CNCF, TSDB,PromQL,Grafana
Prometheus - Intro, CNCF, TSDB,PromQL,Grafana
 
Nagios Conference 2012 - Dan Wittenberg - Case Study: Scaling Nagios Core at ...
Nagios Conference 2012 - Dan Wittenberg - Case Study: Scaling Nagios Core at ...Nagios Conference 2012 - Dan Wittenberg - Case Study: Scaling Nagios Core at ...
Nagios Conference 2012 - Dan Wittenberg - Case Study: Scaling Nagios Core at ...
 
Tier 2 net app baseline design standard revised nov 2011
Tier 2 net app baseline design standard   revised nov 2011Tier 2 net app baseline design standard   revised nov 2011
Tier 2 net app baseline design standard revised nov 2011
 
Nagios En
Nagios EnNagios En
Nagios En
 
NameNode Analytics - Querying HDFS Namespace in Real Time
NameNode Analytics - Querying HDFS Namespace in Real TimeNameNode Analytics - Querying HDFS Namespace in Real Time
NameNode Analytics - Querying HDFS Namespace in Real Time
 
Hadoop project design and a usecase
Hadoop project design and  a usecaseHadoop project design and  a usecase
Hadoop project design and a usecase
 

Mehr von Ashwin Pawar

Our 5 senses can only perceive representation of reality but not the actual r...
Our 5 senses can only perceive representation of reality but not the actual r...Our 5 senses can only perceive representation of reality but not the actual r...
Our 5 senses can only perceive representation of reality but not the actual r...
Ashwin Pawar
 

Mehr von Ashwin Pawar (20)

16TB Max file size.pdf
16TB Max file size.pdf16TB Max file size.pdf
16TB Max file size.pdf
 
Our 5 senses can only perceive representation of reality but not the actual r...
Our 5 senses can only perceive representation of reality but not the actual r...Our 5 senses can only perceive representation of reality but not the actual r...
Our 5 senses can only perceive representation of reality but not the actual r...
 
E=C+O
E=C+OE=C+O
E=C+O
 
Oracle database might have problems with stale NFSv3 locks upon restart
Oracle database might have problems with stale NFSv3 locks upon restartOracle database might have problems with stale NFSv3 locks upon restart
Oracle database might have problems with stale NFSv3 locks upon restart
 
Is it possible to upgrade or revert ontap versions on a Simulator
Is it possible to upgrade or revert ontap versions on a SimulatorIs it possible to upgrade or revert ontap versions on a Simulator
Is it possible to upgrade or revert ontap versions on a Simulator
 
Cannot split clone snapcenter 4.3
Cannot split clone snapcenter 4.3Cannot split clone snapcenter 4.3
Cannot split clone snapcenter 4.3
 
Network port administrative speed does not display correctly on NetApp storage
Network port administrative speed does not display correctly on NetApp storageNetwork port administrative speed does not display correctly on NetApp storage
Network port administrative speed does not display correctly on NetApp storage
 
How to connect to NetApp FILER micro-USB console port
How to connect to NetApp FILER micro-USB console portHow to connect to NetApp FILER micro-USB console port
How to connect to NetApp FILER micro-USB console port
 
NDMP backup models
NDMP backup modelsNDMP backup models
NDMP backup models
 
How to use Active IQ tool to access filer information
How to use Active IQ tool to access filer informationHow to use Active IQ tool to access filer information
How to use Active IQ tool to access filer information
 
San vs Nas fun series
San vs Nas fun seriesSan vs Nas fun series
San vs Nas fun series
 
Steps to identify ONTAP latency related issues
Steps to identify ONTAP latency related issuesSteps to identify ONTAP latency related issues
Steps to identify ONTAP latency related issues
 
SnapDiff process flow chart
SnapDiff process flow chartSnapDiff process flow chart
SnapDiff process flow chart
 
Volume level restore fails with error transient snapshot copy is not supported
Volume level restore fails with error transient snapshot copy is not supportedVolume level restore fails with error transient snapshot copy is not supported
Volume level restore fails with error transient snapshot copy is not supported
 
Disk reports predicted failure event
Disk reports predicted failure eventDisk reports predicted failure event
Disk reports predicted failure event
 
OCUM shows ONTAP cluster health degraded
OCUM shows ONTAP cluster health degradedOCUM shows ONTAP cluster health degraded
OCUM shows ONTAP cluster health degraded
 
NDMPCOPY lun from 7-mode NetApp to cDOT
NDMPCOPY lun from 7-mode NetApp to cDOTNDMPCOPY lun from 7-mode NetApp to cDOT
NDMPCOPY lun from 7-mode NetApp to cDOT
 
Latency in storage
Latency in storageLatency in storage
Latency in storage
 
NVRAM vs NVMEM
NVRAM vs NVMEMNVRAM vs NVMEM
NVRAM vs NVMEM
 
NAS vs SAN
NAS vs SANNAS vs SAN
NAS vs SAN
 

Kürzlich hochgeladen

Kürzlich hochgeladen (20)

Strategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a FresherStrategies for Landing an Oracle DBA Job as a Fresher
Strategies for Landing an Oracle DBA Job as a Fresher
 
Artificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : UncertaintyArtificial Intelligence Chap.5 : Uncertainty
Artificial Intelligence Chap.5 : Uncertainty
 
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
Apidays Singapore 2024 - Building Digital Trust in a Digital Economy by Veron...
 
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot TakeoffStrategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
Strategize a Smooth Tenant-to-tenant Migration and Copilot Takeoff
 
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
Emergent Methods: Multi-lingual narrative tracking in the news - real-time ex...
 
MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024MINDCTI Revenue Release Quarter One 2024
MINDCTI Revenue Release Quarter One 2024
 
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
Connector Corner: Accelerate revenue generation using UiPath API-centric busi...
 
DBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor PresentationDBX First Quarter 2024 Investor Presentation
DBX First Quarter 2024 Investor Presentation
 
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot ModelNavi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
Navi Mumbai Call Girls 🥰 8617370543 Service Offer VIP Hot Model
 
ICT role in 21st century education and its challenges
ICT role in 21st century education and its challengesICT role in 21st century education and its challenges
ICT role in 21st century education and its challenges
 
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law DevelopmentsTrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
TrustArc Webinar - Stay Ahead of US State Data Privacy Law Developments
 
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
Apidays New York 2024 - Accelerating FinTech Innovation by Vasa Krishnan, Fin...
 
Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024Manulife - Insurer Transformation Award 2024
Manulife - Insurer Transformation Award 2024
 
Exploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone ProcessorsExploring the Future Potential of AI-Enabled Smartphone Processors
Exploring the Future Potential of AI-Enabled Smartphone Processors
 
A Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source MilvusA Beginners Guide to Building a RAG App Using Open Source Milvus
A Beginners Guide to Building a RAG App Using Open Source Milvus
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
MS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectorsMS Copilot expands with MS Graph connectors
MS Copilot expands with MS Graph connectors
 
presentation ICT roal in 21st century education
presentation ICT roal in 21st century educationpresentation ICT roal in 21st century education
presentation ICT roal in 21st century education
 
Ransomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdfRansomware_Q4_2023. The report. [EN].pdf
Ransomware_Q4_2023. The report. [EN].pdf
 
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWEREMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
EMPOWERMENT TECHNOLOGY GRADE 11 QUARTER 2 REVIEWER
 

SnapDiff

  • 1. FAQ on SnapDIff March, 2016 ashwinwriter@gmail.com What is SnapDiff? SnapDiff is NetApp's proprietary Crawler or Indexer. You can simply call it a 'fast indexing engine'. It basically takes two snapshots called base snapshot and diff snapshot as input and returns a list of all files that have been added, deleted, modified and renamed between the two snapshots. What is SnapDiff API? SnapDiff’sApplication Programming Interface (API) is the way third-party Backup Software vendors can integrate their Backup Solutions to make use of this software. For example: 1. IBM - TSM 2. CommVault [IntelliSnap] 3. Bacula 4. Symantec Netbackup What is the need of SnapDIff ? Traditionally, Backup of NAS Volumes are done either over the network (NFS/CIFS) via Windows mount or NFS mount or by using Industry standard NDMP protocol. In case of traditional file system backups – A file system tree walk is performed to identify the files/directories that have changed since the last full back up using traditional file system crawler: A file system crawler is a piece of software that walks down a file system tree and gathers information about each subdirectory and file within that tree. This is both time and resource [CPU & MEM] consuming process for host machine. Even though NDMP is an incremental [either inode file walk or tree walk] snapshot based backups it can still be painful depending upon the number of small files and depth of the directory structure [in Millions]. Slow : During NDMP Incremental backup, the information about file attributes have to be sent to the NDMP Media Agent [DMA]. This is then compared with the Index that the MA had made during the full backup. The media agent then generates a list of files that need to be backed up for the incremental backup which is single stream and can be time consuming depending upon the number of small files changes for that Incremental iteration. This is only an indexing part, data also needs to be copied out of the snapshots to external storage.
  • 2. Dis-advantage of NDMP backups are: a. For TERA & Peta bytes of storage - You can only do '9' Incremental for traditional NDMP Backups and after that you need to once again face the nightmare of long backup windows for Full NDMP dumps, sometimes running into weeks if not days. Clustered Data ONTAP 8.3 onwards supports 32 levels of NDMP dump Backups for NetApp. Level 0 = is a Full Backup. Level 1 through Level 31 are Incremental Backups. However, after '31' Incremental NDMP backups, nightmare of running a Full backup will be back. Whereas, SnapDIff is incremental forever. b. NDMP Dump is always streamed out of the storage to secondary storage media - Either Disk or Tape. In other words, even if you have manageable small number of files and changed blocks, they still need to be copied out to external media. SnapDiff - Is a fast indexer/crawler, which quickly identifies the file and directory differences between two Snapshots (Base, n-1 & Incremental, n). It does it by quickly looking at the block numbers of the inode file's indirect and data blocks, instead of shooting through the inode file sequentially which could contains million files, wasting time and resources. If a block(s) number has not changed, then it does not need to be crawled. As a result, SnapDiff can identify just the data blocks that have changed and crawl only those data (files/dir). With Intellisnap option - You can choose to keep the Primary Snaps on the NetApp itself, that means the Snapshot creation job is only '2' second task. Rest of the time is spent in 'Snap Differencing' for the files/dir inside the snapshot. You can also defer the SnapDiff for later time and even do the SnapDiff on the Secondary NetApp. If that was not enough, NetApp now allows CommVault to do Live browse - Which enables admins to quickly mount the snapshot volume to access data in order to restore it back. Is SnapDIff API free to implement or is it licensed? Only approved [Backup] partners are given accompanying documentation and guidelines on using SnapDiff APIs. Please see this post for more information. https://community.netapp.com/t5/Software-Development-Kit-SDK-and-API- Discussions/snapdiff-API-documentation/td-p/42561 Can NDMP DUMP Backup use 'SnapDIff' API? No. NDMP DUMP Backups are done via NDMP native UNIX 'dump' engine which is completely different from the way the way SnapDIff engine works. Is SnapDiff applicable to both NAS & SAN protocols? No. SnapDiff is only applicable to NAS [CIFS (SMB)/NFS] environment.
  • 3. What do you mean by 'maxdiff' or 'SnapDiffMax? It is the default number [256] of file differences requested in each query. What is the purpose of 'SnapDiffMax' additional setting on the Media Agent? SnapDiffMax additional registry setting allows NAS iDA to increase or decrease the number of files to be exported. However, before making any changes, it is recommended to contact NetApp to verify that the file server configurations can support changes in the values of these additional settings. How is SnapDIff performance measured? SnapDiff performance is not measured in throughput [MB/GB/sec], rather in terms of 'number of files processed per hour', in more technical language it is called 'Diffs/hour'. Please note, SnapDIff is not a backup tool, it is designed to speed up backup. What factors governs the performance of the 'SnapDIff' engine? Performance is not affected by the size of the NetApp volumes in terms of terabytes or petabytes. Nor do the sizes of the files catalogued have any effect on performance. However, SnapDIff performance can get degraded if the source volume contains many millions of files spread over a “deep” (many levels) directory structure. Which Ontap version should the filer be for better SnapDiff performance? Starting with Data ONTAP 8.2 onwards – regardless of FAS physical architecture (Clustered Data ONTAP or 7-mode) – SnapDiff’s performance has been greatly improved. Can increasing the maxdiff registry value from default '256' items to '1024' or more affect filer's performance? Please note - SnapDiff is a low priority process on a NetApp storage system. Processes that support file sharing, such as NFS and/or CIFS, as well as replication functions like SnapVault and SnapMirror all run at a higher priority. If load on a storage system becomes a problem, there are several ways of minimizing the load, such as limiting the number of volumes selected for SnapDiff from a single sub client. What are the requirements for using SnapDiff API with NetApp filer? SnapDiff API requires: XML over HTTP[s] & Vol Lang to be 'UTF-8'. The HTTP[S] connection is used only to transmit administrative data between the backup software and the NetApp filer. The administrative session data includes information such as filer credentials, snapshot information, and file names and attributes that are generated by the snapshot differencing process. The HTTP[S] connection is not used to transmit normal file data that is accessed on the filer by the client through file sharing protocols. Ensure: 1. Http/https is enabled 2. Vol Language on the Filer is set to - UTF-8 [en_US.UTF-8] 3. create_ucode and convert_ucode flag is turned on - Especially if SnapDiff ZAPI is run on volumes on with nfs-only compatible names. 4. Account used for running the SnapDIff on the Filer has this role capability - login-http- admin,api-*
  • 4. How do Backup applications interact with SnapDiff engine? Backup applications makes use of following SnapDiff Zephyr API [ZAPI], in short XML messages transmitted over HTTP(S) snapdiff-iter-start snapdiff-iter-next snapdiff-iter-end Is there a limit on the concurrent SnapDiff session to NetApp controller? DataONTAP limits the number of concurrent SnapDiff sessions to 16 per controller. When planning for NAS backups that perform indexing it is recommended to distribute the schedules so that no more than 16 jobs are running concurrently on a single controller. SnapDiff starting point can be traced with following lines in CVNasFileScan.log on the Media Agent server: 3996 f0c 01/29 15:35:19 173477 ManageONTAP::GetVolLanguage Get volume language. FileServer:[ONTAP7MODE] Volume:[CIFS] 3996 f0c 01/29 15:35:20 173477 SnapDiffStart FileServer:[ONTAP7MODE] Volume:[CIFS] SnapOne:[SP_2_173476_31_1454081328] SnapTwo:[SP_2_173477_32_1454081692] Format:[UTF8] Items In Each Iteration [256] 3996 f0c 01/29 15:35:24 173477 SnapDiffStart File-access-protocol support:[true] version check:[true] 3996 f0c 01/29 15:35:24 173477 SnapDiffStart atime support:[true] version check:[true] 3996 f0c 01/29 15:35:25 173477 SnapDiffStart End-of-diff support: [true] 3996 f0c 01/29 15:35:26 173477 SnapDiffStart Successfully kicked off SnapDiff. SnapDiff API module scans the Base & Diff snapshot and identifies changed files using: 1. Inode [No] 2. Change-type attribute: [Values could be - file_creation | inode_modification | file_deletion ] Steps to capture 'SnapDIff' process from CommVault side. 1. Open Process Manager on the Media Agent Server responsible for IntelliSnap NAS iDA backups. 2. Go to Logging Tab | Select Module 'CVNasFileScan' 3. Increase the DbgLevel | LogFileSize | LogFileMaxVer to higher number as desired
  • 5. In this example: 1. CIFS share is setup on NetApp NAS Volume [/vol/CIFS]. 2. Three Files A.txt, B.txt & C.txt is copied to share. Content of each file are: A.txt - 1,2 B.txt - 1,2,3,4 c.txt - 1, 2, 3, 4, 5, 6 Steps to capture SnapDIff process: 1. Run 'Full' Backup of CIFS share /vo/CIFS using IntelliSnap - Example Job ID:173473 2. Open CVNasFileScan using GxTail on the Media Agent and trace the log for Job ID:173473
  • 6. CVNasFIlescan log shows '3' Files [A, B, C] being identified for Backup with change- type [file_creation] SnapDiff Indexing via inode identification in snapshot: 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Changed file: Original Name [/vol/CIFS/.snapshot/SP_2_173473_29_1454079783/Download/A.txt] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - inode: [99] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - change-type: [file_creation] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ftype: [1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - links: [1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - fattr: [0x1ff] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - [0]=3 bits: UserId,groupId,sticky 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - [777]=Owner,Group,Other rwx bits 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - owner: [0] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - group: [0] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - size: [2] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - crtime: [1454079489] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - mtime: [1454079508] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ctime: [1454079508] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - atime: [1454079785] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ntacl-inode: [-1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - streamdir-inode: [-1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - dos-bits: [1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - nfsacl-inode: [-1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Modified file name [/vol/CIFS/Download/A.txt] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - File Name sent to Indexing [/vol/CIFS/Download/A.txt] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Changed file: Original Name [/vol/CIFS/.snapshot/SP_2_173473_29_1454079783/Download/B.txt] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - inode: [100] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - change-type: [file_creation] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ftype: [1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - links: [1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - fattr: [0x1ff] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - [0]=3 bits: UserId,groupId,sticky 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - [777]=Owner,Group,Other rwx bits 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - owner: [0] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - group: [0] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - size: [4] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - crtime: [1454079494] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - mtime: [1454079513] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ctime: [1454079513] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - atime: [1454079785] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ntacl-inode: [-1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - streamdir-inode: [-1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - dos-bits: [1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - nfsacl-inode: [-1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Modified file name [/vol/CIFS/Download/B.txt] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - File Name sent to Indexing [/vol/CIFS/Download/B.txt] 4968 1be4 01/29 15:03:46 173473 IDXCONNECT sendFileInfo: File '/vol/CIFS/Download/B.txt', restart 0, offset 0 size 4 flags 0
  • 7. 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Changed file: Original Name [/vol/CIFS/.snapshot/SP_2_173473_29_1454079783/Download/C.txt] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - inode: [103] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - change-type: [file_creation] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ftype: [1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - links: [1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - fattr: [0x1ff] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - [0]=3 bits: UserId,groupId,sticky 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - [777]=Owner,Group,Other rwx bits 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - owner: [0] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - group: [0] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - size: [6] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - crtime: [1454079502] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - mtime: [1454079523] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ctime: [1454079523] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - atime: [1454079785] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - ntacl-inode: [-1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - streamdir-inode: [-1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - dos-bits: [1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - nfsacl-inode: [-1] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - Modified file name [/vol/CIFS/Download/C.txt] 4968 1be4 01/29 15:03:46 173473 IndexSnapShot_SnapDiff() - File Name sent to Indexing [/vol/CIFS/Download/C.txt] As this is the first 'Full' IntelliSnap Job, both Full & Incremental snaps are same FullSnap: [SP_2_173473_29_1454079783] IncrSnap: [SP_2_173473_29_1454079783] 1. This time we went back to the CIFS shared directory and made following changes: 2. We left the text File 'A' & 'B' untouched. 3. We modified text File 'C' and added two additional integers - 7 & 8 [Previously it had :123456] We ran another SnapDIff Incremental Backup. As we can see below, SnapDiff quickly identified the changed file in seconds, here in this example - we expect to see 'C' text file only, and there it is. We do not see other two text files in the SnapDIff changed files as expected. CVNasFIlescan log shows 'C.txt' being indentified as changed file with change- type [inode_modification] 3560 1984 01/29 15:11:49 173475 VolFetchSnapDiff() - Started SnapDiff fetch for volume [CIFS] 3560 1984 01/29 15:11:49 173475 VolFetchSnapDiff() - Will Fetch the chaged files in queue element [0] 3560 1764 01/29 15:11:51 173475 IndexSnapShot_SnapDiff() - Indexing files at queue element [0] <--This indicates any new files, as there were no new files, the value returned here is 0. 3560 1984 01/29 15:11:59 173475 VolFetchSnapDiff() - Fetched the chaged files in queue element [0] 3560 1984 01/29 15:11:59 173475 VolFetchSnapDiff() - Will Fetch the chaged files in queue element [1] <---This indicates, we have one changed file. 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - Changed file: Original Name [/vol/CIFS/.snapshot/SP_2_173475_30_1454080264/Download/C.txt] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - inode: [103] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - change-type: [inode_modification] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - ftype: [1]
  • 8. 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - links: [1] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - fattr: [0x1ff] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - [0]=3 bits: UserId,groupId,sticky 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - [777]=Owner,Group,Other rwx bits 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - owner: [0] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - group: [0] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - size: [8] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - crtime: [1454079502] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - mtime: [1454080236] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - ctime: [1454080236] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - atime: [1454080236] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - ntacl-inode: [-1] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - streamdir-inode: [-1] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - dos-bits: [1] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - nfsacl-inode: [-1] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - Modified file name [/vol/CIFS/Download/C.txt] 3560 1764 01/29 15:11:59 173475 IndexSnapShot_SnapDiff() - File Name sent to Indexing [/vol/CIFS/Download/C.txt] Incremental SnapDiff: SnapDiffStart FileServer:[ONTAP7MODE] Volume:[CIFS] SnapOne:[SP_2_173473_29_1454079783] SnapTwo:[SP_2_173475_30_1454080264] FullSnap: [SP_2_173473_29_1454079783] IncrSnap: [SP_2_173475_30_1454080264] 1. This time we went back to the CIFS shared directory and made following changes: 2. Just touched the '3' Files | Opened them and closed them. 3. We basically 'changed' the access time of these Files. We ran another SnapDiff Incremental Backup. As we can see below, SnapDIff did not see any changed file, as we only touched the three files: SnapDiffStart FileServer:[ONTAP7MODE] Volume:[CIFS] SnapOne:[SP_2_173475_30_1454080264] SnapTwo:[SP_2_173476_31_1454081328] FullSnap: [SP_2_173473_29_1454079783] IncrSnap: [SP_2_173476_31_1454081328] 1. This time we went back to the CIFS shared directory: 2. We left the text File 'A' & 'B' untouched. 3. We delete 'C.txt' text file. We ran another SnapDIff Incremental Backup. As we can see below, SnapDiff quickly identified the changed file in seconds, here in this example - we expect to see 'C' text file only, and there it is. We do not see other two text files in the SnapDIff changed files as expected. CVNasFIlescan log shows 'C.txt' being indentified as changed file with change-type [file_deletion] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - Changed file: Original Name [/vol/CIFS/.snapshot/SP_2_173476_31_1454081328/Download/C.txt] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - inode: [103] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - change-type: [file_deletion]
  • 9. 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - ftype: [1] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - links: [1] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - fattr: [0x1ff] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - [0]=3 bits: UserId,groupId,sticky 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - [777]=Owner,Group,Other rwx bits 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - owner: [0] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - group: [0] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - size: [8] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - crtime: [1454079502] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - mtime: [1454080236] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - ctime: [1454080236] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - atime: [1454081329] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - ntacl-inode: [-1] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - streamdir-inode: [-1] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - dos-bits: [1] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - nfsacl-inode: [-1] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - Modified file name [/vol/CIFS/Download/C.txt] 3996 f0c 01/29 15:35:27 173477 IndexSnapShot_SnapDiff() - Deleted File Name sent to Indexing [/vol/CIFS/Download/C.txt] Incremental SnapDiff: SnapDiffStart FileServer:[ONTAP7MODE] Volume:[CIFS] SnapOne:[SP_2_173476_31_1454081328] SnapTwo:[SP_2_173477_32_1454081692] FullSnap: [SP_2_173473_29_1454079783] IncrSnap: [SP_2_173477_32_1454081692] Following example shows First Full [Base snapshot] and two Incremental backup jobs: SnapDiffStart FileServer:[CMODE83.TEST.COM] Volume:[/CMODE83_SVM_08/CMODE83_SVM_08_CIFS] Full = SnapOne:[] SnapTwo:[SP_2_173538_83_1454447246] Format:[UTF8] Items In Each Iteration [256] Incremental = SnapOne:[SP_2_173538_83_1454447246] SnapTwo:[SP_2_173539_85_1454448660] Format:[UTF8] Items In Each Iteration [256] Incremental = SnapOne:[SP_2_173539_85_1454448660] SnapTwo:[SP_2_173548_86_1454528619] Format:[UTF8] Items In Each Iteration [256] Differencing between latest and the previous snapshot: FullSnap: [SP_2_173538_83_1454447246] IncrSnap: [SP_2_173538_83_1454447246] FullSnap: [SP_2_173538_83_1454447246] IncrSnap: [SP_2_173539_85_1454448660] FullSnap: [SP_2_173538_83_1454447246] IncrSnap: [SP_2_173548_86_1454528619]
  • 10. SnapDiff API XML interaction can be traced via wireshark on the Media Agent, following screenshots shows SnapDiff APIs in action: snapdiff-iter-start snapdiff-iter-next snapdiff-iter-end
  • 11. SnapDiff API XML interaction captured on the NetApp filer side: snapdiff-iter-start snapdiff-iter-next snapdiff-iter-next
  • 12. snapdiff-iter-end Troubleshooting SnapDiff failures: SnapDiff failures are not easy to troubleshoot especially considering the fact that CommVault makes use of the standard APIs, and all the functionality is basically carried out by SnapDIff on the NetApp filer itself. However, we can certainly do some investigation from our side in order to make sure the basic requirements for SnapDiff operations are met [see the FAQ section]. From a broad perspective, there are three components involved that can cause 'SnapDIff' failures at various API stages such as snapdiff-iter-start, snapdiff-iter-next,snapdiff-iter-end? Three components are: 1.SnapDiff engine 2.SnapDiff ZAPI 3.Underlying TCP/IP network that enables XML Data exchange over HTTP/HTTPS. SnapDIff engine: Unless there is identified bug, SnapDiff should norrmally work fine, there are some bugs identified for SnapDiff are mentioned towards the end of the document. SnapDIff API: Like CommVault there are other Backup vendors who also uses the standard SnapDIff APIs just like us, and unless there is some configuration issue, it should be fine. Network: This is probably the main gray area which causes 90% of the SnapDIff errors. According to NetApp, some of the main causes are: 1. Misconfigured ISL (interswitch link) 2. Jumbo frames misconfigured (set on some points but not all points in the network path) Verify the vendor recommendations for MTU and tcp-adjust-mss settings 3. Host side NIC teaming (Such as HP network teaming) misconfigured
  • 13. Audit logs on the NetApp filer side are quite handy for monitoring ZAPI calls: The Audit logs are located in ‘log’ directoryfilernameETC$underlog)and ETCare$ share ( named ‘auditlog’, ‘auditlog.0’, ‘auditlog.1’,the etcageof. theThelog; theindex numbe bigger the number, the older is the log. filer>options auditlog.enable on Some APIs are considered read-only as these calls are only to obtain information from the filer. By default, APIs considered read-only are not logged to the auditlog. Logging can be manually enabled using the command: filer>options auditlog.readonly_api.enable on A packet trace between the controller and application host server may help diagnose the network level issue, please see the following KB articles from NetApp on how to run the packet traces: Please note - SnapDiff API requires converting information retrieved from Filer in to XML and ‘UTF-8’ is the default character encoding for XML, and therefore, it is important that the 'Volume' against which SnapDiff is run is set to en_US.UTF-8 encoding at the time of Volume creation. **NetApp does not recommend changing the volume language after the 'files' have been copied/written** If you do turn the switches create_ucode and convert_ucode on, later, please ensure: Enable the create_ucode and convert_ucode flag on the volume and do the following: 1) Create a Snapshot copy 2) On windows: Start | Run and type: 'attrib /s /d' command in the volume which is mounted in Windows. This will convert all the names to unicode in the active file system. 3) Disable i2p and then re-enable i2p.
  • 14. Recommended KB Articles from NetApp: How to collect a network trace with pktt in Data ONTAP 7-Mode https://kb.netapp.com/support/index?page=content&id=1010155&actp=LIST How to capture packet traces on Data ONTAP 8 Cluster-Mode systems https://kb.netapp.com/support/index?page=content&id=1011204 NetApp KB articles related to SnapDiff: API error: Parsing error in results: Extra content at the end of the document https://kb.netapp.com/support/index?page=content&id=2018206 NetApp BUGs: http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=533349 Degraded Snapdiff ZAPI performance when running on volume with create_ucode and convert_ucode flag off http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=533349 Snapdiff ZAPI with cifs file-access-protocol fails with "character out of allowed range" xml parse error for filenames with invalid/reserved unicode characters http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=588627 ONTAP panic when API snapdiff or inodepath takes more than 6 secs to process a large link count file. http://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=773781 Controller disruption while using snapdiff to run indexing jobs ashwinwriter@gmail.com March, 2016