SlideShare ist ein Scribd-Unternehmen logo
1 von 3
Snapshots with curly {alphanumeric}
brackets found on the NetApp volume
Background: Recently I was investigating large number of ‘NetApp Volume Snapshots’ on the 7-
mode FILER, which clearly looked like non-regular snapshots. I say non-regular b'cos normally FILER
does not allow you to create snaps with {} curly brackets, give it a try and it will throw an error.
Further into my investigation, I found out that there were hundreds of snaps across multiple
volumes hosting 'Exchange database' and along with {} snapshots there were also snaps such as:
'exchsnap__exch-srv1_MM-DD-YYYY_HH.MM.SS__daily' so that is an indication that there is an
application 'SnapManager for Exchange' behind the Exch DB backups. Hence, I found my source for
further investigation.
Source of the trouble:
Jumped to the server hosting SnapManager for Exchange application. Bydefault, SME works in
conjunction with VSS framework and SnapDrive, SME integrates with Microsoft VSS framework
seamlessly- i.e
VSS Components for SME Backup:
 VSS requestor = Backup Application ‘SnapManager for Exchange’
 VSS writer = MS Exchange writer ‘Comes with your Application : MS Exchange’
 VSS Provider (SAN) = NetApp VSS Hardware provider <-API--> SnapDrive <-ZAPI--> FILER
Please note: Data Ontap VSS Hardware Provider component comes with SnapDrive. In other words,
when you install SnapDrive, VSS Hardware Provider also gets installed.
Typical SnapManager for Exchange Backup process from VSS framework perspective for Exch 2003
onwards: [Volume Shadow Copy Service (VSS) was introduced in Windows Server 2003]
The entire process is orchestrated by the VSS coordination service initiated by SME:
1. SnapManager determines which LUNs it wants to capture and verifies that Exchange
Server is present as a valid writer.
2. SnapManager initiates the shadow copy process.
3. VSS informs Exchange Server and the Data ONTAP VSS Hardware Provider that a shadow
copy is starting. Exchange Server stops writing to disk.
4. VSS ensures that NTFS is in a consistent state.
5. VSS requests the Data ONTAP VSS Hardware Provider to create a shadow copy.
6. The Data ONTAP VSS Hardware Provider requests SnapDrive to create a Snapshot copy of
the storage system volume that contains the specified LUN.
7. SnapDrive requests that the storage system create a Snapshot copy of the specified
volume.
8. When the shadow copy is complete, VSS returns NTFS to a normal state and informs
Exchange that it can resume disk writes.
So, what is Curly bracket snapshot: is basically 'Shadow copy ID', in other words a 'GUID’ that
uniquely identifies a shadow copy' which is absolutely part & parcel of the SME Backup process.
Reason: When SnapManager for Exchange performs a backup, it creates the Snapshot copy using a
VSS-specific [GUID] name format, b’cos SnapManager for Exchange is integrated into VSS framework
and hence it is govern by VSS format, once the shadow-copy is created, 3rd
party application can ask
VSS to forget about the snapshot and do whatever they need to do, depending upon their backup
procedure.
Once the snapshot copy is taken successfully, it is then changed to its SnapManager-specific name
format. This happens automatically. One should be able to view the original VSS-specific name of the
Snapshot copy and the rename process in the SnapManager backup report which logs this name
change.
Examples of Snapshot copy names:
 VSS-specific Snapshot copy name:
{206f538c-ecd0-4799-90f9-baff5d8e0f15}
 SnapManager-specific Snapshot copy name:
exchsnap_exch-srv1_03-22-2006_17.53.33
Question: Why did these GUID snaps were not renamed in this case and were left
‘orphaned’?
Answer: Looks like SnapDrive at some point was unable to communicate with Storage
system (Initiator lost connection to the specified LUN or Snapshot is inconsistent or does not
exist for some reason) and if any of these instances occur, the Snapshot will not be deleted.
I traced this error in the Windows Application event log: In-band SCSI command from LUN (FC) to
storage system returned invalid data. Snapshot doesn't exist.
As it turns out there is a known BUG: 808158
https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=808158
BUG says: Sometimes when deleting backups after creating backups especially in identification and
cleaning up of orphaned Snapshots of either a database or log volume, Snap Drive might not delete
the specified Snapshot because of any of following reasons,
 Snapshot is inconsistent or does not exist for some reason
 Initiator lost connection to the specified LUN
 A Data ONTAP issue
If any of these instances occur, the Snapshot will not be deleted yet Snap Manager for Exchange
deletes its SnapInfo folder (for example, meta-data of the Snapshot from the log volume) as the first
step toward cleaning up the orphaned Snapshot. As a result, the orphaned Snapshot might be still
not cleaned up properly.
Workaround: There is no permanent solution but only workaround - Manually go to the specified
volume (where that DB LUN exists) and delete the orphaned curly {Snapshot}, which was not
cleaned up properly because SnapDrive was unable to delete it.
ashwinwriter@gmail.com
July, 2018

Weitere Àhnliche Inhalte

Mehr von Ashwin Pawar

Mehr von Ashwin Pawar (20)

How to connect to NetApp FILER micro-USB console port
How to connect to NetApp FILER micro-USB console portHow to connect to NetApp FILER micro-USB console port
How to connect to NetApp FILER micro-USB console port
 
NDMP backup models
NDMP backup modelsNDMP backup models
NDMP backup models
 
How to use Active IQ tool to access filer information
How to use Active IQ tool to access filer informationHow to use Active IQ tool to access filer information
How to use Active IQ tool to access filer information
 
San vs Nas fun series
San vs Nas fun seriesSan vs Nas fun series
San vs Nas fun series
 
Steps to identify ONTAP latency related issues
Steps to identify ONTAP latency related issuesSteps to identify ONTAP latency related issues
Steps to identify ONTAP latency related issues
 
SnapDiff
SnapDiffSnapDiff
SnapDiff
 
SnapDiff process flow chart
SnapDiff process flow chartSnapDiff process flow chart
SnapDiff process flow chart
 
SnapDiff performance issue
SnapDiff performance issueSnapDiff performance issue
SnapDiff performance issue
 
Volume level restore fails with error transient snapshot copy is not supported
Volume level restore fails with error transient snapshot copy is not supportedVolume level restore fails with error transient snapshot copy is not supported
Volume level restore fails with error transient snapshot copy is not supported
 
Disk reports predicted failure event
Disk reports predicted failure eventDisk reports predicted failure event
Disk reports predicted failure event
 
OCUM shows ONTAP cluster health degraded
OCUM shows ONTAP cluster health degradedOCUM shows ONTAP cluster health degraded
OCUM shows ONTAP cluster health degraded
 
NDMPCOPY lun from 7-mode NetApp to cDOT
NDMPCOPY lun from 7-mode NetApp to cDOTNDMPCOPY lun from 7-mode NetApp to cDOT
NDMPCOPY lun from 7-mode NetApp to cDOT
 
Latency in storage
Latency in storageLatency in storage
Latency in storage
 
NVRAM vs NVMEM
NVRAM vs NVMEMNVRAM vs NVMEM
NVRAM vs NVMEM
 
NAS vs SAN
NAS vs SANNAS vs SAN
NAS vs SAN
 
NetApp storage layering
NetApp storage layeringNetApp storage layering
NetApp storage layering
 
What is storage from client's perspective
What is storage from client's perspectiveWhat is storage from client's perspective
What is storage from client's perspective
 
Difference between cluster image package show-repository and system image get
Difference between cluster image package show-repository and system image getDifference between cluster image package show-repository and system image get
Difference between cluster image package show-repository and system image get
 
Cannot access NetApp 7-mode admin shares etc$
Cannot access NetApp 7-mode admin shares etc$Cannot access NetApp 7-mode admin shares etc$
Cannot access NetApp 7-mode admin shares etc$
 
How to generate,collect and upload ocum logs
How to generate,collect and upload ocum logsHow to generate,collect and upload ocum logs
How to generate,collect and upload ocum logs
 

KĂŒrzlich hochgeladen

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Safe Software
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
?#DUbAI#??##{{(☎+971_581248768%)**%*]'#abortion pills for sale in dubai@
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
panagenda
 

KĂŒrzlich hochgeladen (20)

Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers:  A Deep Dive into Serverless Spatial Data and FMECloud Frontiers:  A Deep Dive into Serverless Spatial Data and FME
Cloud Frontiers: A Deep Dive into Serverless Spatial Data and FME
 
Real Time Object Detection Using Open CV
Real Time Object Detection Using Open CVReal Time Object Detection Using Open CV
Real Time Object Detection Using Open CV
 
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
+971581248768>> SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHA...
 
Boost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivityBoost PC performance: How more available memory can improve productivity
Boost PC performance: How more available memory can improve productivity
 
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
Apidays New York 2024 - The Good, the Bad and the Governed by David O'Neill, ...
 
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost SavingRepurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
Repurposing LNG terminals for Hydrogen Ammonia: Feasibility and Cost Saving
 
🐬 The future of MySQL is Postgres 🐘
🐬  The future of MySQL is Postgres   🐘🐬  The future of MySQL is Postgres   🐘
🐬 The future of MySQL is Postgres 🐘
 
AWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of TerraformAWS Community Day CPH - Three problems of Terraform
AWS Community Day CPH - Three problems of Terraform
 
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...Workshop - Best of Both Worlds_ Combine  KG and Vector search for  enhanced R...
Workshop - Best of Both Worlds_ Combine KG and Vector search for enhanced R...
 
Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...Apidays New York 2024 - The value of a flexible API Management solution for O...
Apidays New York 2024 - The value of a flexible API Management solution for O...
 
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemkeProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
ProductAnonymous-April2024-WinProductDiscovery-MelissaKlemke
 
The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024The 7 Things I Know About Cyber Security After 25 Years | April 2024
The 7 Things I Know About Cyber Security After 25 Years | April 2024
 
Automating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps ScriptAutomating Google Workspace (GWS) & more with Apps Script
Automating Google Workspace (GWS) & more with Apps Script
 
Why Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire businessWhy Teams call analytics are critical to your entire business
Why Teams call analytics are critical to your entire business
 
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time AutomationFrom Event to Action: Accelerate Your Decision Making with Real-Time Automation
From Event to Action: Accelerate Your Decision Making with Real-Time Automation
 
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live StreamsTop 5 Benefits OF Using Muvi Live Paywall For Live Streams
Top 5 Benefits OF Using Muvi Live Paywall For Live Streams
 
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
Mastering MySQL Database Architecture: Deep Dive into MySQL Shell and MySQL R...
 
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, AdobeApidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
Apidays New York 2024 - Scaling API-first by Ian Reasor and Radu Cotescu, Adobe
 
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data DiscoveryTrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
TrustArc Webinar - Unlock the Power of AI-Driven Data Discovery
 
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
Strategies for Unlocking Knowledge Management in Microsoft 365 in the Copilot...
 

Guid snapshots with curly bracket found in NetApp Volume

  • 1. Snapshots with curly {alphanumeric} brackets found on the NetApp volume Background: Recently I was investigating large number of ‘NetApp Volume Snapshots’ on the 7- mode FILER, which clearly looked like non-regular snapshots. I say non-regular b'cos normally FILER does not allow you to create snaps with {} curly brackets, give it a try and it will throw an error. Further into my investigation, I found out that there were hundreds of snaps across multiple volumes hosting 'Exchange database' and along with {} snapshots there were also snaps such as: 'exchsnap__exch-srv1_MM-DD-YYYY_HH.MM.SS__daily' so that is an indication that there is an application 'SnapManager for Exchange' behind the Exch DB backups. Hence, I found my source for further investigation. Source of the trouble: Jumped to the server hosting SnapManager for Exchange application. Bydefault, SME works in conjunction with VSS framework and SnapDrive, SME integrates with Microsoft VSS framework seamlessly- i.e VSS Components for SME Backup:  VSS requestor = Backup Application ‘SnapManager for Exchange’  VSS writer = MS Exchange writer ‘Comes with your Application : MS Exchange’  VSS Provider (SAN) = NetApp VSS Hardware provider <-API--> SnapDrive <-ZAPI--> FILER Please note: Data Ontap VSS Hardware Provider component comes with SnapDrive. In other words, when you install SnapDrive, VSS Hardware Provider also gets installed.
  • 2. Typical SnapManager for Exchange Backup process from VSS framework perspective for Exch 2003 onwards: [Volume Shadow Copy Service (VSS) was introduced in Windows Server 2003] The entire process is orchestrated by the VSS coordination service initiated by SME: 1. SnapManager determines which LUNs it wants to capture and verifies that Exchange Server is present as a valid writer. 2. SnapManager initiates the shadow copy process. 3. VSS informs Exchange Server and the Data ONTAP VSS Hardware Provider that a shadow copy is starting. Exchange Server stops writing to disk. 4. VSS ensures that NTFS is in a consistent state. 5. VSS requests the Data ONTAP VSS Hardware Provider to create a shadow copy. 6. The Data ONTAP VSS Hardware Provider requests SnapDrive to create a Snapshot copy of the storage system volume that contains the specified LUN. 7. SnapDrive requests that the storage system create a Snapshot copy of the specified volume. 8. When the shadow copy is complete, VSS returns NTFS to a normal state and informs Exchange that it can resume disk writes. So, what is Curly bracket snapshot: is basically 'Shadow copy ID', in other words a 'GUID’ that uniquely identifies a shadow copy' which is absolutely part & parcel of the SME Backup process. Reason: When SnapManager for Exchange performs a backup, it creates the Snapshot copy using a VSS-specific [GUID] name format, b’cos SnapManager for Exchange is integrated into VSS framework and hence it is govern by VSS format, once the shadow-copy is created, 3rd party application can ask VSS to forget about the snapshot and do whatever they need to do, depending upon their backup procedure. Once the snapshot copy is taken successfully, it is then changed to its SnapManager-specific name format. This happens automatically. One should be able to view the original VSS-specific name of the Snapshot copy and the rename process in the SnapManager backup report which logs this name change. Examples of Snapshot copy names:  VSS-specific Snapshot copy name: {206f538c-ecd0-4799-90f9-baff5d8e0f15}  SnapManager-specific Snapshot copy name: exchsnap_exch-srv1_03-22-2006_17.53.33
  • 3. Question: Why did these GUID snaps were not renamed in this case and were left ‘orphaned’? Answer: Looks like SnapDrive at some point was unable to communicate with Storage system (Initiator lost connection to the specified LUN or Snapshot is inconsistent or does not exist for some reason) and if any of these instances occur, the Snapshot will not be deleted. I traced this error in the Windows Application event log: In-band SCSI command from LUN (FC) to storage system returned invalid data. Snapshot doesn't exist. As it turns out there is a known BUG: 808158 https://mysupport.netapp.com/NOW/cgi-bin/bol?Type=Detail&Display=808158 BUG says: Sometimes when deleting backups after creating backups especially in identification and cleaning up of orphaned Snapshots of either a database or log volume, Snap Drive might not delete the specified Snapshot because of any of following reasons,  Snapshot is inconsistent or does not exist for some reason  Initiator lost connection to the specified LUN  A Data ONTAP issue If any of these instances occur, the Snapshot will not be deleted yet Snap Manager for Exchange deletes its SnapInfo folder (for example, meta-data of the Snapshot from the log volume) as the first step toward cleaning up the orphaned Snapshot. As a result, the orphaned Snapshot might be still not cleaned up properly. Workaround: There is no permanent solution but only workaround - Manually go to the specified volume (where that DB LUN exists) and delete the orphaned curly {Snapshot}, which was not cleaned up properly because SnapDrive was unable to delete it. ashwinwriter@gmail.com July, 2018