Troubleshooting – SnapProtect for VMware

Table of Contents

SS0001: Completed with one or more errors

Backup jobs from Virtual Server iDataAgent can be displayed as "Completed w/ one or more errors" in the Job History in the following cases:

  • If the virtual machine, virtual machine disk, or virtual machine configuration file fails to back up.
  • If one or more virtual machines in a backup job fails to back up.
  • If communication fails with vCenter.
  • If a disk included in a backup is not supported (for example, independent disks or physical RDM).

If the metadata collection operation fails during a snap backup job, the job is displayed as "Completed w/ one or more errors" in the Backup Job History of the subclient. You can configure the bCWEJobMDataFails additional setting if you want to display the status as "Completed"  in such scenarios. This is applicable only for backup jobs for which the Enable Granular Recovery option is selected.

SS0002: While performing SnapProtect backup on a Linux VM,  metadata collection is not included in the backup

Metadata collection using the Enable Granular Recovery option is not supported for SnapProtect backups of Linux virtual machines.

Resolution

To recover files and folders from SnapProtect backups of Linux virtual machines, use Live File Recovery.

SS0003: Mount operations on an ESX server are failing using NFS protocol

Resolution

When performing SnapProtect operations on VMware using the NFS file-based protocol, ensure the following:

  • The NetApp storage device name specified in Array Management matches that on the ESX Server.
  • The VMkernel IP address of all ESX servers that are used for mount operations should be added to the root Access of the NFS share on the source storage device.

SS0004: Two snapshots are created

Symptom

For a Windows virtual machine on ESX 4.1, if the disk.UUID attribute is enabled and no user defined snapshots are created for the virtual machine, two snapshots of the virtual machine are created.

Resolution

If you do not want to create two snapshots, disable the disk.UUID attribute.

SS0005: File level revert is not supported for a Virtual Machine

Symptom

The Revert job gets killed with the following error:

File level revert is not supported for a Virtual Machine [%VMNAME%] which is on VMFS datastore. [Datastore(s) [%DatastoreName%] does not support file level revert.

Cause

The virtual machine may have NFS and iSCSI disks. Ensure that all the disks reside on the NFS data store. If the data does not reside on the NFS data store, you cannot perform the revert operation.

Resolution

Perform the regular Container Restore to bring the data back to the point-in-time.

SS0006: Exchange mining operation fails

Symptom

Exchange Snap Mining operation may fail if SAN mode is used to perform the backup and snapshots were exposed in read-only mode.

Resolution

It is recommended to use NBD mode for Exchange Snap Mining backup. However, if you want to use SAN mode, ensure that all the disks exposed to the proxy computer have read and write permissions.

Follow the steps give below to clear read-only attributes of any SAN shared disk:

  1. Open the Command Prompt on the proxy computer.
  2. Enter the following commands:

    diskpart

    san policy=onlineall

SS0007: The Operation is not allowed in the current state

Symptom

During the backup copy operation, the registration of a virtual machine fails with the following error:

The Operation is not allowed in the current state

Cause

The ESX server which hosts the virtual machine is in the maintenance mode.

Resolution

Before initiating the backup copy operation, ensure that the ESX server is not in the maintenance mode.  If you are performing an inline backup copy, before initiating the SnapProtect operation, ensure that the any host involved in the backup, is not in the maintenance mode.

SS0008: Driver cannot be found

Symptom

You may get following error while restoring files and folders:

Driver cannot be found

Cause

The VDDK version on the proxy computer has changed.

Resolution

Reboot the proxy computer.

Additional Information

See Troubleshooting issues with virtual machines or datastore names containing special characters (2046088).

SS0010: Discovery Failed. Unable to access host

Symptom

When you are mounting any snapshot on a host, the system is unable to mount the snapshot and you may get following error:

Discovery Failed. Unable to access host

Cause

The Snaps created during SnapProtect operation dialog box displays a list of all the snapshots on a storage array. If you open the Snaps created during SnapProtect operation dialog box by one of the following methods, you can only view the list of snapshots and cannot mount the snapshots:

  • Right click the snapshot copy of a storage policy and select List Snaps.
  • From the Control Panel, double click Array Management. Select the required array in the Array Management dialog box and click List Snaps.

Resolution

For more information about correct method mounting a snapshot, refer to Mount Snapshots.

VMW0064: Changed block tracking verification failed for disk [%s] on virtual machine [%s]

Symptom

For specific virtual machines in a subclient, an incremental or differential backup automatically converts into a full backup. The following event message is displayed:

Changed block tracking verification failed for disk [%s] on virtual machine [%s]; performing a full backup of the disk. The virtual machine may have been vMotioned since the last backup. If this issue persists it may be necessary to reset changed block tracking on the virtual machine.

Cause

The first backup for the virtual machine becomes a full backup after you migrate the virtual machine to a new datastore.

This is because Changed Block Tracking (CBT) is reset after a storage vMotion operation. For more information, refer to Change Block Tracking is reset after a storage vMotion operation in vSphere 5.x (2048201).

Resolution

All subsequent backups in the backup cycle are performed according to the schedule for the migrated virtual machines.

To reset CBT, see Reset Changed Block Tracking (CBT) for VMware backups.

SS0011: NFS datastore becomes inaccessible after mounting the snapshot

Symptom

Successfully mounted datastore becomes inaccessible when the browse operation is performed or data is accessed.

Cause

This issue is caused by incorrect MTU settings between ESX proxy host and NFS storage.

Resolution

To check if the issue is occurring due to incorrect MTU settings, use ping command with DF “do not fragment” option by providing different MTU values and check if Ping is working as expected.

VMW0070: File-level restore fails when destination client has a MediaAgent

Symptom

A file-level restore fails when restoring from an SnapProtect backup to a destination client that has a MediaAgent.

Cause

When mounting the virtual machine, the restore operation tries to use the MediaAgent on the destination client rather than the MediaAgent on the source client, and the mount fails.

Resolution

  1. Initiate the restore.
  2. During the restore operation, select the content to be restored and the destination client.
  3. On the Restore Options dialog, click Advanced, then click the Data Path tab.
  4. In the Use MediaAgent field, select the MediaAgent for the source client.
  5. In the Use Proxy field, select the source proxy.
  6. Complete the restore.

When the source MediaAgent is used, the mount succeeds and the file-level restore completes successfully.

SS0012: Unable to create a Virtual Machine snapshot

Symptom

A VM backup fails and presents the following error code:

Error Code 91:26 Unable to create a Virtual Machine snapshot of [].

Cause

The snapshot exceeds the maximum file size for the server.

The following message is shown in the JobManager.log file on the CommServe:

30368 75c4 10/30 14:15:33 1188013 Servant [---- IMMEDIATE BACKUP REQUEST ----], taskid [21876] Clnt[...] AppType[Virtual Server][106] BkpSet[Initial VM Test] SubClnt[Single VM Test (Only) -VC02] BkpLevel[Full][1]

30368 81e8 10/30 14:17:18 1188013 Scheduler Set pending cause [Unable to create a Virtual Machine snapshot of [...]. Please check the Virtual Machine snapshot tree. It may be necessary to upgrade/reinstall the VMware tools on the Virtual Machine. Please consult http://kb.vmware.com/kb/1009073 for additional information.]::Client [...] Application [vsbkp] Message Id [1526726682] RCID [0] ReservationId [0]. Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0].

Review the vsbkp.log file located on the Virtual Server Agent Client.

Default Log location:  C:\Program Files\CommVault\Simpana\Log Files

5028 5 10/30 14:17:00 1188013 ### CreateSnapshot_Task --- Started Create Snapshot task task-2967 for VM [...] 5028 5 10/30 14:17:16 1188013 ### _WaitForTask --- Task [task-2967] status [error] waited for [00:00:15.4438020] task took [00:00:14.5466880] Wait [10/30/2012 2:17:00 PM to 10/30/2012 2:17:16 PM] Task [10/30/2012 2:17:00 PM to 10/30/2012 2:17:15 PM] 5028 5 10/30 14:17:16 1188013 ### CreateSnapshot_Task --- Failed to create Snapshot from VM [...] - File is larger than the maximum size supported by datastore ' 5028 eec 10/30 14:17:16 1188013 CVMWareInfo::_CreateVMSnapshot() - Unable to create snapshot of VM [...], trying without quiesing filesystem 5028 5 10/30 14:17:16 1188013 ### CreateSnapshot_Task --- Started Create Snapshot task task-2968 for VM [...] 5028 5 10/30 14:17:18 1188013 ### _WaitForTask --- Task [task-2968] status [error] waited for [00:00:02.1215728] task took [00:00:01.6874780] Wait [10/30/2012 2:17:16 PM to 10/30/2012 2:17:18 PM] Task [10/30/2012 2:17:16 PM to 10/30/2012 2:17:17 PM] 5028 5 10/30 14:17:18 1188013 ### CreateSnapshot_Task --- Failed to create Snapshot from VM [...] - File is larger than the maximum size supported by datastore ' 5028 eec 10/30 14:17:18 1188013 CVMWareInfo::_CreateVMSnapshot() - Failed to create Snapshot of VM [...] Guid [...]

Resolution

Check the virtual machine snapshot tree. It may be necessary to upgrade or reinstall the VMware tools on the virtual machine. For more information, see Unable to take a quiesced VMware snapshot of a virtual machine (1009073).

The error message "File is larger than the maximum size supported by datastore" is a VMware error; there is a VMware Knowledge Base article that covers the proper way to fix the issue: 

 Creating a snapshot for a virtual machine fails with the error: File is larger than maximum file size supported (1012384)

Review and follow all steps in the VMware Knowledge Base article. Compare the base disk size of the VMDK or virtual-mode RDM of the virtual machine against the block size of the datastore that contains the working directory of the virtual machine.

By default, the working directory contains the virtual machine's .vmx configuration file. The maximum file size differs among versions of ESX and ESXi, and among versions of VMFS.

SS0013: VMware snapshot removal failures

Symptom

Snapshots are not removed in VMware after the Virtual Server Agent (VSA) has finished running jobs.

Cause

Issue 1:

In the Tasks log for specific VMs there is a job called "Remove snapshot" that has a status of "Unable to access file since it is locked". This is usually because the vCenter server is too busy at the time.

The trigger for the VMware Consolidated Helper-0 snapshot not getting deleted is one of two items:

  1. High I/O on the virtual machine is preventing the snapshot from going away.
  2. High I/O on the ESX side, not just the VM. If you have a machine causing high I/O on the ESX host, this causes the Consolidated Helper-0 snapshot to be locked and not get deleted. This can happen on every VM, including ones that are 99% idle.

See the section titled "Snapshot consolidation failure when cleaning up after HotAdd backup" in the VDDK 5.0 U1 Release Notes.

Issue 2:

CommVault Software sends the snapshot removal command to vCenter and vCenter sends it to the ESX which does the removal; but VMware does not process the snapshot removal as expected.

The SnapProtect software might deliver the following error message for VMware API-reported issues.

Error Code 91:63: Failed to create snapshot of VM [...].

A more generic error message might be delivered if some VMs succeeded and others did not.

Error code 91:19: At least one VM failed to back up.

Resolution

Starting with the release of SnapProtect 9.0 SP6A, the latest VDDK 5.0 update 1 is included with the service pack and is part of the service pack install.

The CommServe system and MediaAgents must be at the same SP level and must be at the same SP level or a higher SP level then clients. All other SP level variations create an unsupported CommCell condition.

The VMware VDDK 5.0 U1 update has the fix to resolve the Consolidated Helper-0 issue for HotAdd backups.

For Issue 1:

If you are unable to update to SnapProtect 9.0 SP6A or higher, implement the following workarounds for the snapshot removal in the backup environment. VMware supplied this workaround for the Consolidated Helper-0 snapshot issue.

  1. Try to schedule VM backups so that a maximum of four VMs are simultaneously backed up.
  2. Try not to back up VMs from the same datastore at the same time.
  3. Try not to back up the busiest VMs at the same time.

These suggested steps should ensure that the ESX service console is not too busy, enabling it to delete the snapshots.

For Issue 2:

VSA-created orphaned snapshots older than 24 hours will be removed during the next backup of the VM.

  1. For rare instances where the default 24-hour interval is insufficient, the nExpireSnapshotHours additional setting can be configured on the server where the Virtual Server Agent (VSA) is installed:
    • Name: nExpireSnapshotHours
    • Category: Virtual Server
    • Type: INTEGER
    • Value: 24 (Decimal) # of hours (0 = disable)
  2. In most cases go into the Virtual Infrastructure Client ( VIC ) and create a new snapshot, and then afterward, use the option to "Delete all snapshots" to get them to clean up.
  3. Confirm the cleanup has occurred.

VMW0036: Virtual machine client names are created with '_1' appended to the original client name

Symptom

When viewing virtual machines in the Client Computers list, you may see duplicated client names (for example, client_name and client_name_1).

Cause

If the instance GUID of the virtual machine changes, a duplicate virtual machine client is created with a numeric suffix (for example, client_name_1).

Resolution

For deployments using SnapProtect Service Pack 10 or later:

The QS_SetVMClient script merges backup history for duplicate virtual machine clients, and then deletes the duplicate virtual machine client. All subsequent jobs will be associated with the merged client. The script is available in the Base directory for SnapProtect software.

Perform the following steps to merge duplicate clients. Run the script for each set of duplicate clients.

  1. If you are not logged on to the CommServe computer, run the qlogin command.
  2. At a command prompt, navigate to the software installation path, log in to the CommServe machine, and run the following script:

    qoperation execscript -sn QS_SetVMClient -si @Client1='client_name_1' -si @Client2='client_name'

    If the client name contains spaces, you can run the QScript without explicit parameter names:

    qoperation execscript -sn QS_SetVMClient -si 'client name_1' -si 'client name'

    The duplicate clients are merged using the following criteria:

    • If both clients have a SnapProtect agent installed, the command fails.
    • If one of the clients has a SnapProtect agent installed, backup history is merged into the client with the SnapProtect agent.
    • If one of the clients has a numeric suffix in the client name (such as ‘_1’), backup history is merged into the client without the numeric suffix in its name.
    • If neither client has a SnapProtect agent installed or a numeric suffix in the client name, backup history is merged into the client with the latest backup.
    • If both clients have a numeric suffix in the client name, and do not have a SnapProtect agent installed, backup history is merged into the client with the latest backup.
  3. To log off the CommServe computer, run the qlogout command.
  4. Run an incremental backup and confirm that virtual machines are shown correctly in the Virtual Machine Status tab.

For deployments using SnapProtect Service Pack 9 or earlier:

  1. If you are not logged on to the CommServe computer, run the qlogin command.
  2. At a command prompt, navigate to the software installation path, log in to the CommServe machine, and run the following script:

    qoperation execscript -sn QS_SetVMClient -si @sourceClient='client_name_1' -si @destClient='client_name'

    where client_name is the original client name and client_name_1 is the duplicate.

    This script reassigns all backup history from client_name_1 to client_name. This enables you to view backup history, and to generate Job Summary Reports with the Include Protected VMs option enabled.

  3. To log off the CommServe computer, run the qlogout command.
  4. Remove the duplicate clients:
    1. In the CommCell Console, go to Control Panel | User Preferences.
    2. Click the Client Computer Filter tab.
    3. Select the Show Virtual Server Discovered Clients option.
    4. Delete the duplicate clients from the CommCell Browser.

VMW0039: You do not have access rights to this file

Symptom

When performing a backup in SAN mode, the following error is shown in the vsbkp.log and VixDisk.log:

You do not have access rights to this file

Cause

Windows 2008 R2 provided access for SAN mode transport when disks were offline; but in Windows 2012 disks must be online to be accessed. If the SAN policy for disks is Offline, disks are not accessible unless they are set to Online on the proxy.

Resolution

To enable backups using SAN transport on Windows 2012 servers, use one of the following approaches:

  • Set the disk to Online on the proxy.
  • Change the SAN policy to onlineall and set the SAN disk to read-only except for restores. You can use the diskpart utility to clear the read-only flag.

VMW0068: Reset Changed Block Tracking (CBT) for VMware backups

Symptom

If CBT is disabled or not functioning properly, the size of incremental backups can be larger than expected. In some cases issues with CBT can cause a full backup to take longer and, when restoring a lazy zero thick provisioned disk, might result in the disk becoming eager zero thick provisioned.

You can verify CBT issues in the CommCell Console by checking the Status Details dialog for a virtual machine backup job. The CBT Status field shows Disabled.

To display the Status Details dialog, right-click a backup job and select View Job Details, then click the Virtual Machine Status tab and double-click a virtual machine entry.

The vsbkp.log file includes information such as the following:

4168 6 08/15 13:37:42 98795 ### QueryChangedDiskAreas --- Exception during QueryChangedDiskAreas Error caused by file /vmfs/volumes/50f432d3-ba380107-2b26-90b11c070562/VM/VM-000003.vmdk

Note: Backups for ESXi 4.x or ESXi 5.x can also increase because of a VMware error as described in Increase in Backup Time after Fix for VMware CBT Error for expanded VMDKs.

Cause

Changed Block Tracking (CBT) is a VMware feature that identifies blocks of data that have changed or are in use. It enables incremental backups to identify changes from the last previous backup, writing only changed or in-use blocks. If CBT is disabled or not functioning properly, an incremental backup might back up a complete virtual machine disk instead of only changed blocks, or revert to a full backup.

Resolution

You can disable CBT to restart change tracking for subsequent backups. This resolution is applicable for ESX hosts 4.0 or later, and for virtual machines at hardware version 7 and later.

Before resetting CBT, ensure that there are no snapshots on the virtual machine.

To reset CBT for a specific virtual machine, you must disable CBT, power cycle the virtual machine, and run a new backup. When the new backup is run, CBT is automatically re-enabled.

You can also reset CBT on multiple virtual machines using a PowerCLI script that resets CBT without requiring that virtual machines be powered down.

See the following articles to find detailed instructions for resetting CBT:

As a result of resetting CBT, the time required for backups increases because the first backup after resetting CBT backs up all blocks for any virtual machines on which CBT was reset. This change can also increase disk storage requirements for backups. The change does not affect scheduled backup cycles or data aging.

Increase in Backup Time after Fix for VMware CBT Error for expanded VMDKs

VMware ESXi 4.x or ESXi 5.x can return incorrect information about virtual machine disk sectors after a virtual machine disk (VMDK) is expanded beyond 128 GB or any doubling of 128 GB, and Changed Block Tracking (CBT) can produce unrecoverable backups. This is a flaw in VMware APIs used by backup software, and can lead to data loss if not addressed immediately. Any virtual machine is at risk if VMDKs have been expanded beyond 128 GB, 256 GB, 512 GB, 1024 GB, or any doubling of 128 GB. For example, the issue can occur after expanding a VMDK from 100 GB to 150 GB, or after expanding a VMDK from 150 GB to 300 GB. Any backup performed since the disk expansion can contain invalid data, including full backups. For more information, including patches for ESXi 5.x, see QueryChangedDiskAreas command returns incorrect sectors after extending virtual machine vmdk file with Change Block Tracking (CBT) enabled (2090639).

Important: The patches provided by VMware prevent this problem from re-occurring, but do not correct VMDKs that were extended prior to the patch.

SnapProtect  Version 11, or Version 10 Service Pack 9 and later, include a fix for this issue to prevent data loss. The SnapProtect software automatically detects expanded disks and resets CBT:

  1. The first backup job resets CBT for all virtual machines with a disk size greater than 128 GB.

    The CBT Status for the affected virtual machines is displayed as Reset.

  2. The software stores the size of each VMDK file in the .vmx file for each virtual machine.
  3. For any backup jobs going forward, the backup job compares the current size of each VMDK file with the size recorded in the .vmx file.
  4. If a disk has been expanded, the software resets CBT and backs up all blocks for the VMDKs attached to the virtual machine.

Note: With this fix, the time and disk space for backups is increased.

For older service packs for SnapProtect Version 10 and for SnapProtect Versions 9 or 8, you can use the CheckChangeTracking tool to reset CBT on specific affected virtual machines. For more information, see KB article VMW0009. The tool automatically re-enables CBT for the next backup job, which will back up all blocks for VMDKs even if an incremental backup is requested. CBT is also enabled for future incremental backups.

If you apply VMware patch ESXi600-201511001 to the VMware ESXi 6.0, 6.0 Update 1, or 6.0 Update1a hosts, SnapProtect detects the new ESXi build version and reverts back to using CBT. The next incremental backup will accurately protect any blocks that have changed since the last successful backup.

VMW0040: Failed to start the virtual machine

Symptom

When a virtual machine has been replicated in vSphere and backed up, and a full VM restore is performed from the backup, the following status message might be displayed in vCenter:

Failed to start the virtual machine.

Cause

If the Power ON Virtual Machine After Restore option is selected when performing a full VM restore, VMware attempts to power on the restored VM before disabling replication.

Resolution

Clear the Power ON Virtual Machine After Restore option when initiating the restore. VMware automatically disables replication when the restore is completed, and you can power on the virtual machine manually.

SS0021: Unable to browse files on snapshot for Windows 2008 R2 virtual machine - Disk:[<Name>] filtered during SnapProtect operation

Symptom

A user is unable to browse files on a snapshot for a Windows 2008 R2 virtual machine (ESXi or ESX 4.1 and higher).

The following messages appear in the cvd.log file:

Disk:[<Name>] filtered during snap protect operation

Unable to open any disks on VM [<Name>_GX_BACKUP] XXX attempted

Cause

File-level browse of a virtual machine snapshot fails if the page files for the VM are filtered out. The virtual machine snapshot cannot be mounted because it needs the page file for the VM. The page file is unavailable because it resides on a datastore that is filtered for backup.

This issue occurs when the disk.EnableUUID attribute is set to true for the virtual machine. The disk.EnableUUID attribute enables application-level quiescing. If the UUID is not enabled, VMware performs file-system consistent quiescing.

Resolution

During the restore, file-level browsing succeeds if the user selects a MediaAgent that has the Virtual Server Agent installed and has access to the datastore that contains the page files. On the Advanced Restore Options dialog, go to the Data Path tab to specify a MediaAgent.

To resolve this issue, set the disk.EnableUUID attribute to false and run the SnapProtect backup again on the virtual machines where page files were filtered.

To modify the UUID setting in vSphere 5.1:

  1. Select the VM and power it off.
  2. On the Summary tab, click Edit Settings.
  3. On the Virtual Machine Properties dialog, go to the Options tab.
  4. Select the General field under Advanced.
  5. Click the Configuration Parameters button.
  6. Enter the disk.EnableUUID attribute and set the value to false.

As long as there is not a backup copy or auxiliary copy in progress, users should be able to browse files from the VM snapshot.

Disabling the UUID attribute does not affect applications that do not use Volume Shadow Copy Services (VSS), such as Microsoft SQL, Microsoft Exchange, or Active Directory.

Additional Information

See the following articles:

VMW0041: Cannot restore files from a Windows 2012 virtual machine using deduplication

Symptom

When restoring from a backup of a Windows 2012 virtual machine that has deduplication enabled, a file-level restore completes successfully but only creates stub files.

Cause

Windows 2012 or later provides its own native deduplication capability. If a Windows 2012 VM has native deduplication enabled, file-level restores from backups are not supported because the backed up files are only stored as stubs.

Resolution

Use one of the following methods to restore files:

  • When using a Virtual Server Agent proxy on Windows 2012 or later with Windows deduplication enabled, use Live File Recovery to restore files that have been dehydrated by Windows deduplication. A MediaAgent running Windows 2012 or later, with the Virtual Server Agent installed and with the Windows deduplication role enabled, must be used as the VSA proxy when restoring the dehydrated files.
  • To retrieve files from a backup for a Windows 2012 VM using deduplication, use the Virtual Machine files option on the Restore Options dialog to restore the virtual machine disk that contains the files.
  • Install a local file system agent on the Windows 2012 VM to enable file-level backups. As a result, the data is backed up in a rehydrated state so that you can restore file-level data from that backup.

VMW0043: Backup or restore of a virtual machine fails: "[91:30] Unable to open the disk(s) for virtual machine [<VMName>]."

Symptom

When using VDDK 5.1 or 5.5, backup or restore of a virtual machine fails.

The Job Details dialog shows the following error:

Error Code: [91:30]
Description: Unable to open the disk(s) for virtual machine [<VMName>]. Please ensure the following: 1) the proxy is able to communicate with the ESX host and resolve the ESX host address 2) the correct transport mode has been selected 3) the disk types configured to the virtual machine are supported.
Source: <proxy>, Process: vsbkp

The following message is included in the vixdisklib.log file:

Not licensed to use this function. Error 16064 at 2357.

Cause

The VDDK 5.1 or 5.5 APIs require that user permissions be applied at the vCenter level, as described in Restore or backup of virtual machines using VDDK API fails with the error: Not licensed to use this function. Error 16064 at 2357 (2063054).

Resolution

Assign user permissions for the vCenter entity using the vSphere Client or WebClient.

For step-by-step instructions, see Add a Custom User with Limited Scope.

VMW0045: Unable to create a virtual machine snapshot of [<ClientName>]. Snapshot hierarchy is too deep.

Symptom

VMware snapshots fail with the following error message:

Unable to create a virtual machine snapshot of [<ClientName>]. [Snapshot hierarchy is too deep. Snapshot creation attempt took [<Time>]] Please check the virtual machine snapshot tree. It may be necessary to upgrade/reinstall the VMware tools on the virtual machine.

Cause

Additional snapshots cannot be created If there are too many snapshots of the virtual machine, or if there are too many unconsolidated snapshots. This can be caused by:

  • A large number of manually or automatically created snapshots on the virtual machine.
  • Outdated version of VMware Tools on the virtual machine, which is unable to quiesce the file system and applications.
  • Failures during snapshot consolidation because a parent disk was open.

Resolution

  1. Delete snapshots on the virtual machine.
  2. Update VMware Tools to the latest version.
  3. Check for delta disks on the virtual machine or a virtual machine in the “needs consolidate” state. If the virtual machine cannot be manually consolidated, the owner of the lock on the disk needs to be identified. If HotAdd transport mode is being used for backups, the disks may still be attached to the backup proxy; if so they can be safely removed from the proxy.

Additional Information

VMW0047: Unable to mount snapshot: "The iSCSI initiator could not establish a network connection to the target."

Symptom

This issue occurs when trying to mount a snapshot SnapProtect for VMware in a VMware cluster using iSCSI. An SnapProtect backup is successful, and the LUN is successfully created and visible in the storage array. When an attempt is made to mount the snapshot on an ESX proxy that is not part of the cluster, the mount fails.

VMWare Mount snapshot failed.
...
JPR Set - Error [60611, VMWare Mount snapshot failed. (MM.60611)]

In VMware, the ESX proxy can see the LUN but the following message is displayed:

The iSCSI initiator could not establish a network connection to the target.

Cause

This issue can occur:

  • When the ESX server has multiple network interface cards (NICs) and the mount is attempted using the wrong NIC.
  • When both Fibre Channel (FC) and iSCSI are used by the Virtual Server Agent.

Resolution

You can configure the following additional settings to resolve this issue:

  • When the nIscsiEnable additional setting is configured, the snapshot always attempts the mount using the iSCSI method. If both FC and iSCSI are available and this key is not enabled, the SnapProtect backup attempts the mount using the FC method first.
  • By default the snapshot is exposed to the first host bus adapter (HBA) on the ESX server. To expose the snapshot to a specific HBA, configure the sPortInfo additional setting.

For more information, see Advanced Snapshots - SnapProtect for VMware.

VMW0049: Metadata collection fails on MBR disk: "GetVolumeInformation Failed!!"

Symptom

After performing an SnapProtect backup of a virtual machine running Windows Server 2012 R2, attempting a Live Browse of the virtual machine snapshot fails.

The following entries are shown in the cvd.log file for Live Browse or in the vsbkp.log file for the SnapProtect backup:

CVMWareInfo::_GetFileMountInfo_VCB4() - GetVolumeInformation Failed!! for [<MountPath>] err=13
CVMWareInfo::_GetFileMountInfo_VCB4() - Failed to get the volume size for [<MountPath>], 19

The following event is shown in the event viewer:

The default transaction resource manager on volume <VolumeID> encountered a non-retryable error and could not start. The data contains the error code.
The system has rebooted without cleanly shutting down first. This error could be caused if the system stopped responding, crashed, or lost power unexpectedly.
The system failed to flush data to the transaction log. Corruption may occur in VolumeId: <VolumeID>, DeviceName: <VolumePath>.

Cause

If a data set spans multiple partitions on a Master Boot Record (MBR) disk, metadata collection fails; Volume Shadow Copy Services (VSS) are unable to set the read-only attribute for some partitions and dismount the snapshot.

Resolution

To resolve this issue, either reconfigure the MBR disk to use a single partition or convert the MBR disk to a GUID Partition Table (GPT) disk as described in Auto-recovery fails on all but the first volume of an MBR disk.

Additional Resources

Change a Master Boot Record Disk into a GUID Partition Table Disk

Creating a quiesced snapshot of a Window 2008 R2 virtual machine generates Event IDs 50, 57, 137, 140, or 12289 (2006849)

VMW0056: MediaAgent and VSS Provider has to be installed to Enable Snap Backups

Symptoms

If a virtualization client for a vCenter also has a File System iDataAgent installed, you cannot enable SnapProtect on the client. The following error message is displayed when you select Enable SnapProtect on the General tab of the Advanced Client Properties dialog.

MediaAgent and VSS Provider has to be installed to Enable Snap Backups

Resolution

  1. Create a new virtualization client for the vCenter directly under Client Computers. Name it differently than the original virtualization client.
  2. On the newly created virtualization client, configure the vCenter connection:
    1. From the CommCell Console, navigate to Client Computers | virtualization_client | Virtual Server .
    2. Right-click the VMware object and select Properties.
    3. On the Virtual Server Instance Properties | General tab, click Change and enter the vCenter credentials.
    4. Click
  3. On the newly created virtualization client, enable SnapProtect:
    1. From the CommCell Console, navigate to Client Computers.
    2. Right-click the virtualization client and select Properties.
    3. Click Advanced.
    4. On the Advanced Client Properties dialog (General tab), select Enable SnapProtect.
    5. Click OK.

VMW0002: Virtual Machines residing on NFS storage become unresponsive during a snapshot removal operation

For more information, see KB Article VMW0002.

VMW0010: Error downloading or uploading virtual machine configuration file

Errors can occur during downloads or uploads of VMware virtual machine configuration files.

Symptom

The following errors can occur when transferring configuration files:

  • A backup job fails with one of the following errors:

    Error downloading virtual machine config file.

    Unable to download VMX files.

  • The following error message is displayed when trying to download a .vmx file from a datastore through vCenter:

    Error code: 91:29 - Unable to download the VMX file for virtual machine [...].

Cause

Issues can occur for several reasons:

  • Downloads or uploads can fail because of connectivity issues or high usage in vCenter.
  • The names of one or more virtual machines, datastores, or clusters included in a backup contain the following characters:

    + & @ % = # % * $ # ! \ / : * ? " < > | ; '

  • The management agent on the ESX host might need to be restarted.
  • For vCenter 4.0 or 4.1, updates might need to be installed.
  • Failure when trying to download a .vmx or .nvram file from a datastore through vCenter is an identified VMware vSphere VADP issue. The vCenter client datastore browser connected to vCenter server fails to download .vmx or .nvram files.

    For more information, see Using the vCenter Server 4.x/5.x datastore browser to download or copy a powered-on virtual machine's .vmx and .nvram files fails (1019286).

    There is an identified issue for VMware and Windows 2008 R2 Enterprise; the issue was resolved for other Windows operating systems in versions 4.0 U2 and 4.1.

Resolution

To resolve the issue, check the following items:

VMW0066: Logs are not truncating during VMware SnapProtect backup

Symptom

During a VMware backup using SnapProtect with the Application aware backup for item based recovery and Truncate Database Logs options selected, logs for Microsoft Exchange or Microsoft SQL Server are not truncated.

The following messages are shown in the vsbkp.log file:

CVMWareInfo::_PerformSnapOperation() - Skipping quiesce of VM [vm_client_name]. quiesce option [1], CVD port [0], powered state [1]

Cause

Log truncation is not performed if the client name for the virtual machine in the CommCell Console does not match the virtual machine name in the VMware inventory.

If an agent is installed on a virtual machine, the client name for the virtual machine is not renamed during VM discovery.

This issue can occur when an agent is installed with a hostname, but the virtual machine name in the inventory is different.

Resolution

To resolve this issue, use the CommCell Console to rename the virtual machine client to match the name used in the VMware inventory.

VMW0013: Backup or restore fails when a non-default port is specified for vCenter server

For more information, see KB Article VMW0013.

VMW0014: Guest file restore from Snap backup goes pending with message "Snapshot is currently mounting on a different host"

For more information, see KB Article VMW0014.