Troubleshooting Backup - Macintosh File System iDataAgent

The following section provides information on some of the troubleshooting scenarios related to backup:

Backup Failures

Backup fails The subclient content might have remote scripts with one or more blank lines at the top.

Ensure that you delete the blank lines at the top of any remote scripts.

Backup hangs indefinitely

Ensure that there are no stale NFS mounts on the server included in the backup.

If there are stale NFS mounts on the server, backups hang indefinitely. This can be avoided by ensuring that the NFS mount points are available for backups.

If the backups hang frequently because of this issue, mount the NFS share using the soft,timeo=10 option.

Error while loading shared libraries On Linux clients, the below error appears when we run any process or service:

<process name>: error while loading shared libraries: <lib>.so: cannot enable executable stack as shared object requires: Permission denied

For example,

#./ifind -getmnt -all
./ifind: error while loading shared libraries: libCvOnTap.so: cannot enable executable stack as shared object requires: Permission denied

As a workaround, do the following steps:

  1. Check if ASL (Atomic Secured Linux) is configured on client.

    #uname –r
    2.6.32.59-17.art.i686.PAE

    .art indicates that ASL is configured on the client.
  2. Check for the presence of the below logs in /var/log/messages file.

    May 24 22:01:08 rhel6 kernel: Aborting core
    May 24 22:01:08 rhel6 kernel: PAX: execution attempt in: <anonymous mapping>, bfc46000-bfc5b000 bffeb000
    May 24 22:01:08 rhel6 kernel: PAX: terminating task: /usr/libexec/paxtest/mprotstack(mprotstack):13201, uid/euid: 0/0, PC: bfc5acf4, SP: bfc5acdc
    May 24 22:01:08 rhel6 kernel: PAX: bytes at PC: c3 1a a3 ae 2b ac 9f ae f4 0f 9f ae 00 00 00 00 f4 0f 9f ae

  3. Run the following command.

    chpax –ps ../iDataAgent/<process/service>

    E.g.:

    #chpax –ps /opt/snapprotect/iDataAgent/ifind

Priority of SnapProtect Processes

By default, SnapProtect processes run at default priority on the client computers. If there are available CPU cycles, then SnapProtect processes will use the available CPU for backup and restore operations. If the CPU is being used by other application or system processes, SnapProtect processes will not preempt them. In such cases, if you want to give higher priority to other application or system processes, which are running at the default priority, you can modify the priority of the SnapProtect process using the following steps:
  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client> and click Properties.
  3. Click Advanced and then click Additional Settings tab.
  4. Click Add.
  5. In the Name field, type dNICEVALUE.

    The Category and Type fields are populated automatically.

  6. In the Value field, type the appropriate value.

    For example, 15.

  7. Click OK.

Restart the services on the client after setting this key.

Completed with one or more errors

By default the backup jobs from Macintosh File System iDataAgent complete without any error messages.

In the Scan phase files that cannot be accessed will be populated in scan_failures.cvf, in the following scenarios:

  • Stat failed.
  • If a file is deleted or renamed.
  • Unable to open a directory during scan

To view the "Completed w/ one or more errors" message in the Job History in the Scan phase, follow the steps given below:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client> and click Properties.
  3. Click Advanced and then click Additional Settings tab.
  4. Click Add.
  5. In the Name field, type sReportScanCompleteWithError.

    The Category and Type fields are populated automatically.

  6. In the Value field, type y or Y to enable the error message display or n or N to disable the error message display..
  7. Click OK.

During backups the jobs will be marked partial success and reported into Failures.cvf, in the following scenarios:

  • Stat failure on a backup file (moved/accessed, etc).
  • Unable to open a directory.
  • Error to get the file attributes.
  • If File is locked.
  • Path is a symbolic link and you are unable to read the symbolic link path.
  • Critical error while reading extended attributes or ACLs of the file or directory.
  • Size of the extended attributes or ACLs buffer exceeded the backup buffer size (default pipeline buffer size is 63kb).

To view the "Completed w/ one or more errors" message in the Job History after a backup, follow the steps given below:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client> and click Properties.
  3. Click Advanced and then click Additional Settings tab.
  4. Click Add.
  5. In the Name field, type sReportBackupCompleteWithError.

    The Category and Type fields are populated automatically.

  6. In the Value field, type y or Y to enable the error message display or n or N to disable the error message display..
  7. Click OK.

Troubleshooting Performance Issues

If you are experiencing performance issues during backup, you can troubleshoot them by enabling logging of performance details in the log files. These performance counters contain information that help in resolving the performance related issues during backups.

The performance counters printed in the log files include:

FileRead Average time spent in system call to read the file.
CheckFileHole Average time spent to verify if buffer read from the disk is a hole.
Close File Average time spent in system call to close the file.
AllocBuffer Average time spent waiting for the network buffer.
WriteBuffer Average time spent in sending the network buffer to the network.
Disk Read Average speed of reading from the disk.
Write Average speed of writing to the network.

Displaying Performance Details of a Backup Job

Procedure

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client> and click Properties.
  3. Click Advanced and then click Additional Settings tab.
  4. Click Add.
  5. In the Name field, type clBackupChild_PERF.

    The Category and Type fields are populated automatically.

  6. In the Value field, type Y.
  7. Click OK.

Displaying Performance Details for Each File in a Backup Job

In addition to viewing the performance details for a backup job, you can also view the performance details of each file in the backup job. Note that the file names are not printed in the log.

Procedure

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client> and click Properties.
  3. Click Advanced and then click Additional Settings tab.
  4. Click Add.
  5. In the Name field, type clBackupChild_PERF_FILE.

    The Category and Type fields are populated automatically.

  6. In the Value field, type Y.
  7. Click OK.

    This key should be used in conjunction with the clBackupChild_PERF registry key.

MAC0001: Macintosh OS X File System iDataAgent install getting FIFO error message

Symptom

The following FIFO error is shown during Macintosh File System iDataAgent installation in the /tmp/.gxsetip/cvpkgadd.log file:

>>> Executing: /opt/snapprotect/Base/svc_ctrl -focus Instance001 -create cvd /opt/snapprotect/Base/cvd >/dev/null 2>/dev/null ...
Failed to create FIFO /opt/snapprotect/Base/Temp/locks/mutex_UXService.fifo: Invalid argument

Resolution

Apple has reported this as a bug, the bug is a false positive. During the creation of the file mutex_UXService.fifo the operating system reports an Invalid Argument, this error should not appear as the file does get created.

Apple has identified a bug in the Macintosh OS X. It is a known limitation of OS X that mkfifo is not supported on certain file systems such as FAT32.

To make sure that this is the issue and to provide proof to Apple please run the following commands:

# mkfifo -m 666 /opt/snapprotect/Base/Temp/locks/mutex_UXService.fifo

# ls -l /opt/snapprotect/Base/Temp/locks/mutex_UXService.fifo
prw-rw-rw- 1 gbuilder admin 0 Mar 12 19:55 /opt/snapprotect/Base/Temp/locks/mutex_UXService.fifo

# rm /opt/snapprotect/Base/Temp/locks/mutex_UXService.fifo

# mkfifo /opt/snapprotect/Base/Temp/locks/mutex_UXService.fifo

# ls -l /opt/snapprotect/Base/Temp/locks/mutex_UXService.fifo
prw-r--r-- 1 gbuilder admin 0 Mar 12 19:56 /opt/snapprotect/Base/Temp/locks/mutex_UXService.fifo

Please contact Apple for an operating system upgrade or a fix.