Troubleshooting Restore - Linux File System

The following sections provide information on some of the troubleshooting scenarios related to restore:

Table of Contents

Browse and Restore

Symptom

Some of the files included in the user-defined subclient are not restored.

Solution

You may be performing the restore of user-defined subclient along with the restore of default subclient.

When you are recovering data backed up by the default subclient, you cannot recover the data backed up by a user-defined subclient.

Symptom

Browse from default subclient displays No Backup at Time error.

Solution

Ensure that you do not include the root directory (/) as the subclient content.

When performing point-in-time restore from the default subclient, include the data/folders under the root directory as the subclient content.

Symptom

Restore fails when trying to restore to a Unix FAT32 directory.

Solution

You may be restoring the full contents of a Unix directory that contains more than 32,767 files to a single Unix FAT32 directory.

The number of entries in a single FAT32 directory cannot exceed 32,767.

Symptom

Restores from UNIX to Windows fails.

Solution

Restores from UNIX to Windows may fail in the following circumstances:
  • If the files contain UNIX-specific device files such as block, character, or named pipe.
  • If the UNIX files contain the "?" character in their filename.
  • Windows allows 1024 characters for filenames, including the path. A filename, including the path, with more than 1024 (1023 for AIX) characters will not be restored from a UNIX computer to a Windows computer.
  • If the UNIX files contain hard links.

Make sure to avoid these while restoring the file system from UNIX to Windows.

Case-Sensitive Filenames Will Not Restore from UNIX to Windows

Case-sensitive filenames will not restore from UNIX to Windows because Windows does not maintain case-sensitive filenames. For example, if you are trying to restore files named "CASE_SENSITIVE" and "case_sensitive" from UNIX to Windows, then whichever file is restored last will overwrite the first one that was restored.

Symptom

ACLs and other extended attributes may not be restored.

Solution

Sometimes, when you restore data to an NFS-mounted file system, ACLs and other extended attributes may not be restored.

Symptom

Restore fails when trying to restore a running executable file.

Solution

Ensure that you are not including any running executable files in the restore operation.

LNX0002: Example of Linux full system restore on an LVM based server

This example will show an In-Place Restore of a complete Linux operating system that was installed using LVM based storage.

Prior to performing a restore of this nature the existing operating system must be examined to capture file system sizes and disk partition layouts.

This can be done using the following commands:

  1. cat /etc/fstab
  2. cat /proc/partitions
  3. fdisk -l
  4. df
  5. lvm
    1. pvdisplay sub command within lvm
    2. vgdisplay
    3. lvdisplay

The following is the captured output from these commands run on a Linux computer that will be used to demonstrate this restore:

# cat /etc/fstab

/dev/VolGroup00/LogVol00 / ext3 defaults 1 1
/dev/VolGroup00/LogVol03 /var ext3 defaults 1 2
/dev/VolGroup00/LogVol02 /usr ext3 defaults 1 2
LABEL=/boot /boot ext3 defaults 1 2
tmpfs /dev/shm tmpfs defaults 0 0
devpts /dev/pts devpts gid=5,mode=620 0 0
sysfs /sys sysfs defaults 0 0
proc /proc proc defaults 0 0
/dev/VolGroup00/LogVol01 swap swap defaults 0 0

# cat /proc/partitions

major minor #blocks name

8 0 10485760 sda
8 1 104391 sda1
8 2 10377990 sda2
253 0 1572864 dm-0
253 1 1441792 dm-1
253 2 5242880 dm-2
253 3 2097152 dm-3
 

# fdisk -l

Disk /dev/sda: 10.7 GB, 10737418240 bytes
255 heads, 63 sectors/track, 1305 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Device Boot Start End Blocks Id System
/dev/sda1 * 1 13 104391 83 Linux
/dev/sda2 14 1305 10377990 8e Linux LVM

# df

Filesystem 1K-blocks Used Available Use% Mounted on
/dev/mapper/VolGroup00-LogVol00 1523568 476028 968900 33% /
/dev/mapper/VolGroup00-LogVol03 1396600 101824 1222688 8% /var
/dev/mapper/VolGroup00-LogVol02 5078656 2892516 1923996 61% /usr
/dev/sda1 101086 12634 83233 14% /boot
tmpfs 1029784 0 1029784 0% /dev/shm

# lvm
lvm>
lvm> pvdisplay

--- Physical volume ---
PV Name /dev/sda2
VG Name VolGroup00
PV Size 9.90 GB / not usable 22.76 MB
Allocatable yes (but full)
PE Size (KByte) 32768
Total PE 316
Free PE 0
Allocated PE 316
PV UUID BmTTrd-Tpe9-Lf6D-ic0f-HGCj-4E31-FrWJ7y

lvm> vgdisplay

--- Volume group ---
VG Name VolGroup00
System ID
Format lvm2
Metadata Areas 1
Metadata Sequence No 5
VG Access read/write
VG Status resizable
MAX LV 0
Cur LV 4
Open LV 4
Max PV 0
Cur PV 1
Act PV 1
VG Size 9.88 GB
PE Size 32.00 MB
Total PE 316
Alloc PE / Size 316 / 9.88 GB
Free PE / Size 0 / 0
VG UUID x94Vk1-MeGP-TTGG-FRhb-jUdZ-WA7q-SujSiy

lvm> lvdisplay

--- Logical volume ---
LV Name /dev/VolGroup00/LogVol00
VG Name VolGroup00
LV UUID XAFA3v-1D3k-S646-7Hzg-AC8c-2Xjt-mT6fW0
LV Write Access read/write
LV Status available
# open 1
LV Size 1.50 GB
Current LE 48
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:0


--- Logical volume ---
LV Name /dev/VolGroup00/LogVol03
VG Name VolGroup00
LV UUID jDCqyo-0r0A-UbiM-wmZi-jdf7-4g3U-JPqiJN
LV Write Access read/write
LV Status available
# open 1
LV Size 1.38 GB
Current LE 44
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:1


--- Logical volume ---
LV Name /dev/VolGroup00/LogVol02
VG Name VolGroup00
LV UUID F7ZuKi-Jn1C-sRxG-rw4F-unNy-o3ls-WE9r2Y
LV Write Access read/write
LV Status available
# open 1
LV Size 5.00 GB
Current LE 160
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:2


--- Logical volume ---
LV Name /dev/VolGroup00/LogVol01
VG Name VolGroup00
LV UUID JtwYci-7RhO-JPvC-ZGSN-u1Zq-qeTz-c9KsCF
LV Write Access read/write
LV Status available
# open 1
LV Size 2.00 GB
Current LE 64
Segments 1
Allocation inherit
Read ahead sectors auto
- currently set to 256
Block device 253:3

lvm> exit

Solution

Once this information has been gathered, and a full backup has been run on the Linux computer, the client needs to be powered off, and the system disk has to be replaced with a same sized disk. In addition to this disk, another disk also needs to be added to install a basic Linux operating system which will be used to restore the original disk. Be sure to replace the original system disk with the same type of disk using the same target ID. Add the additional new disk to a different SCSI ID from the original system disk. This will ensure that reconfiguring the restored operating system goes smoothly.

On this computer, the original 10GB sda disk is replaced with another sda disk (SCSI target 0), and a new 4GB sdb disk (SCSI target 1) is added. A bare minimum Linux operating system will be installed onto the sdb disk which will then get SnapProtect file system agent installed on to it. At that point a restore will be performed, restoring the data back on to sda (as it was originally installed).

From the output above it can be noted that the root (/) partition was located on /dev/VolGroup00/LogVol00, a swap partition was located on /dev/VolGroup00/LogVol01, /usr was located on /dev/VolGroup00/LogVol02, /var was located on /dev/VolGroup00/LogVol03, and the /boot partition was located on a regular non-LVM /dev/sda1 partition. The VolGroup00 was located on disk partition /dev/sda2.

Here is a table showing the original disk layout:

Device Mount Point Volume Group Logical Volume Size
/dev/sda1 /boot - - 100MB
/dev/sda2 - VolGroup00 LogVol00 9.88GB
/dev/VolGroup00/LogVol00 / VolGroup00 LogVol01 1.5GB
/dev/VolGroup00/LogVol01 swap VolGroup00 LogVol02 2GB
/dev/VolGroup00/LogVol02 /usr VolGroup00 LogVol03 5GB
/dev/VolGroup00/LogVol03 /var VolGroup00 LogVol04 1.38GB

The disk layout on /dev/sda will be re-created during the installation of Linux operating system onto the second disk (sdb).

When installing Linux operating system on the second disk, the grub boot loader will get installed onto /dev/sda, and will point to /dev/sda1 for the boot information. A swap partition will be created on /dev/sda2 in LogVol1, which will be what is used for the Linux install to sdb, so no swap partition will be needed on /dev/sdb.

For the install to the sdb disk, the following partitions should be configured on /dev/sda and /dev/sdb during the installation. Make sure to size the partitions and logical volumes appropriately using the output gathered in the beginning of this example.

Below is a table showing the disk layout and mount points of sda & sdb prior to restore:

Device Mount Point Volume Group Logical Volume Size
/dev/sda1 /boot - - 100MB
/dev/sda2 - VolGroup00 - 9.88GB
/dev/VolGroup00/LogVol00 /mnt/restore VolGroup00 LogVol00 1.5GB
/dev/VolGroup00/LogVol01 swap VolGroup00 LogVol01 2GB
/dev/VolGroup00/LogVol02 /mnt/restore/usr VolGroup00 LogVol02 5GB
/dev/VolGroup00/LogVol03 /mnt/restore/var VolGroup00 LogVol03 1.38GB
/dev/sdb1 / - - 4GB
  1. After the operating system is installed onto the sdb disk, renew the client certificates. See Renewing a Revoked Certificate for step-by-step instructions.
  2. Install the File System iDataAgent (using the standard defaults).
  3. Unmount the /dev/sda1 partition from /boot and re-mount it under the /mnt/restore/boot directory to restore the original boot files.

    For example:

    # mkdir /mnt/restore/boot

    # umount /boot

    # mount /dev/sda1 /mnt/restore/boot

  4. Initiate a restore from the CommCell Console to restore / (the entire system) to the /mnt/restore directory. Select the Unconditionall Overwrite option to force the restore to overwrite the files in /mnt/restore/boot.
  5. After the restore completes, a /proc directory needs to be created in the restored root partition. This can be completed using the command

    mkdir /mnt/restore/proc

  6. Set up the boot loader to boot from the newly restored disk sda. The following procedure will accomplish this task..
    1. Mount proc & dev to the top level of the restored root file system
    2. Run the chroot command to point to the /restore partition/directory
    3. Run the grub install on the sda device.

    Here is an example execution of this procedure:

    # mount -t proc none /mnt/restore/proc

    # mount -o bind /dev /mnt/restore/dev

    # chroot /mnt/restore

    # grub-install /dev/sda
    Installation finished. No error reported.
    This is the contents of the device map /boot/grub/device.map.
    Check if this is correct or not. If any of the lines is incorrect,fix it and re-run the script `grub-install'.

    # this device map was generated by anaconda
    (fd0) /dev/fd0
    (hd0) /dev/sda

  7. Copy the operating system files that have hardware information in them to the /mnt/restore directory if the hardware on the computer is different from that of the source computer.

    For example, on RHEL compatible operating systems like CentOS and Oracle Linux, you need to copy the following files:

    cp –rp /etc/udev/rules.d/70-persistent-net.rules /mnt/restore/etc/udev/rules.d/70-persistent-net.rules
    cp –rp /etc/sysconfig/network-scripts/ifcfg-ethX /mnt/restore/etc/sysconfig/network-scripts/ifcfg-ethX
    cp –rp /etc/fstab /mnt/restore/etc/fstab

    Remove the /mnt/restore/etc/blkid/blkid.tab file if present.

    For example, on RHEL compatible operating systems like CentOS and Oracle Linux, you need to exclude the following files:

  8. Edit the /mnt/restore/etc/fstab file to remove /mnt/restore from the mount points:

    For example:

    /mnt/restore -> /

    /mnt/restore/usr -> /usr

    /mnt/restore/var -> var

  9. At this point the computer can be powered off, and the sdb disk can be removed. After removing the second disk, the computer will boot normally using the newly restored LVM partitioned sda disk.
  10. Renew the client certificates again. See Renewing a Revoked Certificate for step-by-step instructions.

Recovering data associated with deleted clients and storage policies

Symptom

In a disaster recovery scenario, use the following procedure to recover data associated with the following entities:

  • Deleted storage policy
  • Deleted client, agent, backup set or instance

Before You Begin

This procedure can be performed when the following are available:

  • You have a Disaster Recovery Backup that contains information on the entity that you are trying to restore. For example, if you wish to recover a storage policy (and the data associated with the storage policy) that was accidentally deleted, you must have a copy of the disaster recovery backup that was performed before deleting the storage policy.
  • Media containing the data you wish to recover is available and not overwritten.
  • If a CommCell Migration license was available in the CommServe when the disaster recovery backup was performed, no additional licenses are required. If not, obtain the following licenses:
    • IP Address Change license
    • CommCell Migration license

    See License Administration for more details.

  • A standby computer, which is used temporarily to build a CommServe.
Recovering Deleted Data
  1. Locate the latest Disaster Recovery Backup that contains the information on the entity (storage policy, client, agent, backup set or instance) you are trying to restore.
    • Check the Phase 1 destination for the DR Set or use Restore by Jobs for CommServe DR Data to restore the data.
    • If the job was pruned and you know the media containing the Disaster Recovery Backup, you can move the media in the Overwrite Protect Media Pool. See Accessing Aged Data for more information. You can then restore the appropriate DR Set associated with the job as described in Restore by Jobs for CommServe DR Data.
    • If the job is pruned and you do not know the media containing the Disaster Recovery Backup, you can do one of the following:
      • If you regularly run and have copies of the Data on Media and Aging Forecast report, you can check them to see if the appropriate media is available.
      • If you do not have an appropriate report, and know the media that contains the DR Backup, catalog the media using Media Explorer. Once the cataloging process is completed, details of the data available in the media are displayed.
  2. On a standby computer, install the CommServe software. For more information on installing the CommServe, see Install the CommServe.
  3. Restore the CommServe database using the CommServe Disaster Recovery Tool from the Disaster Recovery Backup described in Step 1. (See CommServe Disaster Recovery Tool for step-by-step instructions.)
  4. Verify and ensure that the NetApp Client Event Manager NetApp Communications Service (EvMgrS) is running.
  5. If you did not have a CommCell Migration license available in the CommServe when the disaster recovery backup was performed, apply the IP Address Change license and the CommCell Migration license on the standby CommServe. See Activate Licenses for step-by-step instructions.
  6. Export the data associated with the affected clients from the standby CommServe as described in Export Data from the Source CommCell.

    When you start the Command Line Interface to capture data, use the name of the standby CommServe in the -commcell argument.

  7. Import the exported data to the main CommServe as described in Import Data on the Destination CommCell.

    This brings back the entity in the CommServe database and the entity is visible in the CommCell Browser. (Press F5 to refresh the CommCell Browser if the entity is not displayed after a successful merge.)

  8. You can now browse and restore the data from the appropriate entity.

    As a precaution, mark media (tape media) associated with the source CommCell as READ ONLY before performing a data recovery operation in the destination CommCell.

Installation

Symptom

Error while loading shared libraries.

Solution

On Linux clients, the below error appears when we run any process or service:

<process name>: error while loading shared libraries: <lib>.so: cannot enable executable stack as shared object requires: Permission denied

For example,

#./ifind -getmnt -all
./ifind: error while loading shared libraries: libCvOnTap.so: cannot enable executable stack as shared object requires: Permission denied

As a workaround, do the following steps:

  1. Check if ASL (Atomic Secured Linux) is configured on client.

    #uname –r
    2.6.32.59-17.art.i686.PAE

    .art indicates that ASL is configured on the client.

  2. Check for the presence of the below logs in /var/log/messages file.

    May 24 22:01:08 rhel6 kernel: Aborting core
    May 24 22:01:08 rhel6 kernel: PAX: execution attempt in: <anonymous mapping>, bfc46000-bfc5b000 bffeb000
    May 24 22:01:08 rhel6 kernel: PAX: terminating task: /usr/libexec/paxtest/mprotstack(mprotstack):13201, uid/euid: 0/0, PC: bfc5acf4, SP: bfc5acdc
    May 24 22:01:08 rhel6 kernel: PAX: bytes at PC: c3 1a a3 ae 2b ac 9f ae f4 0f 9f ae 00 00 00 00 f4 0f 9f ae

  3. Run the following command.

    chpax –ps ../iDataAgent/<process/service>

    For example:

    #chpax –ps /opt/snapprotect/iDataAgent/ifind