Troubleshooting - DB2 iDataAgent

The following section provides information on troubleshooting restores.

Table of Contents

Restore Failures

Out of place restore failures The restore to a new database or another instance from the CommCell Console fails.

Do the following:

  • Make sure that database is not created on a different directory other than the default location.
  • The  DBHEAP, UTIL_HEAP, and/or APP_CTL_HEAP_SZ configuration parameters for the (source) database that you want to restore should contain the default value.

Viewing Restore Logs

You can troubleshoot restores by checking the CLDb2Agent.log restore log located in  <software installation path>\Log Files. You can also check details on the restore by issuing the db2ckrst DB2 command and then reviewing the output. The output for this command lists all of the backup images involved in the restore. The command syntax is as follows:

db2ckrst [-d <database name>] [-t <timestamp string>]

where:

<database name> is the name of the database and <timestamp string> is a numerical value representing the backup image involved in the restore.

Timestamps are displayed in the Restore Options | Restore Arguments dialog box in the CommCell Browser.

For example:

In this example, the timestamp for the database image included in the command appears twice in the output — in the first line and in the last line. This indicates that this (delta or incremental) backup image was the last backup image in the backup cycle and therefore was restored first and then last among all of the backup images in the cycle.

db2ckrst -d db_one -t 20030224125749

might generate the following output:

Suggested restore order of images using timestamp 20030224125749

for database db_one

============================================================

restore db db_one incremental taken at 20030224125749

restore db db_one incremental taken at 20030224124314

restore db db_one incremental taken at 20030224125211

restore db db_one incremental taken at 20030224125749

=============================================================

General

DB2 iDataAgent issue with backup or restore - List of Items for Effective Troubleshooting When you encounter a DB2 iDataAgent issue with backup or restore, provide the following information to the Customer Support for troubleshooting the issues:
  • Failed backup/restore Job Type, backup or restore with JOB ID number – Initiated either from CommCell Console or DB2 Command line.
  • Complete set of logs from the CommServe, MediaAgents and the DB2 iDataAgent having an issue. Do not collect logs by Time Range or JOB ID options as this will not provide the information required to troubleshoot a DB2 Backup or Restore issue.
  • The exact OS platform for UNIX or Windows servers that the DB2 Application is running on with FixPack or Maintenance levels.
  • Log on to the DB2 server using the DB2 user account configured in the CommCell Console and perform the following commands:

    db2level
    (This command will provide the current DB2 application level)
    db2 list db directory
    (This command will provide all of the DB2 databases currently configured)
    db2 get db cfg for database_name
    (The database_name for each database listed in the above command. This will need to be run multiple times, once for each database_name listed)

In most cases, you may be required to send the DB2 applications db2diag.log. This log is not part of the NetApp Software components. It is the DB2 application database log file.

DB20011: Restore Fails with Permissions Issue

Issue:

The restore fails due to issues accessing the SnapProtect registry, log files and base directories.

Solution

Run the Database Readiness Check.

DB20012 Target Database Cannot Access Latest Log Chain During a Cross-Machine Restore

Issue:

When the DB2 log chain is being used during a cross-machine restore and the target database does not have access to the latest log chain, the log retrieval restore fails during the backup.

Solution:

Prior to running the restore:

  1. Ensure the latest DB2 FixPack is installed on the target DB2 database.
  2. In the CommCell Console, delete the target database instance, which also removes references to previous data.
  3. Delete the database on the target server and recreate it.
  4. In the CommCell Console, re-create the instance and the backup set for the target database.
  5. Re-run the cross-machine restore to refresh the production database to the target database.

Recovering data associated with deleted clients and storage policies

Symptom

In a disaster recovery scenario, use the following procedure to recover data associated with the following entities:

  • Deleted storage policy
  • Deleted client, agent, backup set or instance

Before You Begin

This procedure can be performed when the following are available:

  • You have a Disaster Recovery Backup that contains information on the entity that you are trying to restore. For example, if you wish to recover a storage policy (and the data associated with the storage policy) that was accidentally deleted, you must have a copy of the disaster recovery backup that was performed before deleting the storage policy.
  • Media containing the data you wish to recover is available and not overwritten.
  • If a CommCell Migration license was available in the CommServe when the disaster recovery backup was performed, no additional licenses are required. If not, obtain the following licenses:
    • IP Address Change license
    • CommCell Migration license

    See License Administration for more details.

  • A standby computer, which is used temporarily to build a CommServe.
Recovering Deleted Data
  1. Locate the latest Disaster Recovery Backup that contains the information on the entity (storage policy, client, agent, backup set or instance) you are trying to restore.
    • Check the Phase 1 destination for the DR Set or use Restore by Jobs for CommServe DR Data to restore the data.
    • If the job was pruned and you know the media containing the Disaster Recovery Backup, you can move the media in the Overwrite Protect Media Pool. See Accessing Aged Data for more information. You can then restore the appropriate DR Set associated with the job as described in Restore by Jobs for CommServe DR Data.
    • If the job is pruned and you do not know the media containing the Disaster Recovery Backup, you can do one of the following:
      • If you regularly run and have copies of the Data on Media and Aging Forecast report, you can check them to see if the appropriate media is available.
      • If you do not have an appropriate report, and know the media that contains the DR Backup, catalog the media using Media Explorer. Once the cataloging process is completed, details of the data available in the media are displayed.
  2. On a standby computer, install the CommServe software. For more information on installing the CommServe, see Install the CommServe.
  3. Restore the CommServe database using the CommServe Disaster Recovery Tool from the Disaster Recovery Backup described in Step 1. (See CommServe Disaster Recovery Tool for step-by-step instructions.)
  4. Verify and ensure that the NetApp Client Event Manager NetApp Communications Service (EvMgrS) is running.
  5. If you did not have a CommCell Migration license available in the CommServe when the disaster recovery backup was performed, apply the IP Address Change license and the CommCell Migration license on the standby CommServe. See Activate Licenses for step-by-step instructions.
  6. Export the data associated with the affected clients from the standby CommServe as described in Export Data from the Source CommCell.

    When you start the Command Line Interface to capture data, use the name of the standby CommServe in the -commcell argument.

  7. Import the exported data to the main CommServe as described in Import Data on the Destination CommCell.

    This brings back the entity in the CommServe database and the entity is visible in the CommCell Browser. (Press F5 to refresh the CommCell Browser if the entity is not displayed after a successful merge.)

  8. You can now browse and restore the data from the appropriate entity.

    As a precaution, mark media (tape media) associated with the source CommCell as READ ONLY before performing a data recovery operation in the destination CommCell.

Vendor Code Troubleshooting

Return Code 23

Issue:

  1. The requested backup image or log has been successfully restored.

Solution

  1. There is no action required.

Return Code 25

Issue:

  1. There was an I/O error.

Solution

  1. Check permissions on the DB2 archive paths..
  2. Verify the disk space on the host that the data is being restored to.

Return Code 30

Issue:

  1. There was a critical error.

    11403590 304 05/12 18:03:18 1048366 Db2Check::SqlInfoPrint() - 0:
    ---- error report ----
    app. message = backing up the database
    line = 4804
    file = ClDb2AgentLib.cpp
    SQLCODE = -2079
    SQL2079N An error was reported by the shared library
    "/opt/simpana/Base64/libDb2Sbt.so". Return code: "30".

    --- end error report ---
    11403590 304 05/12 18:03:18 1048366 ClDb2Agent::DataBackup() - 1: SetPendingCauseAndEvent:DB2_SBTCRITICAL_ERROR

Solution

  1. Contact support with the following information ready to send.
    • The database configurations,
    • the db2dump directory,
    • all crash dumps