Troubleshooting Restore - Oracle RAC iDataAgent

The following section provides information on troubleshooting restores.

Browse Failures

Point in time Table Browse Failures When you have encryption enabled for the client, point in time table browse operation fails with the following error message:

Pass-phrase protection is on for client [80], but pass-phrase was not specified.

Make sure that the pass phrase is exported to the MediaAgent when encryption is enabled for the client.

  1. From the CommCell Browser, right-click the client and select Properties.
  2. Click the Encryption tab.
  3. Click Via Pass-Phrase.
  4. Click Export.
  5. In the Destination Computer box, select the MediaAgent.
  6. In the Pass-Phrase box, type the pass-phrase used for encryption.
  7. In the Re-enter Pass-Phrase box, re-type the pass-phrase to confirm.
  8. Click Export.
  9. Click OK.

Restore Failures

Table Restore Failures
  • For Windows, make sure that the Oracle Services are running as Local System.

The table restores may fail with the following error message:

RMAN-05502: the target database must be mounted when issuing a DUPLICATE command.

  • Make sure to use connect string instead of ‘/’ when you configure an instance. For example, sys/pwd12@orcl4.
Database Restore Failures After performing an Oracle restore operation from the CommCell Console where options were selected for Redirect, Rename and Recover at the same time, you must click the Refresh button on the Subclient Properties (Content) tab or run a backup after the restore operation has completed before proceeding with another restore. This is necessary to ensure that the CommCell Console recognizes the changes that were made to the Oracle database and control file, so that it reflects the current structure of the database to be restored, otherwise the restore will fail.
Unable to create a Duplicate Database
  • When you create a duplicate database make sure that one of the instances use a proper connect string instead of ‘/’. For example, sys/pwd12@orcl4. This is required when you perform a duplicate database restore operation since we need to connect two instances in the same RMAN operation.
  • If a duplicate database restore fails with error PLS-00553: character set name is not recognized; then make sure that the  character sets are the same between the location from where you are running RMAN, and the location of the target database. As this is an Oracle related issue, please contact Oracle support for more information.
Increase in sbtio.log File Size

Sometimes, jobs fail due to increase in the size of sbtio.log file in the $UDUMP directory.

To resolve this, set the size limit for the sbtio.log file using the sMAXORASBTIOLOGFILESIZE registry key. Once the specified size limit is reached, the sbtio.log file gets pruned automatically.

Intermittent Table Level  Restore Failures The  table level restore operation may fail intermittently due to an error in the Oracle's DataPump utility and the following error message will be displayed:

UDE-00008: operation generated ORACLE error 31623

ORA-31623: a job is not attached to this session via the specified handle

In such cases, set the sNODATAPUMPEXPORT registry key to Y on the client and re-submit the job.

Control File Restores Failures Ensure that the DBID is assigned for the instance. Make sure that the DBID value for the database you are restoring is automatically displayed in Instance Properties.
Commandline Restore Failure Verify the availability of the required resource then rerun the RMAN command line operation
Unable to create a Standby Database Standby database fails with the following error message:

temporary file TEMP01.DBF conflicts with file used by target database

Make sure that the Standby Role Initialization parameter, DB_FILE_NAME_CONVERT, is set to add all the temp datafiles from the primary database location to the standby database location, as follows:
DB_FILE_NAME_CONVERT='<primary_database_temp_datafile_old_location>','

<standby_database_temp_datafile_new_location>'

ORA-12571: TNS: packet writer error If ORA-12571: TNS: packet writer failure error is shown in RMAN log while performing Oracle RAC table level restore, add the following line in sqlnet.ora file under ORACLE_HOME\network\admin:

SQLNET.EXPIRE_TIME= 10

Optimizing memory allocation for table restores

When restoring large tables, the restore operation may fail if there is insufficient memory allocation for creating the auxiliary instance.

Use the following steps to optimize the memory allocation for the auxiliary instance:

Allocating Memory for Auxiliary Instance

By default, 16MB pool size is allocated for the auxiliary instance. Use the following steps to increase this size limit:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client>, and then click Properties.
  3. Click the Registry Key Settings tab.
  4. Click Add.
  5. On the Add Registry Key dialog:
  • In the Name box, type sLARGEPOOLSIZE.
  • In the Location box, select iDataAgent.
  • In the Type box, select REG_SZ.
  • For Unix Clients, enter a <value>, for example 32M.
  1. Click OK.

Setting the UNDO Retention Period

Whenever a transaction is committed, the old undo information, is retained by default for a period of 1800 seconds. Modify this value, using the following steps:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client>, and then click Properties.
  3. Click the Registry Key Settings tab.
  4. Click Add.
  5. On the Add Registry Key dialog:
    • In the Name box, type sUNDORETENTIONSIZE.
    • In the Location box, select iDataAgent.
    • In the Type box, select REG_SZ.
    • For Unix Clients, enter a <value>, for example 32M.
  6. Click OK.

Viewing RMAN errors

  • If the system notifies you that there was an "RMAN error" during a backup or restore operation, we advise you to view the RMAN Output Log to identify which error was encountered. To view the RMAN Output Log, do the following:
    1. From the CommCell Browser, right-click the <Instance>, point to View, and then click either Backup History or Restore History.
    2. Click OK.
    3. On the Job History window, right-click the job, and then select View Log File.The RMAN Output Log will display for the job.

    Once you have identified the specific RMAN error, consult one of the following publications from Oracle Corporation for information on resolving the specific error, in conjunction with your on-site Oracle database administrator:

    • Oracle - Backup and Recovery Guide
    • Oracle - Recovery Manager User’s Guide and Reference
    • Oracle - Recovery Manager User’s Guide

CommCell Console errors

  • If only command line backups have been performed, and a Browse Restore operation is subsequently attempted from the CommCell Console without first opening and closing the subclient properties, then the system will display a message indicating that no data was backed up. If this happens, ensure that you open and then close the subclient properties of the Oracle iDataAgent you are trying to restore, then try the Browse Restore operation again.
  • If the line "set linesize 80" is present in the file $ORACLE_HOME/sqlplus/admin/glogin.sql, it may cause the SrvOraAgent server process on the CommServe to fail when browsing database contents or executing a backup. To avoid such failures, comment out that line from the file and re-try the browse or backup operation.
  • For Oracle 10g and higher, if the instance is configured for autobackup with flash recovery, then restoring the SPFile from the CommCell Console will not work. The following work-arounds are available in this situation:
    • Work-around 1:

      Comment out the option "db_recovery_file_dest" from the PFile, then re-create the SPFile using the new PFile. Afterwards, restoring the SPFile from the CommCell Console should work.

    • Work-around 2:

      Restore the SPFile manually using RMAN with the script below:

      RMAN> run {
      2> ALLOCATE CHANNEL c1 DEVICE TYPE DISK;
      3> restore spfile from autobackup;
      4> }

Point-in-time recovery

When you recover a database to a point in time, the RMAN command ALTER DATABASE OPEN RESETLOGS is executed which will reset the SCN (System Change Number) and time stamp on every object of the database (i.e., datafiles and control files). Also, only the archived redo logs that match the RESETLOGS SCN and timestamp value will be applied to the database, thus recovering the database to a time that is not current. This is a very useful operation if the point-in-time to which you are trying to recover is certain and known, but can be counterproductive if you are guessing at the point-in-time.

If you are not sure about the point-in-time for the recovery, it is recommended to restore the data and the control files to a point in time without recovery. This method will allow you to restore the database to a state that you can make the determination whether or not you have achieved the correct point-in-time, without invoking the"ALTER DATABASE OPEN RESETLOGS" statement that would reset SCNs and time stamps on the database objects.

After determining the correct point-in-time through this method, you can recover the database to the point in time to reset your Oracle database to the desired incarnation.

Sample scripts are provided below for your Oracle database administrator to use as reference for developing custom scripts that you can run from the RMAN command line, to perform special operations apart from the CommCell Console.

Sample Script for Resetting a Database after RESETLOGS

The following example resets a database after performing an incomplete media recovery:

run {
allocate channel dev1 type disk;
set until logseq 1234 thread 1;
restore database skip tablespace readonly;
recover database;
sql "ALTER DATABASE OPEN RESETLOGS";
release channel dev1;
}

reset database;

Sample Script for Resetting the Database to an Old Incarnation

The following command makes an old incarnation of database PROD1 current again:

# obtain primary key of old incarnation

list incarnation of database prod1;

List of Database Incarnations
DB Key
------
Inc Key
-------
DB Name
-------
DB ID
-----
CUR
---
Reset SCN
---------
Reset Time
----------
1 2 PROD1 1224038686 NO 1 02-JUL-98
1 582 PROD1 1224038686 YES 59727 10-JUL-98

shutdown immediate;

# reset database to old incarnation

reset database to incarnation 2;

# recover it
run {
allocate channel dev1 type disk;
restore controlfile;
startup mount;
restore database;
recover database;
sql "ALTER DATABASE OPEN RESETLOGS";
release channel dev1;
}

Completed with one or more errors

Restore jobs from Oracle iDataAgent will be displayed as "Completed w/ one or more errors" in the Job History in the following cases:

  • During a table restore, if the export or import of table fails.
  • RMAN recovery is completed, but an incorrect open mode is selected for restore.

Recovering data associated with deleted clients and storage policies

Symptom

In a disaster recovery scenario, use the following procedure to recover data associated with the following entities:

  • Deleted storage policy
  • Deleted client, agent, backup set or instance

Before You Begin

This procedure can be performed when the following are available:

  • You have a Disaster Recovery Backup that contains information on the entity that you are trying to restore. For example, if you wish to recover a storage policy (and the data associated with the storage policy) that was accidentally deleted, you must have a copy of the disaster recovery backup that was performed before deleting the storage policy.
  • Media containing the data you wish to recover is available and not overwritten.
  • If a CommCell Migration license was available in the CommServe when the disaster recovery backup was performed, no additional licenses are required. If not, obtain the following licenses:
    • IP Address Change license
    • CommCell Migration license

    See License Administration for more details.

  • A standby computer, which is used temporarily to build a CommServe.
Recovering Deleted Data
  1. Locate the latest Disaster Recovery Backup that contains the information on the entity (storage policy, client, agent, backup set or instance) you are trying to restore.
    • Check the Phase 1 destination for the DR Set or use Restore by Jobs for CommServe DR Data to restore the data.
    • If the job was pruned and you know the media containing the Disaster Recovery Backup, you can move the media in the Overwrite Protect Media Pool. See Accessing Aged Data for more information. You can then restore the appropriate DR Set associated with the job as described in Restore by Jobs for CommServe DR Data.
    • If the job is pruned and you do not know the media containing the Disaster Recovery Backup, you can do one of the following:
      • If you regularly run and have copies of the Data on Media and Aging Forecast report, you can check them to see if the appropriate media is available.
      • If you do not have an appropriate report, and know the media that contains the DR Backup, catalog the media using Media Explorer. Once the cataloging process is completed, details of the data available in the media are displayed.
  2. On a standby computer, install the CommServe software. For more information on installing the CommServe, see Install the CommServe.
  3. Restore the CommServe database using the CommServe Disaster Recovery Tool from the Disaster Recovery Backup described in Step 1. (See CommServe Disaster Recovery Tool for step-by-step instructions.)
  4. Verify and ensure that the NetApp Client Event Manager NetApp Communications Service (EvMgrS) is running.
  5. If you did not have a CommCell Migration license available in the CommServe when the disaster recovery backup was performed, apply the IP Address Change license and the CommCell Migration license on the standby CommServe. See Activate Licenses for step-by-step instructions.
  6. Export the data associated with the affected clients from the standby CommServe as described in Export Data from the Source CommCell.

    When you start the Command Line Interface to capture data, use the name of the standby CommServe in the -commcell argument.

  7. Import the exported data to the main CommServe as described in Import Data on the Destination CommCell.

    This brings back the entity in the CommServe database and the entity is visible in the CommCell Browser. (Press F5 to refresh the CommCell Browser if the entity is not displayed after a successful merge.)

  8. You can now browse and restore the data from the appropriate entity.

    As a precaution, mark media (tape media) associated with the source CommCell as READ ONLY before performing a data recovery operation in the destination CommCell.

Restore completed with warnings

Restore jobs from Oracle for Oracle RAC iDataAgent will be displayed as "Completed with Warnings" in the Job History in the following case:
  • When the database restore succeeds but it fails when recovering the database or opening the database.

Oracle errors

If you receive an Oracle error during an Oracle restore operation, we recommend that you follow procedures published by Oracle Corporation on resolving the specific error. We also advise you to consult with your on-site Oracle database administrator, as needed.