Troubleshooting Restore - Oracle Agent

Table of Contents

Browse Failures

ORCL0022: Point-in-time browse failures

Issue

When encryption is enabled for the client, a point in time table browse operation fails with the following error message:

Pass-phrase protection is on for client [80], but pass-phrase was not specified.

Resolution

Make sure that the pass phrase is exported to the MediaAgent when encryption is enabled for the client.

  1. From the CommCell Browser, right-click the client and select Properties.
  2. Click the Encryption tab.
  3. Click Via Pass-Phrase.
  4. Click Export.
  5. In the Destination Computer box, select the MediaAgent.
  6. In the Pass-Phrase box, type the pass-phrase used for encryption.
  7. In the Re-enter Pass-Phrase box, re-type the pass-phrase to confirm.
  8. Click Export.
  9. Click OK.

Restore Failures

ORCL0023: Table restore failures

Issue

The table restores may fail with the following error message:

RMAN-05502: the target database must be mounted when issuing a DUPLICATE command.

Resolution

For Windows, make sure that the Oracle Services are running as Local System.

Use connect string instead of ‘/’ when you configure an instance. For example, sys/pwd12@orcl4.

ORCL0024: Database restore failures

Issue

A database restore fails when the Redirect, Rename and Recover options were selected at the same time.

Resolution

After performing an Oracle restore operation from the CommCell Console click the Refresh button on the Subclient Properties (Content) tab or run a backup after the restore operation has completed before proceeding with another restore. This is necessary to ensure that the CommCell Console recognizes the changes that were made to the Oracle database and control file, so that it reflects the current structure of the database to be restored, otherwise the restore will fail.

ORCL0025: Unable to create duplicate database

Issue

A duplicate database restore fails with error PLS-00553: character set name is not recognized

Resolution

Make sure that the character sets are the same between the location from where you are running RMAN, and the location of the target database. As this is an Oracle related issue, contact Oracle support for more information.

When you create a duplicate database use a proper connect string instead of ‘/’ for one of the instances. For example, sys/pwd12@orcl4. This is required when you perform a duplicate database restore operation since we need to connect two instances in the same RMAN operation.

ORCL0026: Unable to restart a duplicate database restore job

Issue

The restore job is failing with the following error:

RMAN-00571: ===========================================================
RMAN-00569: =============== ERROR MESSAGE STACK FOLLOWS ===============
RMAN-00571: ===========================================================
RMAN-03002: failure of Duplicate Db command at XX/XX/XXXX 16:28:20
RMAN-05500: the auxiliary database must be not mounted when issuing a DUPLICATE command
RMAN>

Resolution

The Auxiliary database is remaining in mount state when a suspended restore job is resumed.

This issue may occur due to the wrong syntax value (extra Slash (/) value) after Oracle home. You will receive a different status when connected to a database instance, so the duplicate database restore is failing.

Remove the (extra Slash (/) value) after Oracle home in the syntax under instance properties.

For example:

If the Oracle Home was set to the following under instance properties.

/u01/app/oracle/product/11.2.0/dbhome/

You must update the syntax as following:

/u01/app/oracle/product/11.2.0/dbhome

ORCL0027: Job fails due to the sbtio.log size

Issue

Sometimes, jobs fail due to the increase in the size of sbtio.log file in the $UDUMP directory.

Resolution

Set the size limit for the sbtio.log file using the sMAXORASBTIOLOGFILESIZE additional setting. Once the specified size limit is reached, the sbtio.log file gets pruned automatically.

ORCL0028: Table restore intermittent failures

Issue

The table level restore operation may fail intermittently due to an error in the Oracle's DataPump utility and the following error message is displayed:

UDE-00008: operation generated ORACLE error 31623

ORA-31623: a job is not attached to this session via the specified handle

Resolution:

Set the sNODATAPUMPEXPORT additional setting to Y on the client and re-submit the job.

ORCL0033: Restore Error on UNIX when Oracle uses large pages

Issue

When restoring an Oracle database on UNIX clients, if the Switch database mode for restore option is selected to put the database in the correct mode during the restore, the database may not restart after switching the database mode. Also, the restore operation may fail with the following error message.

RMAN Script execution failed with error [RMAN-04014: startup failed: ORA-27137: unable to allocate large pages to create a shared memory segment]. Please check the Logs for more details.

Resolution

This issue occurs if the Oracle user has a higher ulimit configuration than the root user. To resolve this issue, apply the ulimit value of Oracle user for the restore using the following steps:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client>, and then click Properties.
  3. Click Advanced.
  4. Click the Additional Settings tab.
  5. Click Add.
  6. In the Additional Settings dialog box:
    • In the Name box, type OracleUser.
    • In the Category box, select or type OracleAgent from the list.
    • In the Type box, select String.
    • In the Value box, type the Oracle user name (for example, oracle).
  7. Click OK.
  8. Restart the SnapProtect Services on the client.

ORCL0029: Command line restore failures

Issue

Command Line restore fails.

Resolution

Verify the availability of the required resource then rerun the RMAN command line operation.

ORCL0030: Command line backup fails for large backups

Issue

Sometimes, the third party command line jobs may hang when you perform large restores.

Resolution

Sometimes, the third party command line jobs may hang when you perform large backups and restores.
This happens because ClOraControlAgent updates the job manager for every 100MB of data transfer and this causes the thread failure for large backups/ restores after transferring some of the data. The following exception is seen. clOraControlAgent.log:

5710030 304 02/22 03:47:23 608119 OraAgentBase::NotifyCommServeJobContinue() - m_jobObject->setUnCompBytesToAdd(105119744) ...
5710030 304 02/22 03:47:24 608119 CvThread::start_func() - Unhandled exception.
5710030 405 02/22 03:47:37 608119 ClOraControlAgent::OnClientTimeout() - Got timed out while waiting for msg from client 0

Set sBYTESDIFFMBS additional setting <value> in MBs in OracleAgent/.properties. This will update the job manager at every <value> in MBs specified in the key.

ORCL0031: Unable to create a standby database

Issue

The standby database fails with the following error message:

temporary file TEMP01.DBF conflicts with file used by target database

Resolution

Set the Standby Role Initialization parameter, DB_FILE_NAME_CONVERT to add all the temp datafiles from the primary database location to the standby database location, as follows:

DB_FILE_NAME_CONVERT='<primary_database_temp_datafile_old_location>','

<standby_database_temp_datafile_new_location>

ORCL0032: The block size in the backup and restore are different

Issue

The Block Size during the backup and restore are different. The following error is shown.

Error Code: [19:1336]

Description: Oracle Restore [Job[xxxxx] thread[xxxxxxxx]: Did not read correct number of bytes, expected: xxxxxx, read: yyyyyy.]

Resolution

Use the same block size that you have used during the backup. For example, if you have used 128 KB as the block size during the backup, make sure to use the same 128 KB as the block size during the restore of that backup. An increase in the block size during the restore may cause the restores to fail.

ORCL0035: Restore Fails with Permissions Issue

Issue:

The restore fails due to issues accessing the SnapProtect registry, log files and base directories.

The RMAN restore fails because it cannot load the CommVault SBT Media Management library.

Solution

Run the Database Readiness Check.

ORCL0037: Multiple Jobs for Oracle Third Party Command Line Operations

Issue:

For Oracle 12c, when performing Oracle multiple streams for third party command line operations, multiple jobs may be kicked off.

Solution

Add the user to Local security policy

  1. From Local Security Policy, navigate to User Right Assignment.
  2. Right-click Act as part of the operating system and then select Properties.
  3. Click on Add User or Group and then click OK.
  4. Right-click Create a token object and then select Properties.
  5. Click on Add User or Group and then click OK.
  6. Right-click Replace a process level token and then select Properties.
  7. Click on Add User or Group and then click OK.

Recovering data associated with deleted clients and storage policies

Symptom

In a disaster recovery scenario, use the following procedure to recover data associated with the following entities:

  • Deleted storage policy
  • Deleted client, agent, backup set or instance

Before You Begin

This procedure can be performed when the following are available:

  • You have a Disaster Recovery Backup that contains information on the entity that you are trying to restore. For example, if you wish to recover a storage policy (and the data associated with the storage policy) that was accidentally deleted, you must have a copy of the disaster recovery backup that was performed before deleting the storage policy.
  • Media containing the data you wish to recover is available and not overwritten.
  • If a CommCell Migration license was available in the CommServe when the disaster recovery backup was performed, no additional licenses are required. If not, obtain the following licenses:
    • IP Address Change license
    • CommCell Migration license

    See License Administration for more details.

  • A standby computer, which is used temporarily to build a CommServe.
Recovering Deleted Data
  1. Locate the latest Disaster Recovery Backup that contains the information on the entity (storage policy, client, agent, backup set or instance) you are trying to restore.
    • Check the Phase 1 destination for the DR Set or use Restore by Jobs for CommServe DR Data to restore the data.
    • If the job was pruned and you know the media containing the Disaster Recovery Backup, you can move the media in the Overwrite Protect Media Pool. See Accessing Aged Data for more information. You can then restore the appropriate DR Set associated with the job as described in Restore by Jobs for CommServe DR Data.
    • If the job is pruned and you do not know the media containing the Disaster Recovery Backup, you can do one of the following:
      • If you regularly run and have copies of the Data on Media and Aging Forecast report, you can check them to see if the appropriate media is available.
      • If you do not have an appropriate report, and know the media that contains the DR Backup, catalog the media using Media Explorer. Once the cataloging process is completed, details of the data available in the media are displayed.
  2. On a standby computer, install the CommServe software. For more information on installing the CommServe, see Install the CommServe.
  3. Restore the CommServe database using the CommServe Disaster Recovery Tool from the Disaster Recovery Backup described in Step 1. (See CommServe Disaster Recovery Tool for step-by-step instructions.)
  4. Verify and ensure that the NetApp Client Event Manager NetApp Communications Service (EvMgrS) is running.
  5. If you did not have a CommCell Migration license available in the CommServe when the disaster recovery backup was performed, apply the IP Address Change license and the CommCell Migration license on the standby CommServe. See Activate Licenses for step-by-step instructions.
  6. Export the data associated with the affected clients from the standby CommServe as described in Export Data from the Source CommCell.

    When you start the Command Line Interface to capture data, use the name of the standby CommServe in the -commcell argument.

  7. Import the exported data to the main CommServe as described in Import Data on the Destination CommCell.

    This brings back the entity in the CommServe database and the entity is visible in the CommCell Browser. (Press F5 to refresh the CommCell Browser if the entity is not displayed after a successful merge.)

  8. You can now browse and restore the data from the appropriate entity.

    As a precaution, mark media (tape media) associated with the source CommCell as READ ONLY before performing a data recovery operation in the destination CommCell.

Optimizing memory allocation for table restores

When restoring large tables, the restore operation may fail if there is insufficient memory allocation for creating the auxiliary instance.

Use the following steps to optimize the memory allocation for the auxiliary instance:

Allocating Memory for Auxiliary Instance

By default, 16MB pool size is allocated for the auxiliary instance. Use the following steps to increate this size limit:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client> and then click Properties.
  3. Click Advanced.
  4. Click the Additional Settings tab.
  5. Click Add.
  6. On the Add Additional Settings dialog box:
    • In the Category box, select or type OracleAgent from the list.
    • In the Type box, select String.
    • In the Value box, type the memory for the auxiliary instance (for example, 32M).
  7. Click OK.

Allocating Memory for Oracle Streams

By default, the system allocates 48 MB for the Oracle streams. You can modify this value using the following steps:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client> and then click Properties.
  3. Click Advanced.
  4. Click the Additional Settings tab.
  5. Click Add.
  6. On the Add Additional Settings dialog:
  • In the Name box, type sSTREAMSPOOLSIZE.
  • In the Category box, select or type OracleAgent from the list.
  • In the Type box, select String.
  • In the Value box, type the memory for Oracle streams (for example, 96M).
  1. Click OK.

Setting the UNDO Retention Period

Whenever a transaction is committed, the old undo information is retained by default for a period of 1800 seconds. You can modify this value, using the following steps:

  1. From the CommCell Browser, navigate to Client Computers.
  2. Right-click the <Client> and then click Properties.
  3. Click Advanced.
  4. Click the Additional Settings tab.
  5. Click Add.
  6. On the Add Additional Settings dialog:
  • In the Name box, type sUNDORETENTIONSIZE.
  • In the Category box, select or type OracleAgent from the list.
  • In the Type box, select String.
  • In the Value box, type the retention period (for example 2200 seconds).
  1. Click OK.

Viewing RMAN errors

  • If the system notifies you that there was an "RMAN error" during a backup or restore operation, we advise you to view the RMAN Output Log to identify which error was encountered. To view the RMAN Output Log, do the following:
    1. From the CommCell Browser, right-click the <Instance>, point to View and then click either Backup History or Restore History.
    2. Click OK.
    3. On the Job History window, right-click the job and then select View Log File. The RMAN Output Log will display for the job.

    Once you have identified the specific RMAN error, consult one of the following publications from Oracle Corporation for information on resolving the specific error, in conjunction with your on-site Oracle database administrator:

    • Oracle - Backup and Recovery Guide
    • Oracle - Recovery Manager User’s Guide and Reference
    • Oracle - Recovery Manager User’s Guide

CommCell Console errors

  • If only command line backups have been performed, and a Browse Restore operation is subsequently attempted from the CommCell Console without first opening and closing the subclient properties, then the system will display a message indicating that no data was backed up. If this happens, ensure that you open and then close the subclient properties of the Oracle Agent you are trying to restore, then try the Browse Restore operation again.
  • If the line "set linesize 80" is present in the file $ORACLE_HOME/sqlplus/admin/glogin.sql, it may cause the SrvOraAgent server process on the CommServe computer to fail when browsing database contents or executing a backup. To avoid such failures, comment out that line from the file and re-try the browse or backup operation.
  • For Oracle 10g and higher, if the instance is configured for autobackup with flash recovery, then restoring the SP File from the CommCell Console will not work. The following work-arounds are available in this situation:
    • Work-around 1:

      Comment out the option "db_recovery_file_dest" from the PFile, then re-create the SPFile using the new PFile. Afterwards, restoring the SP file from the CommCell Console should work.

    • Work-around 2:

      Restore the SP file manually using RMAN with the script below:

      RMAN> run {
      2> ALLOCATE CHANNEL c1 DEVICE TYPE DISK;
      3> restore spfile from autobackup;
      4> }

Point-in-time recovery

When you recover a database to a point in time, the RMAN command ALTER DATABASE OPEN RESETLOGS is executed which will reset the SCN (System Change Number) and time stamp on every object of the database (datafiles and control files). Also, only the archived redo logs that match the RESETLOGS SCN and timestamp value is applied to the database, thus recovering the database to a time that is not current. This is a very useful operation if the point-in-time to which you are trying to recover is certain and known, but can be counterproductive if you are guessing at the point-in-time.

If you are not sure about the point-in-time for the recovery, it is recommended to restore the data and the control files to a point in time without recovery. This method will allow you to restore the database to a state that you can make the determination whether or not you have achieved the correct point-in-time, without invoking the "ALTER DATABASE OPEN RESETLOGS" statement that would reset SCNs and time stamps on the database objects.

After determining the correct point-in-time through this method, you can recover the database to the point in time to reset your Oracle database to the desired incarnation.

Sample scripts are provided below for your Oracle database administrator to use as reference for developing custom scripts that you can run from the RMAN command line, to perform special operations apart from the CommCell Console.

Sample Script for Resetting a Database after RESETLOGS

The following example resets a database after performing an incomplete media recovery:

run {
allocate channel dev1 type disk;
set until logseq 1234 thread 1;
restore database skip tablespace readonly;
recover database;
sql "ALTER DATABASE OPEN RESETLOGS";
release channel dev1;
}

reset database;

Sample Script for Resetting the Database to an Old Incarnation

The following command makes an old incarnation of database PROD1 current again:

# obtain primary key of old incarnation

list incarnation of database prod1;

List of Database Incarnations
DB Key
------
Inc Key
-------
DB Name
-------
DB ID
-----
CUR
---
Reset SCN
---------
Reset Time
----------
1 2 PROD1 1224038686 NO 1 02-JUL-98
1 582 PROD1 1224038686 YES 59727 10-JUL-98

shutdown immediate;

# reset database to old incarnation

reset database to incarnation 2;

# recover it
run {
allocate channel dev1 type disk;
restore controlfile;
startup mount;
restore database;
recover database;
sql "ALTER DATABASE OPEN RESETLOGS";
release channel dev1;
}

Completed with One or More Errors

Restore jobs from Oracle Agent are displayed as "Completed w/ one or more errors" in the Job History in the following cases:

  • During a table restore, if the export or import of table fails.
  • RMAN recovery is completed, but an incorrect open mode is selected for restore.

Restore Completed with Warnings

Restore jobs from Oracle for Oracle Agent are displayed as "Completed with Warnings" in the Job History in the following case:
  • When the database restore succeeds but it fails when recovering the database or opening the database.

Oracle Errors

If you receive an Oracle error during an Oracle restore operation, we recommend that you follow procedures published by Oracle Corporation on resolving the specific error.

For example, Restoring an RMAN Backup to Another Node [ID 73974.1].

We also advise you to consult with your on-site Oracle database administrator, as needed.