Backup Troubleshooting - SAP Oracle iDataAgent

Table of Contents

Checklist for General Troubleshooting

In case you encounter errors while performing backups, you can perform the following mentioned general troubleshooting steps to resolve the issue:

  1. Verify that the SAP Oracle iDataAgent instance properties are correct.
  2. Verify the path of SAPEXE file is correct on the client computer.
  3. Verify if the parameter files (init<SID>.ora, init<SID>.sap, init<SID>.utl) have correct information.
  4. Verify the SAP user profile for correctness.
  5. Increase the debug level for the log files are present on the client computer. For steps to set debug level for a log file, see Setting the Debug Level.
    • ClSapAgent.log
    • backint_oracle.log
    • ORASBT.log
  6. The use must have the permission to access the SAPEXE and SAPDATA_HOME directories
  7. SnapProtect group should be associated during the SAP Oracle iDataAgent installation.
  8. SAP BR detail file and Summary file should be present an the following paths:
    • $SAPDATA_HOME/sapbackup
    • $SAPDATA_HOME/saparch
  9. On the Windows platform, verify if the SAP service user has been explicitly assigned full control permissions to the SnapProtect directory and the client registry.
  10. For SAP Oracle clients, include the SAPBACKUP directory when a sendlog file job is initiated.

SAPOR0003: UTIL_FILE_ONLINE backup operation fails on Linux

The UTIL_FILE_ONLINE backup operation hangs on Linux and fails after 30 minutes of wait time.

Symptom

Switch files are created at the default location $ORACLE_HOME/sapbackup and are used for communicating between BRTOOLS and BACKINT.

After the backup is initiated, BRBACKUP could not find the switch files at the default location as BACKINT creates them at a new location $SAPDATA_HOME/sapbackup. This causes a lack of communication between BRTOOLS and BACKINT, hence delaying the backup operation which eventually fails after 30 minutes of waiting.

Resolution

Create a symbolic link from $ORACLE_HOME/sapbackup to $SAPDATA_HOME/sapbackup. This will ensure the expected communication between BRTOOLS and BACKINT to complete the backup operation successfully.

SAPOR0005: BR*Tools do not have the privilege to bring database down for a backup

Symptom 1

The following  error message is displayed when you run an offline backup.

ERROR CODE [18:95]: SAP Oracle Error [SQL error -1033 at location BrDbConnect-2, SQL statement:'CONNECT /' ORA-01033: ORACLE initialization or shutdown in progress Detail File=[/oracle/CER/sapbackup/beqiunem.anf] ].

Resolution

  1. Verify that the instance exists in the CommCell Console, with the following parameters configured.

On the General tab of the Instance Properties dialog box:

  1. In the User Account box, the login credentials to access the SAP Oracle client.

    For UNIX configurations, use <SID>adm for this value.

    For Windows configurations, use <domain_name>\<SID>adm for this value.

  2. In the ORACLE HOME box, the SAP Oracle application install path.

    Use the following procedures to obtain the Oracle home path.

    Expand All

    UNIX

    Log on to the SAP Oracle server with the <SID>adm user.

    On the command line, type the following command:

    su - <SID>adm
    echo $ORACLE_HOME

    Windows

    Log on to the SAP Oracle server with the <domain_name>\<SID>adm user.

    On the command line, type the following command:

    echo %ORACLE_HOME%

  3. In the SAP DATA PATH box, the SAP Oracle data and log file path.

    The SAP server environment variables contain this information.

    Use the following procedures to obtain the SAP data path.

    Expand All

    UNIX

    Log on to the SAP Oracle server with the <SID>adm user.

    On the command line, type the following command:

    su - <SID>adm
    echo $SAPDATA_HOME

    Windows

    Log on to the SAP Oracle server with the <domain_name>\<SID>adm user.

    On the command line, type the following command:

    echo %SAPDATA_HOME%

  4. In the SAP EXE PATH box, the SAP Oracle executable files.

    Use the following procedures to obtain the SAP data path.

    Expand All

    UNIX

    Log on to the SAP Oracle server with the <SID>adm user.

    On the command line, type the following command:

    su - <SID>adm
    echo $SAPEXE

    Windows

    Log on to the SAP Oracle server with the <domain_name>\<SID>adm user.

    On the command line, type the following command:

    echo %SAPEXE%

Symptom 2

The BR*Tools must be able to bring down the SAP Oracle database for a backup.

For BRBACKUP to run offline backup, OPS$external user should have the SYSOPER & SYSDBA privilege.

Resolution

On Windows:

Add the OS Database Administrators with the Local and Domain Administrator user on the "Oracle Administration Assistant for Windows" wizard.

On Unix:

When a DBA allows external authentication for an Oracle database, they must create the corresponding OPS$<username > in the database and then grant dba level privileges.

Run the following query.

Sql> grant dba to ops$<SID>adm;

SAPOR0006: Backups Fail in a Windows Environment with Multiple Oracle Installs or Oracle Instances

Symptom:

The backup fails in Windows environments that have multiple Oracle instances or multiple Oracle installs. For additional information, go to the SAP Oracle Community Network site and search for "SAP Note 556232 - Environment settings for R/3/Oracle on Windows".

Resolution

When you use several databases on the same server, you cannot set the following Oracle variables on the SYSTEM environment. You must set them in the database user environment.

  • LOCAL
  • ORA_NLS33
  • DIR_LIBRARY
  • auth_shadow_upgrade TNS_ADMIN (except with R/3 3.1I or lower or Oracle 10g)
  • TWO_TASK

SAPOR0009: The BRBACKUP command with the -u / Option or SQLPLUS / fails

Symptom:

The BRBACKUP command with the -u / option or SQLPLUS / fails.

Resolution

Windows Configuration

  1. Create a user called OPS$<domain account> if you use a domain account, or create a user called OPS$<local admin> if you use a local account.
  2. Grant the dba privilege to the OPS$<domain account> user if you use a domain account, or the OPS$<local admin>  user if you use a local account.
  3. Shutdown the Oracle database.
  4. If the "os_authent_prefix=OPS$" entry does not exist in the init<SID>.ora file, add the entry and save the file.
  5. Start the Oracle database.
  6. Connect to the Oracle database by using the run sqlplus / command.

UNIX Configuration

  1. Create a user "OPS$ORACLE" that is externally identified.
  2. Grant the dba privilege to the "OPS$ORACLE" user.
  3. Commit the changes.
  4. Connect to the Oracle database by using sqlplus /.

When SQLPLUS / works, the BRBBACKUP command with the -u / option will work.

Backup Failure

From CommCell Console due to Invalid User Permissions Make sure to use the following user accounts when creating SAP Oracle instances:

For Unix clients:

<SID_name>adm

For Windows clients:

<client_name>/<SID_name>adm

On AIX when the CIO option is turned on Issue:

The online backup of an Oracle database does not work with JFS2 and Oracle 10g .

You receive various errors when using the external tools to access a file.

cp: A system call received a parameter that is not valid.
0653-902 Cannot open the specified file for reading.
DBV-00100: Specified FILE not accessible errno(22) A system call received a parameter that is not valid.

Additional Terminology

BR*Tools, brbackup FILESYSTEMIO_OPTIONS cio BKI4008E dbv cp dd jfs2 AIX 5.3 6.1 O_CIO O_CIOR

Reason and Prerequisites

Beginning with Oracle 10g, the Oracle database server uses the advanced I/O capabilities of JFS2. If the file system type detected by the Oracle database is JFS2, the Concurrent IO (CIO) option is used to access all Oracle datafiles. The CIO option results in enhanced Oracle IO performance because it bypasses the AIX system cache.

Set the FILESYSTEMIO_OPTIONS=SETALL initialization parameter in the spfile to use CIO with JFS2 for Oracle 10g / 11g together with ASYNC IO operation. This is the default mode for JFS2 file systems with Oracle 10g.

Datafiles opened in CIO mode by Oracle using O_CIO, are prohibited by the AIX OS from being accessed in the regular file system cache by tools (for example cp). Do not open Oracle datafiles using the CIO option, or the tools receive an error from the AIX operating system when trying to access the datafiles on an open Oracle database. Use the following solutions to prevent backups from failing.

Solution

AIX 5.3 or ( AIX 6.1 with Oracle < 11.2.0.2):

Mount all JFS2 file systems containing Oracle datafiles (sapdata directories) with the mount -o cio option, which forces a bypass of the file system cache. Use the mount -o cio option for Oracle datafiles, including the Oracle redo log that are on JFS2. Access to the Oracle datafiles by all other programs which do not support CIO may be impacted, because caching and read ahead are no longer available.

Reduce the performance impact during the database backup by using the dd command with a large BLOCKSIZE parameter as shown in the initSID sap file example.

tape_copy_cmd = dd
disk_copy_cmd = dd
dd_flags = "bs=1024k"
dd_in_flags = "bs=1024k"
compress_cmd = "dd bs=1024k if=$ | ( compress -c > $ ) 2>&1"

Example restore:

uncompress_cmd = "uncompress -c $ | dd bs=1024k of=$ 2>&1"

Example verify:

uncompress_cmd = "uncompress -c $ > $"

Example "compress-only" (option "-k only"):

compress_cmd = "dd bs=1024k if=$ | compress -c > $

Alternatively, use

  • the BACKINT backup utility which supports CIO.,
  • Oracle RMAN, controlled by BR*Tools.

With Oracle RMAN as backup utility, the mount-option -cio for JFS2 is not required when mounting file systems holding Oracle datafiles.

Do not mount JFS2 file systems holding $ORACLE_HOME and log files with the -o cio option.

>=AIX 6.1 with Oracle >= 11.2.0.2:

The open flag O_CIOR option introduced in AIX 6.1 allows open calls without CIO. Tools (for example cp) can access the database files in read only mode Oracle 11.2.0.2 opens the JFS2 files using the O_CIOR option when it detects AIX 6.1. Do not use the –o cio option to mount the file system. Remove the mount option "-o cio" option on Oracle 11.2.0.2/AIX6.1 to avoid the error below.

cp: A system call received a parameter that is not valid.
0653-902 Cannot open the specified file for reading.
">DBV-00100: Specified FILE not accessible
errno(22) A system call received a parameter that is not valid.

Shared Memory Error Issue:

The backup failed because the shared memory on the HP-UX PA-RISC client has not been configured per operational guidelines.

Resolution:

Add the DisableIPC_GLOBAL file in the /apps/simpana/Base directory on the client where the backup failed.

  1. Stop the SnapProtect software.
  2. Create an empty file called DisableIPC_GLOBAL in the /apps/simpana/Base directory. From the command line, enter the following:

    touch /apps/simpana/Base/DisableIPC_Global

  3. Restart the SnapProtect software.

Backup Fails with Permissions Issue

Issue:

The backup fails due to issues accessing the SnapProtect registry, log files and base directories.

The RMAN backup fails because it cannot load the CommVault SBT Media Management library.

The SAP backint backup fails because the symbolic links are configured and correctly pointing to the SnapProtect backint.

Solution

Run the Database Readiness Check.

Backups from the CommCell Console are successful but third party Command Line backups fail

Verify if the information in the parameter files is correct. On a Windows platform, the Oracle database init<SID> files may be located in $ORACLE_HOME\database folder. On UNIX platforms, the file is $ORACLE_HOME/dbs/init<SID>.utl file.

Fill in the following parameters:

  • CvInstanceName <Name of the instance>
  • CvClientName <name of client>
  • numstreams <number of streams>

To learn more about running Third party command line backups for SAP Oracle iDataAgent, see BR*Tools Backups.

RMAN third party Command Line backups are not running

Before you run backups from the RMAN command line for the SAP Oracle iDataAgent, set the SBT_LIBRARY path and environment variables for CvClientName and CvInstanceName in the RMAN script. For example, on a Solaris client, provide the path as given below:

util_par_file = <ORACLE_HOME>/dbs/init@.utl
rman_parms="BLKSIZE=1048576,SBT_LIBRARY=/opt/simpana/Base64/libobk.so,ENV=(CvClientName=sunsign,CvInstanceName=Instance001)" rman_channels=1

where Cvclientname and CvInstancename are the names of the client and instance (for example, Instance001) where the SAP Oracle DataAgent is installed.

On a Windows client, edit the $ORACLE_HOME\database\init<SID>.sap file and provide the parameter as given below

util_par_file = <ORACLE_HOME>\database\init@.utl
RMAN_PARMS="SBT_LIBRARY=,BLKSIZE=1048576,ENV=(CvClientName=<client>,CvInstanceName=<client_name>)"

where Cvclientname and CvInstancename are the names of the client and instance (for example, Instance001) where the SAP Oracle DataAgent is installed.

The SBT_LIBRARY for the various platforms are listed below:

Platform SBT_LIBRARY
AIX with 64 bit Oracle <Client Agent Install Path>/Base/libobk.a(shr.o)
HP UX PA RISC 64 bit Oracle <Client Agent Install Path>/Base64/libobk.sl
Solaris with 64 bit Oracle <Client Agent Install Path>/Base64/libobk.so
All Other Unix platforms <Client Agent Install Path>/Base/libobk.so

NOTE: The SBT_LIBRARY parameter is not applicable on Windows platforms.

When you use the RMAN utility on Solaris client, set the following parameter on the client computer:

crle -64 -c /var/ld/64/ld.config -l/opt/snapprotect/Base64:/lib/64:/usr/lib/64

When you use RMAN_UTIL after the logs have been reset, use the util_file interface to backup all the archive logs. Subsequent log backups can be performed using the RMAN_util interface.

For more information on the command line interface for the SAP Oracle iDataAgent, see BR*Tools Backups.

Jobs Completed with Warnings

A job is marked as Completed with Warnings [CWW], if Brbackup or Brarchive returns warnings during backups or restores or the database restore succeeds, however the database could not be opened or recovered successfully.

The SAP Oracle backup may complete with a warning. This error may occurred if the SAPSECULIB (libsapsecu.so) library cannot be found or the permissions are not correct. To correct the problem, complete the following steps:

  1. Ensure that the permissions for the br* executables are correct.
  2. Ensure that the SHLIB_PATH and DIR_LIBRARY environment variables are set correctly.
  3. On Unix systems, ensure that the DIR_LIBRARY environment variable is set on the directory that contains the libsapsecu library; for example:

    DIR_LIBRARY = /usr/sap/<SID>/SYS/exe/run

    This is especially important if you use an external scheduler (for example, cron, at, or BACKINT scheduler) to start the BR*Tools. Also, ensure that DIR_LIBRARY is set for both user ora<sid> and <sid>adm.

  4. Ensure that the libsapsecu.so library is accessible.
  5. Ensure that the SAPSYSTEMNAME environment variable is set correctly.
  6. Ensure that the contents of the database (i.e., SAP tables and SAP owner) are accessible. This is especially important if you use an external scheduler (for example, cron, at, or BACKINT scheduler) to start BR*Tools. In such a case, the SAPSYSTEMNAME variable must be set in the environment of this scheduler (for example, in an environment file); in this case, it must be set within the backup profile of SnapProtect (consult your backup vendor).
  7. Verify this setup by appending "-TRC11" to the brbackup command line that is called by SnapProtect.The created trace file will contain all the active environment variables at runtime of the brbackup command.

There could be other scenarios where a job could complete with a warning. For example, if logs were prematurely deleted without backing up, etc. SAP detail file located in the SAPBACKUP/SAPARCH directory should be reviewed for more details.

Oracle Errors

If you receive an Oracle error during a SAP Oracle backup operation, we recommend that you follow procedures published by Oracle Corporation on resolving the specific error. We also advise you to consult with your on-site Oracle database administrator, as needed.

General Performance Tuning and Troubleshooting

If you are experiencing performance issues during a backup, you can troubleshoot them by enabling logging of performance details in the log files. These performance counters contain information that help in resolving the performance related issues during backups.

  1. Perform a client backup to determine the performance statistics. To perform a backup, see Performing a Full Backup.

    You can track the progress of the job from the Job Controller window of the CommCell Console.

    • Right-click the backup job and click Details and verify the Data Transferred on Network.

      For example, if backup job is using 10 streams, make sure to backup at least 200 GB of data. If you are performing backups using 5 streams, make sure to backup at least 100 GB of data.

    • If the backup transfer rate is very slow, then kill the job by right-clicking the backup job and then click Kill.
  2. View log files of backup job to verify performance counters. See View the Log Files of a Job History for step-by-step instructions.

    Verify the following the performance counters tin the log files:

    Total Oracle I/O Time

    Time spent per SBT thread for reading the data from disk.

    Total MA I/O Time

    Time spent during data transfer to MediaAgent i.e., data read from the network buffer and written to the disk.
  3. In the log file verify the above performance counters.

    If the Total Oracle I/O Time value is more than the Total MA I/O Time value then perform the following to improve performance:

    • Verify Oracle application compression. If it is ON turn OFF the compression.
    • Verify NetApp compression. If it is ON turn OFF the compression from instance and storage policy copy level. See Setting Up Data Compression for step-by-step instructions.
    • Depending upon your environment, modify Data Files per BFS (value to 4 or 8) and Max Open Files. See Performance Tuning.

    If the Total Oracle I/O Time value is lesser than the Total MA I/O Time value then perform the following to improve performance:

    • If the write throughput of the disk is slow, run CvDiskPerf tool to measure the throughput for the disk. See Disk Performance Tool for more information.
    • If the data transfer on the network is slow or you have a low bandwidth network environment, then verify Network Throughput by running CvNetworkTestTool tool. If network throughput is low then enable nNumPipelineBuffers registry key to increase the data transfer throughput from the client. See Increasing Data Transfer Throughput From Client for more information.

Performance Tuning for RMAN_UTIL Interface

To enhance the performance for backups using RMAN_UTIL, edit the $ORACLE_HOME/dbs/init<SID>.sap file or on a Windows platform,

$ORACLE_HOME\database\init<SID>.sap file

Modify the values for the following parameters:

  • BLKSIZE
  • Channels
  • filesperset
  • maxopenfiles

For example:

rman_parms="BLKSIZE=1048576
SBT_LIBRARY=<SimpanaDir/Base>/libobk.so,ENV=(CvClientName=<client1>,CvInstanceName=Instance00x)"
rman_channels=3
rman_filesperset=8
rman_maxopenfiles=2

*You do not need to provide the SBT_LIBRARY PATH parameters on Windows platforms. Please refer to SBT_LIBRARY PATHS for various platforms.

SAP BRTOOLS offers two new options for restore which are currently supported only from third party command line.

  1. NFD - No File Delete
  2. NSC -No Space Check

BR0428W File /<SAPDATA_HOME>/sapdata1/sysaux_1/sysaux.data1 will be overwritten

BR0428W File /<SAPDATA_HOME>/sapdata1/system_1/system.data1 will be overwritten

BR0428W File /<SAPDATA_HOME>/sapdata1/sr3usr_1/users.dbf will be overwritten

BR0360E Not enough disk space in /<SAPDATA_HOME>/sapdata4 for restore, missing at least 2240124.067 MB

NFD example:

Brrestore –d util_file –b last –m all –NFD

The NFD option eliminates the need for BRRESTORE to overwrite files during restore.

NSC example:

Brrestore –d util_file –b last –m all –NSC

The NSC option suppresses the out of space error message and instead a warning message will be output, allowing the user to continue.

Backup failed with error “ENV PATH Variable Exceeds the Limit of 1024” in ClSapAgent.log

This is because, the total character length of the exported PATH has more than 1024 characters. To resolve this issue restart the SnapProtect client services.

Job completed with Errors

A job is marked as Completed with Errors [CWE] if some data is backed up or restored and then the job is killed or fails.