Data Aging - Oracle RAC iDataAgent

Table of Contents

Getting Started

Data Aging is the process of removing old data from secondary storage to allow the associated media to be reused for future backups.

By default, all backup data is retained infinitely. However, you should change the retention of your data based on your needs. Note that if you continue to have infinite retention, you will also need infinite storage capacity.

Setting Up the Basic Retention Rule

  1. From the CommCell Browser, expand Policies | Storage Policies | <Storage Policy>.
  2. Right-click the appropriate storage policy copy, and then click the Properties.
  3. In the Copy Properties dialog box, click the Retention tab.
  4. Under Basic Retention Rule for All Backups, click the Retain For.
    • Enter number of days to retain the data.
    • Enter number of cycles to retain the data.
  5. Click OK.
  6. In the Confirm Basic Retention dialog box, click Yes.
  7. On the ribbon in the CommCell Console, click the Reports tab, and then click Forecast.

    Under Reports pane, Data Retention Forecast and Compliance is selected by default.
  8. Click Run.
  9. The report will display the data to be pruned when a data aging job is run.

    To ensure only data intended for aging is actually aged, it is important to identify the data that will be aged based on the retention rules you have configured. Hence, ensure this report includes only the data you intend to age.

    If necessary, fine-tune your rules so that only the intended data is aged.

    Once you run a data aging job, the data will be pruned.

Running the Data Aging Job

  1. From the CommCell Console, right click the CommServe node, point to All Tasks and then click Data Aging.
  2. In the Data Aging Options dialog box, click OK.

  3. You can track the progress of the job from the Job Controller window. When the job is completed, in the Job Controller the Status will be displayed as Completed.

    Make sure that the job completes successfully. If the job did not complete successfully, rerun the job.

Extended Retention Rules

Extended retention rules allow you to keep specific full (or synthetic full) backups for an additional period of time.

Extended retention rules can be used in the following circumstances:

Extended retention rules allow you to define three additional "extended" retention periods for full (or synthetic full) backups. For example:

  • You may want to retain your weekly full backups for 30 days.
  • You may want to retain your monthly full backup for 90 days.
  • You may want to retain your yearly full backup for 365 days.

A backup job will be selected for extended retention based on its start time. For example: If a backup job starts at 11:55 pm on August 31st and ends at 1 am on September 1st, then it will be selected as the last full backup for the month of August and will be picked up for extended retention.

In all other cases, we recommend you to use Auxiliary Copy for extended storage as it actually creates another physical copy of the data, thereby reducing the risk of data loss due to media failure.

Setting Up the Extended Retention Rules

Use the following steps for setting up the extended retention rules:
  1. Right-click the storage policy copy and click Properties.
  2. Click the Retention tab.
  3. Set the basic retention rules by clicking Retain for and entering the number of days and cycles appropriate for your organization.
  4. Set the extended retention rules as follows:
    1. Click the For button.
    2. Enter the number of Days Total to retain the backup.
    3. Click the Keep drop-down list, and select the desired backup criteria (e.g., Monthly Full, Weekly Full).
    4. Click the Grace Days drop-down list and select the number of days (e.g., 2).

      This allows you to consider the additional number of days along with the Extended Retention rule. For example, if the last full backup job fails with in the defined extended retention criteria, then the next full backup job that ran in the specified grace days will be selected for retention.

  5. Repeat Step 4 to configure additional extended retention.
  6. Click OK.

Data Aging for Transaction, Archive, and Logical Log Backups

Log Backups (transaction, archive, or logical logs) are not considered part of the backup cycle. Therefore, storage policy cycle retention parameters do not apply to them. However, log backups may be linked to data backup operations, which can affect their retention as follows:
  • Log backups are linked to a full backup if they are run at the same time. This is regardless of whether the full backup included data only or data and logs. Such backups follows the standard data aging rules.
  • If a full backup job is run on data and logs, then the next log backup will not be linked to this full backup job. These are unlinked log backups and by default, this will follow the unique data aging rules for log backups as given below:
    • Logs that need to be copied to secondary copies will not be aged both on primary and non-primary source copy
    • Logs that exist only on one copy will be aged when they are older than the oldest data
    • When logs exist on multiple copies, the logs on the copy with longest retention days will be retained with the data and will be aged after the oldest data. The log jobs on the remaining copies will be aged according to copy retention days without checking if the oldest data exists or not.
    • Partial, disabled logs will be aged when they are older than the oldest data
  • If a full backup job is run on data, then the next log backup job will be linked to this full backup job. These are considered as linked or chained log backups and are not aged until the linked data is aged. In addition, these log backups will also follow the unique data aging rules for log backups.

Pruning All Log Backups by Days Retention Rule

Use the following steps to disable log retention rule and enable unlinked log backups to be aged according to the defined days retention rule for the data. Note that when you define the days retention rule, use the same storage policy or retention criteria used for traditional rman backup, backup copy, and  traditional rman log backup operations.

For example, consider the following subclients:

  • Clone subclient using clone storage policy
  • Snap subclient using the snap storage policy
  • Log subclient using the log storage policy

The clone jobs using backup copy are run daily, snap jobs without backup copy are run every hour, and regular log backup jobs are run every 15 minutes.

When you disable log rule, make sure the retention for log storage policy primary copy is same or higher than the retention set for clone storage policy primary copy. Alternatively, you can also assign the same clone storage policy for archive log backups.

  1. From the CommCell Console, click the Storage tab.
  2. Click the Media Management icon.
  3. Click the Data Aging tab.
  4. In the Prune All Database Agent Logs Only By Days Retention Rule, type 1 to enable pruning for all database agent logs based on the days retention.
  5. Click OK.

Data Aging of the Oracle Recovery Catalog Database

When a Data Aging job is run, the BackupPieceName UNAVAILABLE command is automatically issued to RMAN to disable specific backup pieces in the Oracle Recovery Catalog database that were pruned from the Media Manager CommServe tables. Any backup pieces that were aged from the system's database that have exceeded their retention criteria will be marked as unavailable in the Oracle Recovery Catalog database through this methodology. You can delete these specific backup pieces by creating and enabling the OracleDeleteAgedBackupPiece additional setting.

Timeout for Oracle Crosscheck per Instance during Data Aging

By default the timeout for Oracle CROSSCHECK per instance is 600 seconds during data aging operation. You can modify this value (or disable the option) by using the OraCrossCheckTimeOut additional setting.

Effects of Downed Services

When data aging is running, if the Oracle services go down, the data aging operation will complete successfully. However, you need to manually execute Oracle CROSSCHECK to synchronize the Oracle Recovery Catalog database with that of the CommServe database.

Effects on Oracle Archive Logs

Oracle archive logs get deleted for those clients/instances where the Oracle CROSSCHECK has been completed successfully. However, if the timeout for the Oracle CROSSCHECK is small (between 1 - 300) and if there are many archive logs, then the crosscheck will fail with a timeout error (or any other error). In such cases, the archive logs will get deleted from the CommServe database during the next data aging operation.

Effects of Uninstalling the Software

When uninstalling the iDataAgent software, CROSSCHECK will no longer be performed by the system to synchronize entries in the CommServe Database with the RMAN catalog. If either of these iDataAgents is later re-installed, then the next data aging job will synchronize the RMAN catalog with the CommServe Database unless the data on tape has been deleted (such as the case where the tape/volume was used for other backups and has been pruned).

Data Aging Rules for Oracle Archive Index

Oracle archive index is deleted when the associated backup data is deleted. This applies to SnapProtect Backup and Table Level Backup.

Disable Oracle RMAN Crosschecks during Data Aging

By default, during a data aging operation, an Oracle CROSSCHECK is performed by the system to synchronize the entries in the CommServe database with the RMAN catalog. If required, you can disable this CROSSCHECK operation using the Disable RMAN Cross Check option in the Instance Properties (Details) for the specific Oracle RAC instance. For step-by-step instructions, see Disable RMAN Cross Check.

Data Aging Rules for Selective Online Full Backups

A selective online full operation that consists of archive logs and oracle data can also be linked to the logs of a separate job, which was initiated within the time frame of the selective online full operation. These logs and the selective online full are then considered as one entity within the software, regardless of whether or not separate jobs have the same job ID. Therefore, they are copied to synchronous and selective copies together during auxiliary copy operations and are aged together. If any part of the selective online full is missing from a copy, the full will not be considered as a valid full and will not be counted as a cycle during data aging. Consider the following:

  • Data from selective online full backups are considered the same as regular full and offline full backups for each subclient in terms of basic retention rules of cycles and days. However, if any logs on a primary copy have not been fully copied to a secondary copy, the selective online full cannot be aged.
  • Data from selective online full backups are considered the same as offline full backups for each subclient in terms of extended retention rules of days. Selective online full backups and all logs linked with it must be retained together on the same storage policy copy.
  • Those Logs that are linked with a selective online full (and the logs of the selective online full) can be aged only if they are older than the oldest data that can be aged and the corresponding data of the selective online full that can be or have been aged.
  • Selective online full backup jobs that are completed with errors will not be retained by extended retention rules during data aging operation.

Data Aging Rules for Command Line Backups

  • The third party command line log backups can be linked to third party command line data backups as well as any other kind of backup data as per regular data link rule.
  • Data from third-party command line backups is aged differently than data from backups initiated through the CommCell Console. Retention cycles are not used for copies involved in operations from the third-party command line. For such operations, data is aged according to the associated retention time. However, you can manually set the retention time for each third party command line job from the storage policy copy. The command line log backups will be aged according to the retention time set for its associated command line data backup job.
  • The qoperation agedata command can age data and logs simultaneously based on the Job ID, and it is especially useful for aging each of these items separately.
  • Command line full jobs will be aged only when all of the incremental data backups in that cycle are eligible for pruning.

Data Aging Rules for On-Demand and Customized RMAN Script Backups

Data Aging for Oracle On Demand and Customized RMAN Script backup jobs uses days/time, and ignores cycles, as the determining factor for pruning the data. Therefore, once the retention time criteria has been met, all data (for both data and logs) is pruned that was backed up using the storage policy specified in the RMAN script that was run through the Command Line Interface.

An effective storage policy strategy for Oracle On Demand and Customized RMAN Script backups is as follows:

  • The same storage policy should not be used for regular Oracle backups and Oracle On Demand backups or Customized RMAN Script backups.
  • The storage policy copy containing logs of Oracle On Demand backups or Customized RMAN Script backups should have a much longer retention time than other storage policies used by regular Oracle backups for the same instance. This is to prevent the logs of Oracle On Demand backups from being pruned before the data of regular Oracle backups, and allow the database to be fully restored and recovered using the data of old regular Oracle backups and logs afterwards.

Oracle RMAN Retention Policy

An Oracle RMAN retention policy can be configured for each database. When RMAN retention rules are in effect, RMAN considers the backup jobs comprising data files and control files as obsolete, that is, no longer needed for recovery, according to criteria that you specify in the CONFIGURE RETENTION POLICY command. When you run DELETE OBSOLETE or CROSS CHECK operations, RMAN ages data by freeing disk and tape space used by backups that are no longer needed.

Do not configure RMAN retention policy if you want to retain data using the data aging feature provided in the CommCell console. To disable the RMAN retention policy, use the following command: CONFIGURE RETENTION POLICY TO NONE. This ensures that data will only be aged according to the retention rules specified in the associated storage policy copy.

Oracle Crosschecking and Deleting Aged Backups

You can use the RMAN CROSSCHECK BACKUP command to synchronize the RMAN repository with the records that are on the file system or the disk.

You can use the RMAN DELETE OBSOLETE command to delete all backups associated with the specified retention policy that are no longer needed for recovery. RMAN frees the disk or tape spaces associated with the deleted backup.

Note: Before you run the command, you must allocate a channel for maintenance.

Use the following script as an example.

run {
PARMS="BLKSIZE=524288,SBT_LIBRARY=/opt/simpana/Base/, ENV=(CvInstanceName=Instance001)";
delete obsolete ;

Data Aging Rules for Jobs Completed with Errors

Cycle retention is not honored on selective online jobs that complete with errors.  The job gets pruned after the configured storage policy retention (the Basic Retention Rule for All Backups on the Retention tab of the Storage Policy Properties dialog) expires.

To apply extended retention rules to these jobs, exclude jobs that completed with errors during extended retention calculations. This option is applicable only for Selective Online full backup jobs.

  1. From the CommCell Console, click the Storage tab.
  2. Click the Media Management icon.
  3. Click the Data Aging tab.
  4. Change the value for the Ignore Completed With Errors job option for Extended Retention calculations option from 1 to 0.
  5. Click OK.

Advanced Topics

Data Aging - Advanced

Provides comprehensive information on additional Data Aging capabilities.