Data Aging - DB2 iDataAgent

Table of Contents

Getting Started

Data Aging is the process of removing old data from secondary storage to allow the associated media to be reused for future backups.

By default, all backup data is retained infinitely. However, you should change the retention of your data based on your needs. Note that if you continue to have infinite retention, you will also need infinite storage capacity.

Setting Up the Basic Retention Rule

  1. From the CommCell Browser, expand Policies | Storage Policies | <Storage Policy>.
  2. Right-click the appropriate storage policy copy, and then click the Properties.
  3. In the Copy Properties dialog box, click the Retention tab.
  4. Under Basic Retention Rule for All Backups, click the Retain For.
    • Enter number of days to retain the data.
    • Enter number of cycles to retain the data.
  5. Click OK.
  6. In the Confirm Basic Retention dialog box, click Yes.
  7. On the ribbon in the CommCell Console, click the Reports tab, and then click Forecast.

    Under Reports pane, Data Retention Forecast and Compliance is selected by default.
  8. Click Run.
  9. The report will display the data to be pruned when a data aging job is run.

    To ensure only data intended for aging is actually aged, it is important to identify the data that will be aged based on the retention rules you have configured. Hence, ensure this report includes only the data you intend to age.

    If necessary, fine-tune your rules so that only the intended data is aged.

    Once you run a data aging job, the data will be pruned.

Running the Data Aging Job

  1. From the CommCell Console, right click the CommServe node, point to All Tasks and then click Data Aging.
  2. In the Data Aging Options dialog box, click OK.

  3. You can track the progress of the job from the Job Controller window. When the job is completed, in the Job Controller the Status will be displayed as Completed.

    Make sure that the job completes successfully. If the job did not complete successfully, rerun the job.

Retention Rules for Data Only Backups

If a data backup job does not have a linked log backup, it will be retained based on the retention days only and does not follow the retention cycles. However, if the data backup job is linked to a log backup job, the retention will be based on both the number of days as well as the retention cycles.

Retention Rules for Command Line Full Backups

DB2 command line full backups are pruned based on the number of retention days.

Incremental backups performed through the CommCell Console after the command line full backup are kept, but will be orphaned when the command line full backup is aged. The DB2 history file has a record for the  incremental backups performed through the CommCell Console which depend on the command line full but the database is not recoverable from the incremental.

Command line jobs and incremental backups performed through the CommCell Console must not be combined on the same database because they will not be recoverable.

Extended Retention Rules

Extended retention rules allow you to keep specific full (or synthetic full) backups for an additional period of time.

Extended retention rules can be used in the following circumstances:

Extended retention rules allow you to define three additional "extended" retention periods for full (or synthetic full) backups. For example:

  • You may want to retain your weekly full backups for 30 days.
  • You may want to retain your monthly full backup for 90 days.
  • You may want to retain your yearly full backup for 365 days.

A backup job will be selected for extended retention based on its start time. For example: If a backup job starts at 11:55 pm on August 31st and ends at 1 am on September 1st, then it will be selected as the last full backup for the month of August and will be picked up for extended retention.

In all other cases, we recommend you to use Auxiliary Copy for extended storage as it actually creates another physical copy of the data, thereby reducing the risk of data loss due to media failure.

If you perform online data only full backups from the CommCell Console on a DB2 database that is version 9.5 or above, set the sALLOWONLINESELCOPY additional setting to have these backups be eligible for extended retention rules. For more information, see Allowing a Selective Copy of an Online Full Backup.

Note the following regarding extended retention rules:

  • During a full backup, if the data and logs use the same storage policy, the extended retention rules will apply to both the data and the logs.
  • Extended retention rules can be applied to data only full backup jobs only after the subsequent log backup job is linked to the data backup job.
  • Extended retention rules do not apply for log only backups. However, if the log backup job is linked to a data backup job, and uses the same storage policy used by data backups, then the extended retention rules of the data backup job is also applied to the linked log backup job.
  • If the data and linked logs use different storage policy, the extended retention rules will be applied to the data backups only.

Setting Up the Extended Retention Rules

Use the following steps for setting up the extended retention rules:
  1. Right-click the storage policy copy and click Properties.
  2. Click the Retention tab.
  3. Set the basic retention rules by clicking Retain for and entering the number of days and cycles appropriate for your organization.
  4. Set the extended retention rules as follows:
    1. Click the For button.
    2. Enter the number of Days Total to retain the backup.
    3. Click the Keep drop-down list, and select the desired backup criteria (e.g., Monthly Full, Weekly Full).
    4. Click the Grace Days drop-down list and select the number of days (e.g., 2).

      This allows you to consider the additional number of days along with the Extended Retention rule. For example, if the last full backup job fails with in the defined extended retention criteria, then the next full backup job that ran in the specified grace days will be selected for retention.

  5. Repeat Step 4 to configure additional extended retention.
  6. Click OK.

Data Aging for Transaction, Archive, and Logical Log Backups

Log Backups (transaction, archive, or logical logs) are not considered part of the backup cycle. Therefore, storage policy cycle retention parameters do not apply to them. However, log backups may be linked to data backup operations, which can affect their retention as follows:
  • Log backups are linked to a full backup if they are run at the same time. This is regardless of whether the full backup included data only or data and logs. Such backups follows the standard data aging rules.
  • If a full backup job is run on data, then the next log backup job will be linked to this full backup job.

    These are considered as linked or chained log backups and are not aged until the linked data is aged. In addition, the following is also considered:

    • Logs that need to be copied to secondary copies will not be aged both on primary and non-primary source copy
    • Logs that exist only on one copy will be aged when they are older than the oldest data
    • Logs that exist on multiple copies will be aged according to copy retention days
    • Logs that exist on multiple copies with the longest retention days will be aged when they are older than the oldest data
    • Partial, disabled logs will be aged when they are older than the oldest data
  • If a full backup job is run on data and logs, then the next log backup will not be linked to this full backup job.

    As this is an unlinked log backup, by default, this will follow the unique data aging rules for log backups. If you want such log backups to be aged according to the defined days retention rule for the data, you can do so as follows:

    1. From the CommCell Console, click the Storage tab.
    2. Click the Media Management icon.
    3. Click the Data Aging tab.
    4. In the Prune All Database Agent Logs Only By Days Retention Rule, type 1 to enable pruning for all database agent logs based on the days retention.
    5. Click OK.

Data Aging Rules for Command Line Backups

  • The third party command line log backups can be linked to third party command line data backups as well as any other kind of backup data as per regular data link rule.
  • Data from third-party command line backups is aged differently than data from backups initiated through the CommCell Console. Retention cycles are not used for copies involved in operations from the third-party command line. For such operations, data is aged according to the associated retention time. However, you can manually set the retention time for each third party command line job from the storage policy copy. The command line log backups will be aged according to the retention time set for its associated command line data backup job.
  • The qoperation agedata command can age data and logs simultaneously based on the Job ID, and it is especially useful for aging each of these items separately.
  • Command line full jobs will be aged only when all of the incremental data backups in that cycle are eligible for pruning.

Data Aging Rules for Jobs Completed with Errors

A DB2 backup job will be marked as completed with error even if it fails on log phase. Jobs that are completed with errors are treated as a valid full backup jobs. They are pruned based on basic days retention rules and ignore cycle retention rules. However, extended retention rules will exclude these completed with error jobs and Selective copy.

Data Aging Rules for Selective Copy

You can use selective copy to copy specific full backup jobs from a source copy. The source copy can be either a primary or a synchronous copy. Selective copy facilitates better tape rotation.

Consider the following for DB2:

If you use the SnapProtect feature a FULL job is either an offline full or online full job. If you use the traditional backup, a FULL job is online, offline, partial database full or command line full/

  • A DB2 full backup will be copied to a selective copy if the log job is done after the data job. If this is not the case, the data job is not copied to the selective copy and will have a "Log Link Pending" status.
  • A full backup has a one hour delay before it is copied to the  selective copy.
  • Log jobs are not copied if they use a different storage policy. The log jobs will be retained by the log policy rules.
  • If you perform online data only full backups from the CommCell Console on a DB2 database that is version 9.5 or above, set the sALLOWONLINESELCOPY additional setting to have these backups be eligible for selective copy. For more information, see Allowing a Selective Copy of an Online Full Backup.

Additional details on Selective Copy can be found here.

Pruning a Backup Image

You must enable the DB2 triggered pruning for backup images for SnapProtect using the sCvDB2EnablePruning Additional Setting.

  • Set sCvDB2EnablePruning Additional Setting to y in the registry under <SnapProtect Instance>/Db2Agent
    Example:

    Instance001/Db2Agent

  • Now, you must run the following command on a DB2 Client to prune the backup image:

    $ db2 prune history <timestamp> with force option and delete

    Example:
    [db2inst1@standard ~]$ db2 prune history 20130801192856 with force option and delete
    DB20000I The PRUNE command completed successfully.

    When you prune a backup image which includes data and log files, the backup history will still show this backup job until it meets SnapProtect's retention policy. In such a case, you must restore only the log files during Restore To Disk job.

Advanced Topics

Data Aging - Advanced

Provides comprehensive information on additional Data Aging capabilities.