Data Aging - MySQL iDataAgent

Table of Contents

Getting Started

Data Aging is the process of removing old data from secondary storage to allow the associated media to be reused for future backups.

By default, all backup data is retained infinitely. However, you should change the retention of your data based on your needs. Note that if you continue to have infinite retention, you will also need infinite storage capacity.

Setting Up the Basic Retention Rule

  1. From the CommCell Browser, expand Policies | Storage Policies | <Storage Policy>.
  2. Right-click the appropriate storage policy copy, and then click the Properties.
  3. In the Copy Properties dialog box, click the Retention tab.
  4. Under Basic Retention Rule for All Backups, click the Retain For.
    • Enter number of days to retain the data.
    • Enter number of cycles to retain the data.
  5. Click OK.
  6. In the Confirm Basic Retention dialog box, click Yes.
  7. On the ribbon in the CommCell Console, click the Reports tab, and then click Forecast.

    Under Reports pane, Data Retention Forecast and Compliance is selected by default.
  8. Click Run.
  9. The report will display the data to be pruned when a data aging job is run.

    To ensure only data intended for aging is actually aged, it is important to identify the data that will be aged based on the retention rules you have configured. Hence, ensure this report includes only the data you intend to age.

    If necessary, fine-tune your rules so that only the intended data is aged.

    Once you run a data aging job, the data will be pruned.

Running the Data Aging Job

  1. From the CommCell Console, right click the CommServe node, point to All Tasks and then click Data Aging.
  2. In the Data Aging Options dialog box, click OK.

  3. You can track the progress of the job from the Job Controller window. When the job is completed, in the Job Controller the Status will be displayed as Completed.

    Make sure that the job completes successfully. If the job did not complete successfully, rerun the job.

Extended Retention Rules

Extended retention rules allow you to keep specific full (or synthetic full) backups for an additional period of time.

Extended retention rules can be used in the following circumstances:

Extended retention rules allow you to define three additional "extended" retention periods for full (or synthetic full) backups. For example:

  • You may want to retain your weekly full backups for 30 days.
  • You may want to retain your monthly full backup for 90 days.
  • You may want to retain your yearly full backup for 365 days.

A backup job will be selected for extended retention based on its start time. For example: If a backup job starts at 11:55 pm on August 31st and ends at 1 am on September 1st, then it will be selected as the last full backup for the month of August and will be picked up for extended retention.

In all other cases, we recommend you to use Auxiliary Copy for extended storage as it actually creates another physical copy of the data, thereby reducing the risk of data loss due to media failure.

If Binary Logging is enabled i.e., if the MySQL server is running with LOG_BIN ON, then only Full Jobs which contain both data and log is applicable for extended retention rule.

If Binary Logging is disabled i.e., if the MySQL server is running with LOG_BIN OFF, then only Full Jobs which contain data only is applicable for extended retention rule.

Extended retention rules can be applied to full backup jobs only. If a database is added and is backed up as part of incremental job, then that job will still be treated as incremental backup job and extended retention rules cannot be applied to this backup job.

Setting Up the Extended Retention Rules

Use the following steps for setting up the extended retention rules:
  1. Right-click the storage policy copy and click Properties.
  2. Click the Retention tab.
  3. Set the basic retention rules by clicking Retain for and entering the number of days and cycles appropriate for your organization.
  4. Set the extended retention rules as follows:
    1. Click the For button.
    2. Enter the number of Days Total to retain the backup.
    3. Click the Keep drop-down list, and select the desired backup criteria (e.g., Monthly Full, Weekly Full).
    4. Click the Grace Days drop-down list and select the number of days (e.g., 2).

      This allows you to consider the additional number of days along with the Extended Retention rule. For example, if the last full backup job fails with in the defined extended retention criteria, then the next full backup job that ran in the specified grace days will be selected for retention.

  5. Repeat Step 4 to configure additional extended retention.
  6. Click OK.

Data Aging for Transaction, Archive, and Logical Log Backups

Log Backups (transaction, archive, or logical logs) are not considered part of the backup cycle. Therefore, storage policy cycle retention parameters do not apply to them. However, log backups may be linked to data backup operations, which can affect their retention as follows:
  • A full backup job containing data and logs will follow the standard data aging rules provided both data and logs use the same storage policy. In case data and logs use different storage policies and if the retention period of log storage policy is less than the data storage policy, then the log backup jobs will age as per data aging rules for log backups. It is recommended to keep the retention periods for data storage policy and log storage policy same.
  • The logs part is linked to data in a full backup job containing data and logs.

    These are considered as linked or chained logs and are not aged until the linked data is aged. In addition, the following is also considered:

    • Logs that need to be copied to secondary copies will not be aged both on primary and non-primary source copy
    • Logs that exist only on one copy will be aged when they are older than the oldest data
    • Logs that exist on multiple copies will be aged according to copy retention days
    • Logs that exist on multiple copies with the longest retention days will be aged when they are older than the oldest data
    • Partial, disabled logs will be aged when they are older than the oldest data
  • A log backup run after a full backup job is not linked to the previous full backup job.

    As this is an unlinked log backup, by default, this will follow the unique data aging rules for log backups. If you want such log backups to be aged according to the defined days retention rule for the data, you can do so as follows:

    1. From the CommCell Console, click the Storage tab.
    2. Click the Media Management icon.
    3. Click the Data Aging tab.
    4. In the Prune All Database Agent Logs Only By Days Retention Rule, type 1 to enable pruning for all database agent logs based on the days retention.
    5. Click OK.