How Does Index Cache Cleanup Work?

Table of Contents

Overview

Index cache cleanup is an automated, configurable process that you control by setting the Retention Parameters.

Two different types of temporary, working cache file are created during index creation: index cache and intermediate index cache. These temporary files are normally deleted at the point in the indexing process where the final index file is archived. You can suppress deletion of the intermediate cache if you want to remove the need for the archived file to be copied back from a network share to the active cache area for the next run of the indexing process (see Preserving Intermediate Cache).

Cache files are deleted by the Centralized Cleanup Service running on the MediaAgent system. These cleanup operations run on a schedule (see Scheduled Cleanup), but may also be triggered to execute between scheduled runs when certain conditions are met (see Event-Driven Cleanup).

Scheduled Cleanup

Every 24 hours, the Centralized Cleanup Service runs, deleting index files older than the value set for Index retention time in days on the MediaAgent Properties (Catalog) tab. The process removes all index files that have not been accessed within the specified number of days.

The archiving process removes the intermediate cache for the currently running subclient when it archives the index at the end of the backup job.

The scheduled cleanup deletes index cache files for any subclient that does not have active, running jobs, whose index retention time has been reached, including de-configured and hard-deleted subclients. This way, index data is pruned for subclients whose data has been aged and can no longer be browsed.

Example:

The Index retention time in days is set to 60 days. One or more index files are found to be 61 or more days old, so those files are deleted by the Centralized Cleanup Service.

Event-Driven Cleanup

Cleanup is performed between scheduled cleanups in these cases:

  • Index Cache Free Space size falls below the value specified in Free Space Warning. In this case:
    1. An event message appears in the Event Viewer, indicating low disk space.
    2. Cached index files older than the Index retention time in days are deleted immediately.
    3. If the used disk space still exceeds the percentage specified in the Index Cleanup Percent, the MediaAgent removes the least-recently used index files in the cache, until the used disk space becomes equal to or less than the amount specified in the Index Cleanup Percent.
  • Index Cache Free Space size falls below the amount specified in Minimum Free Space. In this case:
    1. The MediaAgent is brought offline.
    2. Cached index files older than the Index retention time in days are deleted immediately.
    3. If the used disk space still exceeds the percentage specified in the Index Cleanup Percent, the MediaAgent removes the least-recently used index files in the cache, until the used disk space becomes equal to or less than the amount specified in the Index Cleanup Percent.
    4. The MediaAgent is brought back online within 20 minutes.

Examples:

  • Index Cache Total Space  = 100 GB out of which 99 GB is used.
  • Free Space Warning = 1024 MB

    The amount of free space falls below 1 gigabyte. The MediaAgent places an event message into the Event Viewer.

  • Minimum Free Space = 100 MB

    The minimum free space is the total space that must be available in the index cache. If free space falls below 100 megabytes, event-driven cleanup runs.

  • Index Retention Time = 15 days
  • While the event-driven cleanup is running, one or more index files are found to be 16 or more days old, and are deleted.

  • Index Cleanup Percent = 90

    When the percentage of used disk space exceeds the Index Cleanup Percent, the MediaAgent removes index files in the index cache older than Index Retention Time until the percentage of used disk space is equal to or less than the Index Cleanup Percent.

    In this example, 9 GB of data are aged to reach the 90 GB threshold dictated by applying the Index Cleanup Percent to the Minimum Free Space specified.

Preserving Intermediate Cache

You can add an Additional Setting to the MediaAgent which tells the backup process not to delete the intermediate cache files after it archives the new index. This is useful to avoid having to copy an index file from the library back to the intermediate index cache area for a new index-creation operation. The downside to using this setting is that index files left in the Intermediate Index Cache area take up space that is most often located on a local disk in the MediaAgent, rather than a network storage area. But preserving the file in the intermediate cache can save time and network bandwidth.

Enabling this feature requires more storage in your caching area, since one such intermediate cache file is retained for each subclient that runs on that MediaAgent.

To add this Additional Setting:

  1. In your CommCell Browser, open MediaAgents, right-click the <MediaAgent> and click Properties.
  2. Click the Additional Settings tab, then click Add.
  3. Into Name, type or paste PreserveIntermediateCacheindex.
  4. For Category, select MediaAgent.
  5. For Type, select BOOLEAN.
  6. For Value, select true.
  7. Enter something in Comments if you want to.
  8. Confirm that Enable is selected, and click OK.
  9. Click OK again to complete the process.

Cache Sizing

For overall cache area sizing guidelines, see Index Cache - Prerequisites. Remember that if you enable intermediate cache file retention, your requirements must increase to enable keeping multiple copies of these intermediate cache files. A fair estimate for this is 200GB, based on 200 streams running concurrently, and each intermediate index file averaging about 1GB.