Index: Index Directory Cleanup
Cleanup is the process of removing unneeded data from the Index Directory. The SnapProtect software provides an automated cleanup process, and a report that lists files not deleted during normal cleanup, with a reason. Values referenced in the descriptions on this page, such as Free Space, are found on the MediaAgent Properties dialog box, the Catalog tab.
Note: The process described on this page only applies to Indexing V1. For a description of Index Directory cleanup in Indexing V2, see Version 2 Index Cleanup.
Index Directory Cleanup is an automated, configurable process that is controlled by Retention Parameters.
Indexes in the Index Directory are deleted by the Centralized Cleanup Service that runs on the MediaAgent system. Cleanups run daily (see Scheduled Cleanup), but may also run between scheduled cleanups when certain conditions are met (see Event-Driven Cleanup).
The scheduled cleanup deletes indexes for any subclient that does not have active, running jobs, including de-configured and hard-deleted subclients. This way, index data is pruned for subclients whose data has been aged (see Data Aging: Overview) and can no longer be browsed.
Retain index for n days is set to 60 days. The Centralized Cleanup Service finds indexes 61 or more days old, and deletes them.
Cleanup is performed between scheduled cleanups in two cases:
- Free Space size falls below the Generate alerts when free space falls below value (default: 50GB).
In this case, the cleanup process:
- Places an event message in the Event Viewer that indicates a low disk space condition.
- Begins deleting indexes older than the Retain index for n days setting (default: 15 days), until Clean up until free space is at least n percent is reached (default: 15 percent).
- In the event that, after removing the old indexes, the used disk space still exceeds Clean up until free space is at least n percent, the process continues by removing the least-recently-used indexes in the directory, until the Free Space percent becomes equal to or greater than the Clean up until free space is at least n percent setting.
- Free Space size falls below the value in Take the Index offline when free space falls below n GB (default: 10GB).
In this case, the cleanup process:
- Brings the MediaAgent offline.
- Deletes indexes older than the Retain index for n days setting (default: 15 days).
- If the used disk space still exceeds the percentage specified by Clean up until free space is at least n percent (default: 15 percent), the process removes the least-recently-used index files in the directory, until the Free Space becomes equal to or greater than the Clean up until free space is at least n percent setting.
- Within 20 minutes of the MediaAgent being taken offline, if cleanup activities have brought the disk utilization to or above the free space percent setting, the MediaAgent is brought back online. If after 20 minutes the disk utilization percent setting still has not been met, the MediaAgent is left offline, and administrative staff must manually delete data from the Index Directory (see Correcting Cleanup Issues), then bring the MediaAgent back online manually (see Enable or Disable a MediaAgent), or wait for the next run of the scheduled job to bring it back online.
- Total Size is 300 GB
- Retain index for n days = 15 days
- Generate alerts when free space falls below is set to 50.0 GB
- Take the Index offline when free space falls below n GB is set to 10.0 GB
- Clean up until free space is at least n percent is set to 15 percent
- Indexes are found to be old
While an event-driven cleanup is running, some index files are found to be 16 or more days old, and are deleted.
- Free Space indicates 18 GB (6.00%)
Because the amount of free space is less than 50 gigabytes, but more than 10 gigabytes, the MediaAgent places an event message into the Event Viewer and, if alerts are configured, sends an alert to the administrator. An event-driven cleanup begins. It ends when Free Space is 15 percent or more.
- Free Space indicates 8 GB (2.67%)
Because the amount of free space is less than 10 gigabytes, the MediaAgent goes offline and places an event message into the Event Viewer. If alerts are configured, the MediaAgent sends an alert to the administrator. An event-driven cleanup begins. Within 20 minutes, if the Free Space percent is 15 percent or more, the MediaAgent comes back online. If Free Space is still less than 15 percent, the MediaAgent stays offline, and administrative staff should perform a manual cleanup, then bring the MediaAgent back online when ready.
- Free Space percent shows 9.00 percent (273 gigabytes used)
The percentage of available disk space has fallen below Clean up until free space is at least n percent, so the MediaAgent begins deleting index files that are older than the Retain index for n days setting, until the percentage of available disk space is equal to or more than the Clean up until setting. If this is not sufficient, it continues by deleting least-recently-used indexes until enough space is recovered to meet the free space percentage requirement.
In this example, at least 18 GB worth of indexes are aged out in order to cross below the 255 GB threshold dictated by the Clean up until free space is at least n percent setting.
- Indexes are found to be old
For Index Directory sizing guidelines, see the Index Cache Volume entry on the CommCell Sizing for MediaAgent pages, beginning at Hardware Specifications for MediaAgent.
If cleanup operations driven by the retention settings do not remove enough index data to satisfy the free space requirement (the Clean up until free space is at least n percent setting), you have these options:
- Manually delete additional data from the Index Directory, using Windows Explorer or the command line.
- Change the retention parameters to more accurately reflect your actual disk space usage and requirements.
- Increase the disk available to the Index Directory or move it to another disk (see Moving the Index Directory).
- From the CommCell Browser, expand Storage Resources > MediaAgents.
- Right-click the MediaAgent_name > Properties > Catalog. The current status of the Index Directory is shown.
- Change the retention parameters settings if needed.
- Click OK.
Index files are subject to cleanup every 24 hours as described in Scheduled Cleanup. When this process runs, some files that would be selected for deletion based on retention settings may not actually get deleted.
The Index Directory Cleanup Report is a log file in CSV format that contains a list of index files that were retained during a cleanup operation, when retention criteria would normally have selected them to be deleted. The entry for each file indicates why that file was not deleted.
This report is useful in these cases:
- When an aged index did not get pruned or deleted during cleanup.
For example: A job may have locked the index or the index may have been corrupted.
- To analyze the number of index files present in the MediaAgent computer.
For example: The presence of too many index files may require more aggressive cleanup activity.
This report is created on the MediaAgent computer, in this directory:
The Index Cache Cleanup Report is provided in CSV format. Its entries look like this:
CV_Index\2\1\1427638248,234881274,Index was accessed recently.
CV_Index\2\1\1427724761,234881274,Index was accessed recently.
CV_Index\2\1\1427811348,234881274,Index is in use by a running job.
CV_Index\2\1\1427897725,234881274,Index was accessed recently.
CV_Index\2\42\1427995897,234881274,Index was accessed recently.
CV_Index\2\1\1428070358,234881274,Failed to verify state of index.
When opened in a spreadsheet application such as Microsoft Excel, the file is interpreted this way:
The report contains the three columns listed here.
|Index Directory folder||The location of an index file that was not pruned during the automatic cleanup operation.|
|EventID||A numeric identifier that corresponds to the event message in the Retention Reason column.|
|Retention Reason||Why the index file was not deleted.|
These are some common Event IDs, which uniquely identify the retention reasons shown:
|Event ID||Retention Reason|
|234881273||Index is in use by a running job.|
|234881274||Index was accessed recently.|
|234881275||Index is in use.|
|234881276||Failed to open index, check IndexCacheCleanup.log for more details.|
|234881277||Failed to delete index, check IndexCacheCleanup.log for more details.|
|234881278||Failed to verify state of index.|
By default, the Index Cache Cleanup Report is automatically generated. To disable generation of the Index Cache Cleanup report, add the setting to the MediaAgent by following the steps given in Add or Modify an Additional Setting. Use these values:
- Name: disableIndexCacheCleanupReport
- Category: MediaAgent
- Type: INTEGER
- Value: 1 to disable the report, or 0 to enable it.