Job Controller - Advanced

Table of Contents

Understanding the Job Controller

Information about a job is continually updated and available in the Job Controller window. When a job is finished, the job stays in the Job Controller for five minutes.

Columns in the Job Controller

Customize or add additional columns by right-clicking the menu for any column in the Job Controller window. To view the list of columns available in the Job Controller window, see Job Controller Columns.

Enabling Real Time Information for Active Jobs

The Pause and Play buttons allow you to control how the Job Controller displays real time information from active jobs. The Pause button stops the Job Controller from displaying real time information collected from jobs. The Play button allows the Job Controller to display real time job updates.

To see all the columns in the Job Controller window, use the scroll bar at the bottom of the window.

Use the following steps to view the real time job updates from Job Controller window. This is a default setting:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. In the bottom left of the Job Controller window, the Play button is selected and displays the real time job updates.

Activating Flags for a Job

The Flags column is located on the left-hand side of the Job Controller window and displays an icon for any running high-priority job with a priority level of less than 100, and if the jobs encounters a specific situation like when media cannot be found or if a job has not been updated for over an hour. If neither of such scenarios are present, the Flags column will remain empty.

In order to activate this flag, the JobHighPriorityMarkEnable global parameter must be configured in the GXGlobalParam table with a value of 1. When this entry is present, all jobs with a priority of less than 100 will be given a flag in the Job Controller.

To change the default priority for which flags will be shown, the JobHighPriorityMark entry can be added and configured with the desired priority level. Note that the JobHighPriorityMarkEnable global parameter must still be present and configured.

Use the following steps to view the Error Codes of a job which has not completed successfully:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click any column heading and click Choose Columns | Flags.

    The Flags column will be displayed.

Viewing Job Information

Viewing Job Details

Additional details for a particular job can be viewed from the Job Controller window by right-clicking the job and then clicking Detail. You can view the following job details in the Job Details dialog box:

Job Details Description
General Displays the general information about the selected job, such as the subclient, storage policy, etc.
Progress Displays more specific statistical information about the current phase of the selected job. If a job does not complete successfully, an error code appears in the Reason for job delay box. See Viewing the Error Codes for a Job for additional information.
Streams Displays the data transferred by stream on the MediaAgent the job is using.
Attempts Provides information on each attempt of each phase of the selected job, such as the status of each phase of the job. Each phase has a corresponding client log that can aid in troubleshooting data protection problems.

Note that the Data Size/Transferred field amount includes metadata, and therefore, will be larger than the actual size of the backed up data.

Job Retention Provides the retention information for the data protection job's storage policy. The associated storage policy copies will be listed with their defined retention rules. From here, you can quickly identify whether the storage policy copies are defined with basic or extended retention rules, and the dates until which the data will be retained for each storage policy copy.
Events Displays the events about the selected job from All Found Events window.

For more information about events, see the Event Viewer.

Additional Details Displays the additional details about the job such as Compression details, client status, etc.

Viewing the Error Codes for a Job

If a job has not completed successfully, the Error Code column will display a unique code linking to available troubleshooting and knowledgebase article(s) relevant to that error from the customer support website. These articles may include special considerations for the type(s) of job(s) you are running, suggested workarounds for issues, and common causes for that particular error.

If an error code pertains to more than one issue, the customer support website will display links to all articles for which the code is relevant. Conversely, if an error code does not have any articles associated with it, the customer support website will display a message indicating that no articles exist for that code.

Note that jobs which fail Data Integrity Validation will be moved to pending status. Review the error code and description of the pending job from the Job Controller to identify the reason for failure. See Handling Data Integrity Validation Errors for troubleshooting Data Integrity Validation errors.

Use the following steps to view the error codes of a job which has not completed successfully:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. In the Job Controller window, right-click any column heading and click Choose Columns | Error Code.

    The Error Code column is displayed in the Job Controller window.

  3. For a failed job, click on the error code in the Error Code column to launch the Customer Support website.

  4. Enter your credentials for the Customer Support website.
  5. Review the troubleshooting articles related to the error code.

    The Customer Support website displays links to all relevant articles. If an error code does not have articles associated with it, the customer support website displays a message indicating that.

Viewing Job Status

The current status of the job can be determined from the job status. The following job status levels may appear in the Job Controller window for different jobs:

Job Status Description
Completed The job has completed successfully. Note that pop-up messages for reporting job completion can be enabled or disabled using the F12 key.
Completed With Warning The job has completed successfully but with a notification to the user.
Completed With One or More Errors The job has completed with errors.

The following administration conditions will result in the Completed With One or More Errors status level.

  • Disaster Recovery Backup

    During the operation, Phase 1 failed and Phase 2 completed, or Phase 1 completed and Phase 2 failed.

  • Data Aging

    During the operation, one or more components failed, e.g., subclients failed to be aged or job history failed to be removed.

  • Install Updates

    During the operation, one or more clients failed to be updated.

  • Offline Content Indexing

    During the offline content indexing operation, one or more backup data failed to be content indexed.

  • Information Management

    During an information management operation, if the operation defined in the Automated Content Classification Policy is partially successful.

The following iDataAgent-specific conditions will result in the Completed With One or More Errors status level.

  • Microsoft Exchange Server

    During a backup operation of a storage group assigned to a subclient, one or more databases failed to be backed up.

    During a restore operation, one or more databases failed to be restored.

  • Informix

    During a backup operation, one or more files failed to be backed up.

  • Oracle, Oracle RAC

    During a backup operation, one or more files failed to be backed up.

  • SAP for Oracle, SAP for MAXDB

    During a backup operation, one or more files failed to be backed up.

  • SharePoint Server iDataAgent

    During a backup operation, one or more elements in the subclient content failed to be backed up.

    During a restore operation, one or more elements in the subclient content failed to be restored.

  • Sybase

    During a backup operation, one or more files failed to be backed up.

  • UNIX File System

    During a backup operation, one or more files failed to be backed up.

Dangling Cleanup A job phase was terminated by the job manager, and the job manager is waiting for the completion of associated processes before killing the job phase.
Failed The job failed due to errors or the job was terminated by the job manager.
Interrupt Pending The job manager is waiting for the completion of associated processes before interrupting the job due to resource contention with jobs that have a higher priority, etc.
Kill Pending The job was terminated by the user using the Kill option, and the job manager is waiting for the completion of associated processes before killing the job.
Killed The job is terminated by the user using the Kill option or by the Job Manager.
Pending The Job Manager suspended the job due to phase failure and will restart it without user intervention.
Queued The job conflicted with other currently running jobs (such as multiple data protection operations for the same subclient), and the Queue jobs if other conflicting jobs are active option was enabled from the General tab of the Job Management dialog box. The Job Manager will automatically resume the job only if the condition that caused the job to queue has cleared.

The activity control for the job type is disabled, and the Queue jobs if activity is disabled option was enabled from the General tab of the Job Management dialog box. The Job Manager will automatically resume the job only if the condition that caused the job to queue has cleared.

The Queue Scheduled Jobs option was enabled from the General tab of the Job Management dialog box. Scheduled Jobs can be resumed manually using the Resume option or resumed automatically by disabling the Queue Scheduled Jobs option.

The job started within the operation window's start and end time.

The running job conflicted with the operation window and the Allow running jobs to complete pass the operation window option was not enabled from the General tab of the Job Management dialog box. (This is only applicable for jobs that can be restarted. See Restarting Jobs for more information.)

Running The job is active and has access to the resources it needs.
Running (Cannot be verified) During a running operation, the Job Alive Check failed. See Job Alive Check Interval for more information.
Suspend Pending A job is suspended by a user using the Suspend option, and the Job Manager is waiting for the completion of associated processes before stopping the job.
Suspended A running, waiting or pending job was manually stopped by a user using the Suspend option. The job will not complete until it is restarted using the Resume option.

A job was started in a suspended state using the Start Suspended or Startup in Suspended State options available from the dialog box of the job that was initiated. Restore jobs from Search Console can be started in the suspended state using the Start End User restores in suspended state and Start Compliance restores in suspended state options in the Browse/Recover Option Dialog box in the Control Panel.

System Kill Pending The job has been terminated by the Job Manager, and the Job Manager is waiting for the completion of associated processes before killing the job.
Waiting The job is active but waiting for resources (e.g., media or drive) to become available or for internal processes to start.
Destination Client The restore client machine name. This allows users to verify if the restore data is being written to the correct machine/target.

Changes on Job Status

The status of a job and the preemptibility of the phase of the job in the Job Controller determines the actions (Kill, Suspend, or Resume) that you can perform. Jobs that are pending or have failed, will be killed after being in that state for more than 24 hours. The following table describes the status of a job after an action has been performed on it:

Original Status Actions Available New Status
Running Suspend

Kill

Suspended

Killed

Waiting Suspend

Kill

Suspended

Killed

Interrupt Pending N/A N/A
Pending Suspend

Resume

Kill

Suspended

Returns to original state, resources and other conditions permitting

Killed

Suspend Pending N/A N/A
Queued Suspend

Resume (Scheduled jobs only)

Kill

Suspended

Changes into a state of an active job, resources and other conditions permitting

Killed

Suspended Resume

Kill

Returns to original state, resources and other conditions permitting

Changes into a state of an active job, resources and other conditions permitting

Kill Pending Force Kill Killed
Dangling Cleanup N/A N/A

Viewing Log Files

You can view log files of an active job from the CommCell Console. You can also send the log files by page to an email recipient.

Use the following steps to view and send the log files of an active job in the Job Controller:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click any job and click View Logs.
  3. In the Log File for Job dialog box, click File and then click Send Page.
  4. In the SMTP Parameters dialog box, entering the email recipient's SMTP Parameters.
  5. Click OK.

Viewing Events

You can view the events occurring within a CommCell from the Job Controller. By default, an event is automatically created in the Event Viewer for all the jobs and operations running in a CommCell.

Use the following steps to view the events of an active job in the Job Controller:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.
  2. Right-click any running job and click View Events.

    All the events associated with the selected job will be displayed.

Controlling Jobs

 In the Job Controller window, you can perform a control action on all of the jobs or on a specific job.

Suspending a Job

Suspending a job will temporarily stop the job. A suspended job is not terminated, so it can be resumed. When you suspend a job, a dialog box appears so you can provide a reason for suspending the job. The reason appears in the Description field of the Job Details dialog box.

Only preemptible jobs can be suspended. See Preempting Jobs for information on which jobs are preemptible.

Use the following steps to suspend a running job:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click the job and click Suspend.
  3. Provide the reason for suspending the job in the Warning dialog box and click Yes.

Resuming a Job

Resume a job to return the job to Waiting, Pending, Queued, or Running status depending on the availability of resources or the state of the operation windows and activity control settings.

Use the following steps to resume a suspended job:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click the job and click Resume.

Killing a Job

Jobs can be terminated by a user or by the Job Manager.

Terminating an Active Job

The Job Manager will terminate a job if the number of job retries or the total running time has exceeded the values set in the Job Retry dialog box or if free space in the CommServe installation directory is less than 25 MB. Users can manually terminate jobs. Terminating an active job by a user does not terminate a Pre or Post process initiated by that job.

Use the following steps to terminate the job:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click the job and click Kill.
  3. Provide the reason for killing the job in the Warning dialog box and click Yes.

Terminating a "Kill Pending" Backup Job

When a backup job is terminated by a user using the Kill option, the job manager waits for the associated processes to gracefully end before terminating the job. By default, the job manager waits for a maximum of 30 minutes for the associated processes to end, after which the job is forcefully terminated. To terminate a job immediately, a user can forcefully stop the processes associated with a backup job if it is in Kill Pending status.

Use the following steps to manually terminate a backup job with Kill Pending status:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click the backup job with the Kill Pending status and click Force Kill.

Automatically Terminating Pending Backup Jobs When the Client Is Unreachable

A backup job goes into a pending status if the client is unreachable, for example, the client services are stopped. To have the Job Manager automatically kill pending backup jobs for unreachable clients, add the JMMaxAllowedClientOfflineTimeDesktop additional setting. If your client is configured as a laptop client, see Automatically Terminating Pending Backup Jobs When the Laptop Client Is Unreachable.

  1. From the CommCell Browser, navigate to <CommServe node>.
  2. Right-click <CommServe node> and then click Properties.
  3. In the CommCell Properties dialog box, click the Additional Settings tab and then click Add.
  4. In the Add Additional Settings dialog box, type JMMaxAllowedClientOfflineTimeDesktop in the Name field.
  5. Select CommServDB.GxGlobalParam from the Category box.
  6. Select INTEGER from the Type box.
  7. In the Value box, enter the number of minutes the job should remain in a pending status.

    For example, enter "90" if the backup job should remain in a pending status for 90 minutes before the Job Manager kills it.

  8. Click OK.
  9. Go to the Process Manager on the CommServe computer and restart the CommVault Job Manager service.

    For details on restarting services for your operating system, see Services - Advanced.

Automatically Terminating Pending Backup Jobs When the Laptop Client Is Unreachable

A backup job goes into a pending status if the laptop client is unreachable, for example, the client services are stopped. To have the Job Manager automatically kill pending backup jobs for unreachable laptop clients, add the JMMaxAllowedClientOfflineTimeLaptop additional setting. If your client is not a laptop client, see Automatically Terminating Pending Backup Jobs When the Client Is Unreachable.

  1. From the CommCell Browser, navigate to <CommServe node>.
  2. Right-click <CommServe node> and then click Properties.
  3. In the CommCell Properties dialog box, click the Additional Settings tab and then click Add.
  4. In the Add Additional Settings dialog box, type JMMaxAllowedClientOfflineTimeLaptop in the Name field.
  5. Select CommServDB.GxGlobalParam from the Category box.
  6. Select INTEGER from the Type box.
  7. In the Value box, enter the number of minutes the job should remain in a pending status.

    For example, enter "90" if the backup job should remain in a pending status for 90 minutes before the Job Manager kills it.

  8. Click OK.
  9. Go to the Process Manager on the CommServe computer and restart the CommVault Job Manager service.

    For details on restarting services for your operating system, see Services - Advanced.

Automatically Terminating a Conflicting Job

On file system agents, the Job Manager can automatically terminate conflicting backup jobs if the JMKillPreviousBackupJobForSameSubclient additional setting is added at the CommServe level. If a new backup job is initiated for the same subclient as a job that is currently running, it is considered a conflicting job.

The Job Manager terminates an existing job if the new backup job encompasses the earlier job and if the earlier job has not transferred any data to the media. The following rules apply to this process:

  • A new full backup job can only kill existing incremental and differential jobs.
  • A new differential backup job can only kill an existing incremental job.
  • A new incremental, differential, or full job cannot kill an existing full job.

Resubmitting a Job

You can resubmit a job from the Job Controller window. This is useful if a job has failed, and you want to run it again with the same options. Once you resubmit the job, you can edit the schedule and the job options. The job options vary according to the type of job. For example, if it is a backup job, the job options are the type of backup: full, incremental, differential, or synthetic full.

Resubmitting jobs can only be executed for jobs that have run using the current release of this software.

Use the following steps to resubmit a backup job as an example:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click the selected backup job and click Resubmit Job.
  3. From the Backup Options dialog box, select the backup type to run.
  4. Under the Job Initiation area, select if you want to resubmit the job immediately or schedule it.
  5.  Click Advanced to select the job options appropriate for the job you want to resubmit.
  6. Click OK.

Committing a Silo Backup Job

Use the Commit option in the Job Controller to interrupt and stop a Silo backup job. The next Silo backup job will automatically start from where the previous Silo backup job ended. For example, if a Silo backup job transfers 10 out of 20 volumes to Silo Storage before Commit is clicked, the first 10 volumes are committed and the job is successfully completed. The next Silo backup job will start from volume 11.

Use the following steps to Commit a Silo backup job:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click the silo backup job and click Commit.

The job status may change to Interrupt Pending while the operation completes. Once the operation is complete, the job status will change to Complete.

Changing the Job Priority of an Active Job

You can change the priority of an active job or a group of active jobs from the Change Job Priority dialog box. The lower the priority number, the higher priority the Job Manager gives to the job when allocating resources.

Use the following steps to change the priority of an active job:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.
  2. In the Job Controller window, right-click an active job, and then click Change Priority.
  3. In the Change Job Priority dialog box, enter or select a value (from 0 to 999) in the New Priority box.
  1. Click OK.

Controlling Multiple Jobs Simultaneously

You can control multiple jobs simultaneously in the Job Controller window by selecting the jobs and using the right-click menu or by using the Multi-Job Control dialog box. Use the Multi-Job Control dialog box to control all jobs, all selected jobs, all jobs for a client/agent combination, or all jobs for a MediaAgent. You must have the correct security associations at the proper level for each job selected.

Use the following steps to suspend multiple jobs using the Multi-Job Control dialog box:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Hold the Ctrl key and select multiple jobs.
  3. Right-click the selected jobs and click Multi-Job Control.
  4. In the Action section of the Multi-Job Control dialog box, click Suspend.
  5. In the Selection section of the Multi-Job Control dialog box, click All jobs for this Client, and then select a client from the box.
  6. Click OK to suspend all jobs for the selected client.

Filtering Jobs

You can filter the jobs that are displayed in the Job Controller by creating a job filter. You can create filters for all types of jobs. The filter can be based on an active job for a particular CommCell entity.

CommCell Administrators can use filters created by any user. All other users must use their own filters. If a user account is deleted, that user's filters are automatically deleted.

Creating a Job Filter

Use the following steps to create a job filter:

  1. From the Job Controller window, click the plus  located in the upper right corner next to the Filters box.
  1. In the Filter Definition dialog box, enter the Name, the operations to be filtered, and the CommCell entities to be associated with the filter.

    Note: Admin jobs cannot be filtered by Client Groups or Client Computers.

  2. Click OK.

Applying a Job Filter

Use the following steps to apply an existing filter.

  1. From the Job Controller window, select a filter from the Filters box located in the upper right corner.
  2. To view the definition of the selected filter, click the pencil .

Deleting a Job Filter

Use the following steps to delete a job filter:

  1. From the Job Controller window, select a filter from the Filters box located in the upper right corner.
  2. Click the trash can .
  3. Click Yes in the confirmation window.

Setting Up Alerts

An alert is sent when the job conditions meet the criteria selected for the alert. The threshold and notification criteria determine when and at what frequency alerts are generated. Alerts can be configured globally or they can be job-based. For information on alerts, see Alerts and Notifications - Overview.

Each job can have one alert. If an alert exists for the job, the Add Alert option is not available.

Adding an Alert

Use the following steps to add a job-based alert for an active job in the Job Controller:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click any running job and click Add Alert.
  3. From the Add Alert Wizard dialog box, select the required threshold and notification criteria and click Next.

    For information on the alert criteria available for job types, see Alerts and Notifications - Available Alerts - Job Management.

  4. Select the way in which the alert is to be sent to its intended recipient and click Next. For example, you can click Select [Email] for notification to send the alert as an email.
  5. Select the CommCell users and/or user groups that will receive the alert and then click Next.
  6. Review the options that you selected in the Summary tab and click Finish.

    The alert will be created for the selected job.

Modifying an Alert

Use the following steps to modify an alert configured for an active job in the Job Controller:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click the job associated with the alert and click Modify Alert.
  3. In the Modify Alert Wizard dialog box, make the necessary changes and click Finish.

Deleting an Alert

Use the following steps to delete an alert configured for an active job in the Job Controller:

  1. From the CommCell Console ribbon, click the Home tab and then click Job Controller.

  2. Right-click the job associated with the alert and click Delete Alert.
  3. Click Yes in the confirmation window.

Preempting Jobs

Preemption is defined by the Job Manager at each phase of a job. Jobs that can be interrupted by the Job Manager or by the user and then restarted without having to start the phase from the beginning are called Preemptible jobs. A non-preemptible job is one that cannot be interrupted by the Job Manager or suspended by the user.

If a running job is preemptible, the Job Manager can interrupt the running job and allocate the resources to a higher-priority job if the needed resources are streams, drives or media. The interrupted job enters a waiting state and resumes when the resources it needs become available. Backup and restore operations preempt auxiliary copy and other jobs (except backup).

The following table provides information on the Status of the job in the Job Controller window and the Reason for job delay displayed in the Job Details dialog box when a job is preempted. In addition, a brief explanation on what happens when a job is preempted is also provided.

Jobs Status in the Job Controller Reason for Job Delay Additional Information
Backup Operation Interrupt Pending

 

Waiting

No Job Delay

 

No resources available

Once interrupted, job does not hold on to resources and returns to Waiting status. The job retries for resources. (The Status of the job in the Job Controller window and messages in the Reason for job delay are discussed in What Happens When There are no Resources for a Job.)
Data Recovery Operations (for File System-like agents) Interrupt Pending

 

Waiting

No Job Delay

 

No resources available

Once interrupted, job does not hold on to resources and returns to Waiting status. The job retries for resources. (The Status of the job in the Job Controller window and messages in the Reason for job delay are discussed in What Happens When There are no Resources for a Job.)
Data Recovery Operation (for Database-like agents) Not Preemptible Not Preemptible Not Preemptible
Index Restore (Browse Backup Data) Not Preemptible Not Preemptible Not Preemptible
Auxiliary Copy Interrupt Pending

 

Waiting

No Job Delay

 

No resources available

Once interrupted, job does not hold on to resources and returns to Waiting status. The job retries for resources. (The Status of the job in the Job Controller window and messages in the Reason for job delay are discussed in What Happens When There are no Resources for a Job.)
Synthetic Full Interrupt Pending

 

Waiting

No Job Delay

 

No resources available

Once interrupted, job does not hold on to resources and returns to Waiting status. The job retries for resources. (The Status of the job in the Job Controller window and messages in the Reason for job delay are discussed in What Happens When There are no Resources for a Job.)
Media Refresh Waiting No resources available Once interrupted, job does not hold on to resources and returns to Waiting status. The job retries for resources.

The higher priority job that is doing the resource preemption will display the Reason for Job delay as follows:

Waiting for job[ ] to release the resources.

Preemptible and Non-Preemptible Jobs

In a preemptible phase, the job can be interrupted by the Job Manager or suspended by the user and then restarted without having to start the phase over again from the beginning. Preemptible jobs are always restartable. A File System backup phase is one example of a preemptible phase; the Job Manager can interrupt this phase when resource contention occurs with a higher priority job.

A non-preemptible phase is one that cannot be interrupted by the Job Manager or suspended by the user. It can only run to completion, be killed by administrative action, or be failed by the system. For example, the data recovery operations of database agents are non-preemptible.

The following table lists the types of preemptible and non-preemptible jobs:

Preemptible and Restartable Non-preemptible and Non-Restartable Non-preemptible but Restartable
Data protection operations for most non-database agents. Data recovery operations for database-like agents. Data protection operations for database agents.
DataArchiver archive jobs during the Archive Index and Archive Content Index phases of the job. Media export, erase media, and inventory jobs. The system state phase of Windows File System data protection operations.
Data recovery operations for most File System-like (indexing-based) agents during the restore phase. SAN volume data protection jobs (non-preemptible in its scan phase). Offline Content Indexing jobs.
Data recovery operations from the Search Console. Disk volume reconciliation jobs.  
Most administration jobs including Install Automatic Updates and Download Automatic Updates.    
Silo backup and restore operations.    
Media refresh operations.    
Deduplication database reconstruction job.    

What Happens When There Are No Resources for a Job

Each job requires certain resources for its successful completion. Absence of these resources affects different types of jobs differently. The following table discusses the resources required by each job, the status of the job in the Job Controller window when there are no resources and the corresponding examples of the Reason for job delay displayed in the Job Details dialog box. In addition, a brief explanation on what happens when a job does not have the required resources is also provided.

By default the NetApp Media & Library Manager service on the CommServe cleans up any media and drive reservation that is held by a job which failed to release the resource when it was abruptly terminated, every 1440 minutes. You can modify the frequency using the nRESOURCERELEASEINTERVALMIN registry key.

 
Jobs Resources Status in the Job Controller Reason for Job Delay Additional Information
Backup Operation Streams, Active Media, Drive Waiting

 

Waiting

See Example 1.

 

See Example 2.

Job checks for necessary resources.

If the resources are not available the job retries to reserve the resources when ever they are freed.

Does not hold on to any resource until all the necessary resources are available.

Data Recovery Operations (for File System-like agents) Drive Pending The media is already reserved by some other job(s). If the resources are not available the job retries to reserve the resources when ever they are freed.
Data Recovery Operation (for Database-like agents) Drive Failed

 

Running

See Example 1.

 

See Example 2.

Job checks for necessary resources.

If the resources are not available it retries every 2 minutes to reserve the resources.

Does not hold on to any resource until all the necessary resources are available.
Index Restore Operation (Browse Backup Data) Destination Drives

 

 

 

 

 

 

 

 

 

 

Source Media

 

Pending

 

Waiting

 

Waiting

 

 

 

 

 

 

Running

 

Pending

See Example 1.

 

 

 

See Example 2.

 

 

 

 

 

 

 

 

See Example 2.

Job checks for necessary resources.

Job reserves 2 drives for source and destination media.

If the above resources are not available, it retries every 2 minutes to reserve these resources.

Does not hold on to any resource until all the necessary resources are available.

Once the 2 drives and destination media is obtained job reserves the source media.

If the job encounters resource contention while reserving the source media, (Example 2) it retries every 20 minutes and a maximum of 144 times to obtain the source media.

Holds on to the 2 drives and destination media as long as it is not interrupted and as long as the source media is available.
Synthetic Full Streams, Destination Drives, Destination Media

 

 

 

 

 

 

 

 

Source Media

Waiting

 

Waiting

 

Waiting

 

 

 

 

 

Running

 

 

Pending

See Example 1.

 

 

 

See Example 2.

 

 

 

 

 

 

 

 

See Example 2.
Job checks for necessary resource

Job reserves streams, marks active media full, reserves 2 drives and destination media

If the resources are not available the job retries to reserve the resources whenever they are freed.

Does not hold on to any resource until all the necessary resources are available.

Once the 2 drives and destination media is obtained job reserves the source media.

If the job encounters resource contention while reserving the source media, (Example 2) it retries every20 minutes and a maximum of 144 times to obtain the source media.

Holds on to the 2 drives and destination media as long as it is not interrupted.

Example 1

The operation could not be completed as the drive pool is offline.

The operation cannot be completed as the host is offline.

The operation could not be completed as the library controller is offline.

The operation cannot be completed as the library is offline.

The operation cannot be completed as the master drive pool is offline.

There are not enough drives in the drive pool.

There is no active controller for this library.

Mount path is offline.

Media agent is offline.

Host is offline.

Library controller service is down.

The Library controller is offline.

Example 2

All spares are outside the library.

The operation cannot be completed as the drive is reserved.

The operation cannot be completed as the media is outside the library.

The operation cannot be completed as the mountpath is reserved.

The operation could not be completed as there is not enough media.

The operation could not be completed as there are not enough drives that are not reserved, online and whose controller are alive.

There is no active controller for this drive.

There are no disk paths that are enabled and with the required space.

There are no more spares.

The operation cannot be completed as the opposite side for this media is already reserved.

The operation cannot be completed as the number of drive reservations exceeds the allotted drives.

The number of writers would exceed the maximum allowed limit.

The operation cannot be completed as the media group is reserved.

The operation cannot be completed as the copy is reserved.

The operation cannot be completed as the drive pool is used by storage policy copy.

Requested volume is being mounted/unmounted.

The media is not in library.

Media is being used and is reserved.

Job does not have reservation on the drive.

The requested media is not in any slot of the library.

The requested media is stuck in the drive.

The requested media is exported.

The media is already Reserved by some other Job[s].

The Media is not available. The Job currently using the media was interrupted.

No drives available for reservation.

Not enough good drives available for reservation.

Not enough drives available in Drive Pool.

The interrupted job has not released the drives yet.

The interrupted job has not released the Media yet.

The job has already been interrupted by another job.

Job[s] Interrupted by this Job have not released Resources yet.

Not Enough streams Available for Storage policy [^1%s]. Need ^2%d stream[s] and ^3%d stream[s] are available.

The Media is already reserved by some other Job[s].

Waiting for Jobs [ ] to release the resources.

No resources available.

Setting the Total Running Time for Jobs

Select the Enable Total Running Time check box to set the total amount of time a job can run before it is killed by the Job Manager. The total running time is the maximum elapsed time, in hours and minutes, from the time that the job is created.

The following is the default behavior when Enable Total Running Time is selected:

  • If a job is running when the maximum elapsed time is reached, the job continues running.
  • If the job is not running when the maximum elapsed time is reached, the job is killed.

You can change the default behavior using the following options:

  • Use the Kill running jobs when total running time expires check box to kill a running job when the maximum elapsed time is reached.
  • Use the JMExcludeUserSuspensionTime additional setting to exclude the total time the job was in a suspended state from the total running time calculation.

Setting the Total Running Time

The total running time is the maximum elapsed time, in hours and minutes, from the time that the job is created.

All Jobs

  1. From the CommCell Console ribbon, click the Home tab, and then click Control Panel.
  2. Under the System section, click Job Management.
  3. In the Job Management dialog box, click the Job Restarts tab, and select a job from the Job Type list.
  4. Select the Enable Total Running Time check box and specify the running time in hours and minutes.

    The default total running time is 6 hours.

  5. Click OK.

A Specific Job

For agents that support this capability, the individual setting will override the global setting configured through the Control Panel. The following example uses a backup job.

  1. From the Backup Options dialog box, click Advanced.
  2. In the Advanced Backup Options dialog box, click the Job Retry tab.
  3. Select the Enable Total Running Time check box and specify the running time in hours and minutes.

    The default total running time is 1 hour.

  4. Click OK.

Killing Running Jobs When the Total Running Time Expires

Select the Kill running jobs when total running time expires check box to change the default behavior of the Enable Total Running Time check box. Use the Kill running jobs when total running time expires check box to kill a running job when the maximum elapsed time is reached.

All Jobs

  1. From the CommCell Console ribbon, click the Home tab, and then click Control Panel.
  2. Under the System section, click Job Management.
  3. In the Job Management dialog box, click the Job Restarts tab, and select a job from the Job Type list.
  4. Select the Enable Total Running Time check box and specify the running time in hours and minutes.
  5. Select the Kill Running Job When Total Running Time Expires check box.
  6. Click OK.

A Specific Job

For agents that support this capability, the individual setting will override the global setting configured through the Control Panel. The following example uses a backup job.

  1. From the Backup Options dialog box, click Advanced.
  2. In the Advanced Backup Options dialog box, click the Job Retry tab.
  3. Select the Enable Total Running Time check box and specify the running time in hours and minutes.

  4. Select the Kill Running Jobs When Total Running Time Expires check box.

  5. Click OK.

Excluding the "Suspended" Time from the Total Running Time

Configure the JMExcludeUserSuspensionTime additional setting to change the default behavior of the Enable Total Running Time check box. Use the JMExcludeUserSuspensionTime additional setting to exclude the total time the job was in a suspended state from the total running time calculation.

  1. From the CommCell Browser, navigate to <CommServe node>.
  2. Right-click <CommServe node> and then click Properties.
  3. In the CommCell Properties dialog box, click the Additional Settings tab and then click Add.
  4. In the Add Additional Settings dialog box, type JMExcludeUserSuspensionTime in the Name field.
  5. Select CommServe from the Category box.
  6. Select BOOLEAN from the Type box.
  7. Select true from the Value box.
  8. Click OK.

Queuing Jobs

Setting jobs to be queued allows a job that would otherwise fail to remain in the Job Controller in a Queued state, i.e., waiting. Once the condition that caused the job to be queued clears, the Job Manager will automatically resume the job.

All Data Protection, Data Recovery, Data Collection, Administration Operations jobs can be queued.

Queue Jobs When There is a Conflicting Active Job

Jobs can be queued if they conflict with other currently running jobs such as multiple backup operations for the same subclient.

Use the following steps to queue jobs when there is a Conflicting Active job:

  1. From the CommCell Console ribbon, click the Home tab, and then click Control Panel.
  2. Under the System section, click Job Management.
  3. Select the Queue jobs if other conflicting jobs are active check box.
  4. Click OK.

Queue Scheduled Jobs

You can also set scheduled jobs to be queued. If jobs are scheduled and the Queue Scheduled Jobs option is enabled, these jobs will start in the Job Controller in a queued state at their scheduled time. These jobs can be manually resumed or, if the Queue Scheduled Jobs option is disabled, these jobs will resume automatically.

Selecting this option is especially useful during times of maintenance. Rather than suspend each job manually after it has started, you can enable the Queue Scheduled Jobs option, which will start all the scheduled jobs in the Job Controller in a Queued state. Once you have completed the maintenance, you can manually resume specific scheduled jobs, or simply deselect the Queue Scheduled Jobs option to automatically resume all the scheduled jobs.

Use the following steps to queue scheduled jobs:

  1. From the CommCell Console ribbon, click the Home tab, and then click Control Panel.
  2. Under the System section, click Job Management.
  3. Select the Queue Scheduled Jobs check box.
  4. Click OK.

Queue Jobs When Activity Control is Disabled

Jobs can be queued if the activity control for the job type is disabled.

Use the following steps to queue jobs if the Activity Control is disabled:

  1. From the CommCell Console ribbon, click the Home tab, and then click Control Panel.
  2. Under the System section, click Job Management.
  3. Select the Queue Jobs if activity is disabled check box.
  4. Click OK.

Additional Job Management Settings

Setting the Maximum Number of Simultaneously Running Streams

The Job Controller window displays all the current jobs in the CommCell. A status bar at the bottom of the job controller shows the total amount of jobs; the amount of jobs that are running, pending, waiting, queued and suspended; and the high and low watermarks. The watermarks indicate the minimum and maximum number of streams that the Job Manager can use simultaneously.

Use the following steps to set the high watermark level:

  1. From the CommCell Console ribbon, click the Home tab and then click Control Panel.
  2. Under the System section, click Job Management.
  3. Type the high watermark level for simultaneous running streams in the High Watermark Level box.
  4. Click OK.

Setting a Time Interval for Job Alive Check

You can specify the time interval for which the Job Manager will check active jobs to determine if they are still running. By default, the time interval is set to 2 minutes.

Use the following steps to modify the job alive check interval:

  1. From the CommCell Console ribbon, click the Home tab, and then click Control Panel.
  2. Under the System section, click Job Management.
  3. Type or select the time interval in minutes in the Job Alive Check Interval (Mins) box.
  4. Click OK.

Setting Job Update Interval for Active Jobs

The Job Update Interval allows you to view or modify how often information is updated for backup and restore operations in the Job Details dialog box. The Job Updates Interval list displays all of the available agent types. Specify the protection and recovery time in minutes. You can also set the update interval time for the ContinuousDataReplicator.

Use the following steps to change the Job Update Interval for active jobs:

  1. From the CommCell Console ribbon, click the Home tab, and then click Control Panel.
  2. Under the System section, click Job Management.
  3. Click the Job Updates tab.
  4. Select an Agent from the Agent Type list and click the integer under the Protection (Mins) or the Recovery (Mins) column to change the time interval.
  5. Click OK.

Enabling Jobs to Complete Past the Operation Window Rule

In some cases, an operation launched prior to the time window of an operation window rule may require the ability to run uninterrupted until completion.

Use the following steps to allow running operations to ignore the operation window rule and continue until completion:

  1. From the CommCell Console ribbon, click the Home tab, and then click Control Panel.
  2. Under the System section, click Job Management.
  3. In the General tab of the Job Management dialog box, select the Allow running jobs to complete past the operation window check box.
  4. Click OK.

Preventing Backups on Disabled Clients

Use the following steps to disable backup operations on client computers that are disabled:

  1. From the CommCell Console ribbon, click the Home tab, and then click Control Panel.
  2. Under the System section, click Job Management.
  3. From the General tab in the Job Management dialog box, select the Do not start backups on disabled clients check box.
  4. Click OK.

Command Line Operations

You can perform the following job configurations through the command line:

Viewing Job Summary (qlist jobsummary)

Description

This command lists the current state of all active jobs in the CommServe. Jobs are classified into the following states:

  • Running
  • Suspended
  • Pending
  • Queued
  • Waiting

You can also filter the jobs by client, agent, instance, backup set or subclient.

In case of an error, an error code and description are displayed as: "media: Error errorcode: errordescription"

Usage

qlist jobsummary [-c <client>] [-a <iDataAgent>] [-i <instance>] [-b <backupset>] [-s <subclient>] [-tf <tokenfile>] [-tk <token>] [-h]

Options

-c Client computer name
-a Agent type installed on client computer (see Argument Values - Agent Types)
-i Instance name, required for certain agents
-b Backup set name, required for certain agents
-s Name of the subclient
-tf Reads token from a file
-tk Token string
-h Displays help

Diagnostics

Possible exit status values are:

0 - Successful completion.

1 - CLI usage failures, due to the use of an unsupported option or missing argument.

2 - Any other failure. 

Example

Display job summary of all jobs of client cl1.

qlist jobsummary -c cl1
RUNNING PENDING WAITING QUEUED SUSPENDED TOTAL
 1       10      0       4      1         16

Setting Job Control (qoperation jobcontrol)

Description

This command allows you to kill, resume, suspend, or change/set the progress (in percentage) of a given job. To operate on a single job, specify the Job ID. To operate on more than one job in a specific area, specify the client or MediaAgent name. You can also specify additional levels such as the agent, instance, backup set and/or subclient. The -all option can be used to suspend/resume all jobs.

If the operation is successful, no message is displayed on the command prompt. In case of an error, an error code and description are displayed as: "jobcontrol: Error errorcode: errordescription"

Usage

qoperation jobcontrol [-o <joboperation>] [-j <jobid>] - all [-m <mediaagent>] [-c <client>] [-a <dataagenttype>] [-i <instance>] [-b <backupset>] [-s <subclient>] [-p <priority>] [-tfx <total files to transfer>] [-fx <files transferred>] [-ifx <files transferred since last update>] [-tbx <total bytes to transfer>][-tf <tokenfile>] [-tk <token>] [-h]

Options

-o Operation to be performed on the job. Valid values are:
  • kill
  • suspend
  • resume
  • changepriority
  • setpercentcomplete
-j Job ID
-all All jobs
-m MediaAgent name
-c Client computer name
-a iDataAgent type installed on client computer (see Argument Values - Agent Types)
-i Instance name
-b Backup set name
-s Subclient name
-p Job priority
-tfx Total files to transfer
-fx Files transferred
-ifx Files transferred since last update
-tbx Total bytes to transfer
-tf Reads token from a file
-tk Token string
-h Displays help

Diagnostics

Possible exit status values are:

0 - Successful completion.

1 - CLI usage failures, due to the use of an unsupported option or missing argument.

2 - Any other failure.

Example

  • Kill a job with job ID 175.

    qoperation jobcontrol -j 175 -o kill

  • Suspend all jobs under MediaAgent ma1.

    qoperation jobcontrol -m ma1 -o suspend

  • Suspend all jobs.

    qoperation jobcontrol -all -o suspend

  • Resume all jobs under client cl1 and dataagent "Q_WIN2K_FS".

    qoperation jobcontrol -c cl1 -a Q_WIN2K_FS -o resume

  • Change priority of a job with job ID 175 to 100.

    qoperation jobcontrol -j 175 -p 100 -o changepriority

Setting Job Retention (qoperation jobretention)

Description

This command allows you set retention rules on a given job. The job ID, storage policy, and storage policy copy names must be specified when using this command.

Jobs that are not retained are subject for data aging. Jobs that are retained with a specified retention period will become subject for data aging after the retention period ends. If you retain a job, however, and do not specify the retention period, the job data will never be subject for data aging.

In case of an error, an error code and description are displayed as: "jobretention: Error errorcode: errordescription"

Usage

qoperation jobretention -j <jobid> -sp <storagepolicy> -spc <copy> -rtn <true|false> [-rd <infinite or mm/dd/yyyy hh:mm:ss or yyyy/mm/dd hh:mm:ss>]

Options

-j Job ID
-sp Storage policy name
-spc Storage policy copy name
-rtn Job to be retained (true) or not (false)
-rd Date until the job is to be retained. Valid values are:
  • infinite
  • mm/dd/yyyy hh:mm:ss
  • yyyy/mm/dd hh:mm:ss
-tf Reads token from a file
-tk Token string
-h Displays help

Diagnostics

Possible exit status values are:

0 - Successful completion.

1 - CLI usage failures, due to the use of an unsupported option or missing argument.

2 - Any other failure.

Example

Retain a job - with job ID 175, storage policy sp1 and storage policy copy copy1 - indefinitely by setting the retention period to infinite.

qoperation jobretention -j 175 -rtn true - rd infinite -sp sp1 -spc copy1

Viewing the Last Backup Job (qlist lastjob)

Description

This command displays the summary details of all previous backup jobs completed by a specific client, instance, backup set or subclient. Whenever more than one job is found, the name of each backup job is listed in its own line. The message, "No job found," is displayed on the command prompt whenever backup jobs are not found.

In case of an error, an error code and description are displayed as: "lastjob: Error errorcode: errordescription"

Usage

qlist lastjob -c <client> -a <iDataAgent> -i <instance> -b <backupset> -s <subclient> [-js <jobstatus>] [-tf <tokenfile>] [-tk <token>] [-h]

Options

-c Client computer name
-a Agent type installed on client computer (see Argument Values - Agent Types)
-i Instance name, required for certain agents
-b Backup set name, required for certain agents
-s Name of the subclient
-js The completion status of the job. Use this option to see the last backup job that completed with a particular status. Valid values for this option are:
  • Completed
  • Failed
  • Killed
-tf Reads token from a file
-tk Token string
-h Displays help

Diagnostics

Possible exit status values are:

0 - Successful completion.

1 - CLI usage failures, due to the use of an unsupported option or missing argument.

2 - Any other failure. 

Example

Display the last backup job that ran under client client01.

qlist lastjob -c client01

JOBID STATUS    STORAGE POLICY APPTYPE    BACKUPSET SUBCLIENT INSTANCE  START TIME
----- ------    -------------- -------    --------- --------- --------  ----------
101   Completed SP_12          Filesystem set001    Sub01     <default> 01/01/2013 01:20:55

Viewing a List of Backed Up Files

Use the following steps to generate a list of files which are backed up during a specific job.

  1. Open the Command Prompt and navigate to <Software_Installation_Directory>/Base and run the following command:

    qlogin -cs <commserve_host_name> -u <user name>

  2. Execute the following command from the <Software_Installation_Directory>/Base folder after substituting the parameters:

    ListFilesForJob.exe -job <JOBID> -ma <MAName> [-vm <Instance>] [-tmpdir <TMPDIRPATH>] [-o <OUTFILENAME>]

  3. Navigate to the directory specified in TMPDIRPATH and open the file specified in OUTFILENAME to view the list of files.

The following table displays the parameters that need to be provided before running the command:

Parameter Description of Parameter Values
JobID The job id of the job for which you are generating the list.
MAName Name of the MediaAgent which is used to perform the backup job.
Instance Name of the instance which you have used to install the Windows File System iDataAgent.

This is an optional argument. If you do not specify any value, the job in Instance001 will be used by default to generate the list of files.

TMPDIRPATH The directory in which you want to create the file.

This argument is optional. If you do not specify any directory, the file will be created in the default temporary directory.

The default temporary directory for the software is set using the dGALAXYTEMPDIR registry key. When you install Windows File System iDataAgent, the dGALAXYTEMPDIR registry key gets created at the following location: HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance<xxx>\Base

OUTFILENAME The name of the file in which you want to store the list.

Updating Jobs

Use the following steps to suspend, kill, or resume one or more jobs or to update the reason for suspending, killing, or resuming the jobs.

  1. Download the update_job_template.xml file to the computer where you will run the command.
  2. Open the .xml file and update the XML parameters listed in the table below.
  3. From the command line, go to software_installation_directory/Base and type the following command:

    qlogin -cs commserve_host_name -u user_name

  4. From software_installation_directory/Base type the following command after substituting the XML parameters:

    qoperation execute -af update_job_template.xml

  5. Optional: Verify the jobs were updated by viewing the jobs in the CommCell Console Job Controller window.

The following table displays the XML parameters needed before running the qoperation command:

Parameter Description of Parameter Values
message The jobs that will be updated.

Valid values are:

  • ALL_JOBS, the operation in the operationType parameter will affect all jobs
  • ALL_SELECTED_JOBS, the operation in the operationType parameter will affect the jobs defined in the jobId parameter
operationDescription The reason for suspending, killing, or resuming the job.
operationType The operation to perform on the job.

Valid values are:

  • JOB_SUSPEND
  • JOB_RESUME
  • JOB_KILL
jobId The job IDs for the jobs that will be suspended, killed, or resumed. Use the jobId parameter when the message parameter is set to ALL_SELECTED_JOBS. To add more than one job, add the following line for each job:

<jobs jobId="job_ID" />