Troubleshooting - CommCell Management

CommServe Database

  • CS0001: General performance of the CommCell is not optimal
  • CS0002: Installing clients from previous releases in the CommCell
  • CS0003: CommServe disaster recovery error: [Msg 701] "There is insufficient system memory to run this query"
  • CS0004: Issues on Microsoft SQL Server
  • CS0005: Capture memory and Handle Leak Dumps using the Debug Diagnostic tool
  • CS0006: Schedule policies fail to initiate and cannot be viewed in the scheduler

CommCell Environment

CS0001: General performance of the CommCell is not optimal

Symptom

When a CommServe database is not maintained regularly, you may see one or more of the following symptoms:

  • Sluggish general operation in the CommCell Console
  • Schedules and jobs fail to launch and run properly
  • Excessive file growth of the CommServe or Temp databases

Resolution

Run the DBMaintenance utility against your database. This utility performs a number of maintenance tasks to ensure that your CommServe performs optimally. On the CommServe computer, perform the following steps:

  1. Open the command prompt, navigate to the <Software_Installation_Directory>/Base folder and run the following command:

    dbmaintenance -full

    A full maintenance should be performed the first time the utility is run. It checks for inconsistencies, re-indexes all tables and shrinks the database.

    You should run a full maintenance every 6 months, as we do not recommended shrinking the database on a regular basis.

  2. If you already performed a full maintenance sometime ago, we recommend running the following command every 4-6 weeks:

    dbmaintenance -recommended

    A recommended maintenance checks the database for inconsistencies, and re-indexes the largest and most accessed/modified tables. Unlike the full maintenance, it does not shrink the database.

For more information on this tool, see DBMaintenance Utility.

CS0002: Installing clients from previous releases in the CommCell

Symptom

You want to install clients from previous releases to your version 10 CommCell.

Resolution

An additional setting must be created on the CommServe and client computer. Use the following sections to add the additional settings:

Important: Previous SnapProtect versions may have reached end-of-support. While the product from an unsupported version may continue to be used on an "as-is" basis, fixes/updates will not be provided and new features will not be supported.

On the CommServe

Use the following steps to add the nAllowOlderClients additional setting in order to allow older clients to use the version 10 CommServe during the installation of the software component.

  1. From the CommCell Browser, right-click the <CommServe> node, and then select Properties.
  2. On the CommCell Properties dialog box, click the Additional Settings tab, and then click Add.
  3. On the Add Additional Settings on Windows Client dialog box, complete the following steps:
    1. In the Name box, type nAllowOlderClients. The Category and Type details will be automatically populated.
    2. In the Value box, type 1.
    3. Click OK.
  4. Click OK to close the properties dialog box.

If the CommServe has software version SP5A or earlier, restart the CommServe services after adding the additional setting.

On Windows Clients

On a Windows client computer, create the bIgnoreCommServeVersion registry key using the Windows Registry Editor as follows:

  1. Click Start, click Run, type regedit, and then click OK.
  2. On the Registry Editor window, navigate to the following directory:

    HKEY_LOCAL_MACHINE\SOFTWARE\

  3. Right-click SOFTWARE and click New -> Key.
  4. Name the key as GalaxyInstallerFlags.
  5. Right-click GalaxyInstallerFlags and select New -> DWORD value. Name the key as bIgnoreCommServeVersion.
  6. Double-click the bIgnoreCommServeVersion key and modify the Value data to 1.
  7. Proceed to install the software component.

On UNIX/Linux Clients

On a UNIX/Linux client computer, navigate to Software Installation Package or mount path, and then run the following command:

./cvpkgadd -allow-newer-commserve

If you are installing a SnapProtect Version 9 package, you can create a custom package or a silent install XML file to install the package. In the XML file, update the value of the allowNewerCommServe parameter to 1.

CS0003: CommServe disaster recovery error: [Msg 701] "There is insufficient system memory to run this query"

Symptom

The following errors might be generated during a CommServe disaster recovery backup:

  • From the commserveDR.log file:

    CommservDR::doBackupRestore() - Copying the dump file from staging area to the ER directory
    CommservDR::dumpLog() - Log>> DBCC execution completed. If DBCC printed error messages, contact your system administrator.
    CommservDR::dumpLog() - Log>> There is insufficient system memory to run this query.
    CommservDR::dumpLog() - Log>> Check terminated. A failure was detected while collecting facts. Possibly tempdb out of space or a system table is inconsistent. Check previous errors.
    CommservDR::backupCmd() - CommServeDR: Problems encountered backing up SQL metadata.
    CommservDR::backupCmd() - CommServeDR: Full backup failed.

  • From the JobManager.log file:

    ODBCApi\DBcursors.cpp ,line 465, Problem Declaring/Preparing/Opening cursor for Table APP_SubClientProp (Batch 1)

    [DBErr] SqlState: 42000 NativeError: 701 Line/Row: 1/1 [DBErr] Message: [Microsoft][ODBC SQL Server Driver][SQL Server]There is insufficient system memory to run this query.

    SCInterfaceDB::getProperties() - Open Cursor on [APP_SubClientProp] Table failed. Error [ [DBErr] SqlState: 42000 NativeError: 701 Line/Row: 1/1 [DBErr] Message: [Microsoft][ODBC SQL Server Driver][SQL Server]There is insufficient system memory to run this query.]

    Scheduler Set pending cause [CommServeDR: Error: [Msg 701].]::Client [ ] Application [commserveDR] Message Id [570425409] RCID [0] ReservationId [0]. Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0]. Scheduler Ignored pending cause [CommServeDR: Error: [Msg 8921].]::Client [ ] Application [commserveDR] Message Id [570425409] RCID [0] Reservation Id [0]. Level [0] flags [0] id [0] overwrite [0] append [0] CustId[0]. Scheduler Phase [Failed] message received from [ ] Module [commserveDR] Token [] restartPhase [0]
    JobSvr Obj Phase [1-Backup To Disk] for job Failed and will be restarted.

Resolution

This error indicates that the CommServe is running low on memory and is unable to backup the SQL database. This can occur if the computer has been up for an extended period of time or under heavy load. If the computer is under heavy load, it is due to SQL temp files which can grow large if it is not set back to the default size regularly. One of these temp files is the Templog.ldf log file, which can continue to grow each time SQL is used.

To resolve this issue, stop the SnapProtect services, cycle the SQL services, and then restart the services. For step-by-step instructions on controlling services, see Controlling Services on Windows.

If this workaround fails to resolve the issue, then reboot the CommServe computer. This returns the temp file to the default size and allow the disaster recovery backup to run.

We recommend to use the DBMaintenance utility to maintain the CommServe database and prevent the issues explained above from happening. For more information on this tool, see DBMaintenance Utility.

CS0004: Issues on Microsoft SQL Server 2008

This is a collection of issues and resolutions regarding SnapProtect and the SQL Server.

Symptom 1

The backup of the SQL databases as "Full Text" databases failed for one or more databases. Error Code 30:323 is generated with the following message:

Error encountered when transferring data for db […] to the MediaAgent. Please check the SQL VDI.LOG AND connectivity between client and MediaAgent.

The following error is also found in the SQLiDA.log in the SnapProtect folder:

The backup of full-text catalog 'ProjectDBTemplateUnitAttachment' is not permitted because it is not online. Check errorlog file for the reason that full-text catalog became offline and bring it online. Or BACKUP can be performed by using the FILEGROUP or FILE clauses to restrict the selection to include only online data.

Resolution 1

This issue normally occurs when "Full Text" catalogs are offline. Make sure to review the Full Text indexes to see if there are any problems.

Also, review MSSQLSERVER_9987 (a Microsoft KB article) which displays the same error.

Symptom 2

There is an identified Microsoft issue when installing SQL 2008 on the second node of a failover cluster. The SQL Server setup may encounter the following error:

The current SKU is invalid

Resolution 2

Refer to the following Microsoft KB articles to fix the issue:"

CS0005: Capture memory and Handle Leak Dumps using the Debug Diagnostic tool

Symptom

When you notice that the memory consumed by the CommServe database grows, it might be due to memory leaks. You can use the Debug Diagnostic Tool to capture memory dumps and find out the cause of the memory rise.

Resolution

Setup the Tool and Create a Rule

  1. On the CommServe computer, download and install the Debug Diagnostic Tool v1.2.
  2. After installation, open the tool.

    Navigate to Tools | Options and Settings, and make sure that the folders and search paths are set as shown in the screen. Then, click OK.

    A symcache folder is created automatically under C: drive, which can be later deleted once the memory dumps are captured.

  3. Click Add Rule. The Select Rule Type window appears.

    From this window, select Memory and Handle Leak and click Next.

  4. In the Select Target window, select the process that shows the memory usage increase, and then click Next.
  5. In the Configure Leak Rule window, do the following:
    • Select Start memory tracking immediately when rule is activated.
    • Select Generate final userdump after n minutes of tracking and specify the number of minutes to generate the dump.

      For example, if the process increases to 400 MB in 24 hours, specify 1440 minutes. If the increase of memory usage takes a shorter amount of time to reach 400 MB, then modify the time accordingly.

    • Select Auto-unload LeakTrack when rule is completed or deactivated.
    • Click Next.

  6. By default, the rule name and the location of the user dump is already provided. You can change them if needed.

    Click Next.

  7. Click Activate the Rule now and then click Finish.

    The tool immediately starts to track the memory leak.

Test and Monitor the CommServe Environment

  1. From the CommCell Console, run a few backup jobs to increase the memory consumption.
  2. Check the CommServe database logs to monitor the times at which the jobs were run, so you can later compare it with the memory usage increase.
  3. Take a few memory dumps using the tool. Do this at time intervals when you see a memory usage increase (for example, when a few hundred MBs of memory are consumed).

    To take a memory dump, navigate to the Rules tab, right-click the rule you created, and then proceed to create a dump.

    The time interval depends on the amount of memory consumption between two points in time. For slow growing memory, it could take a few hours before the memory consumption difference reaches a few hundred MBs.

CS0006: Schedule policies fail to initiate and cannot be viewed in the scheduler

Symptom

Schedule Policies fail to initiate and cannot be viewed in the scheduler when the CommServe Time Zone is used.

Resolution

Install the Microsoft Windows update in the following Microsoft KB article: December 2012 cumulative time zone update for Windows operating systems.

CC0001: Sophos anti-virus 7.6.8 and other versions trigger error code [19:599]: "Loss of Control Process..."

Symptom

Sophos AntiVirus version 7.6.8 introduced a new feature that conflicted with the backup and restore processes of the SnapProtect software. For this reason, backup and restore operations may fail on Database agents such as Exchange, SQL, Oracle, MySQL, DB2 and Sybase with the following error: (additional agents might be affected)

Error Code: [19:599] Loss of control process processname.exe

where the processname.exe can be one of the following processes:

  • exTiDbBackup.exe (Application Event triggered in OS Event Viewer)
  • SQLBackup.exe (Application Event triggered in OS Event Viewer)
  • SrvOraAgent.exe (System Event triggered in OS Event Viewer)

The following are examples of how this error is displayed from the log files and from the CommCell Console.

Example 1: From the CommServe system.log File or from the Operating System Event Viewer

[TYPE] Information [TIME] [...] [SOURCE] Application Popup [COMPUTER] [...] [DESCRIPTION] Application popup: StartRestore.exe - Application Error : The application was unable to start correctly (0xc0000005). Click OK to close the application.

[TYPE] Information [TIME] [...] [SOURCE] Application Popup [COMPUTER] [...] [DESCRIPTION] Application popup: SrvOraAgent.exe- Application Error : The application was unable to start correctly (0xc0000005). Click OK to close the application.

Example 2: From the CommCell Console

Error Code: [19:599]
Description: Loss of control process exTiDbBackup.exe. Possible causes:
1. The control process has unexpectedly died. Check DR. Watson log or core file.
2. The communication to the control process machine (server name) might have gone down due to network errors.
3. If the machine (server name) is a cluster, it may have failed over.
4. The machine (server name) may have rebooted.

Application Event Error example:

Log Name: Application
Source: Application
Error Date: Date/Time
Event ID: 1000
Task Category: (100)
Level: Error
Keywords: Classic
User: N/A
Computer: Server Name
Description: Faulting application exTiDbBackup.exe, version 7.0.0.1147, time stamp 0x490f2dd7, faulting module ntdll.dll, version 6.0.6001.18000, time stamp 0x4791adec, exception code 0xc0000005, fault offset 0x00000000000b1188, process id 0x6b4, application start time 0x01ca0a1ff5d96161.

System Event Error example:

TYPE: Information
TIME: Date.Time
SOURCE: Application Popup
COMPUTER: Server Name
DESCRIPTION: Application popup: SrvOraAgent.exe - Application Error : The application was unable to start correctly (0xc0000005). Click OK to close the application.

Resolution

Sophos version 7.6.10 resolved the backup and restore issue. However, with the release of version 7.6.21, the issue was re-introduced.

The problem behind the issue is that the operating system of the server is not set to support 8.3 character length file names. Sophos tries to create REG_SZ registry values using path structures similar to C:\PROGRA~2\Sophos\Sophos Anti-Virus\SOPHOS~1.DLL. In an environment where 8.3 files are not supported, the registry values cannot be created correctly.

Sophos provided the following workaround for this issue:

  1. From the command prompt, run the following query to check if the creation of 8.3 files is enabled on Windows:

    fsutil behavior query disable8dot3

  2. If the output of the query is disable8dot3 = 1 {disabled condition}, then toggle disable8dot3 to 0 using the following command:

    fsutil behavior set disable8dot3 0 {Enable}

  3. Remove the contents of the AppInit_DLLs registry keys in order to leave the key with a blank value. These keys are found in the following locations:
    • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Windows
    • HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Microsoft\Windows NT\CurrentVersion\Windows

    This operation will clear the following values:

    • C:\PROGRA~2\Sophos\Sophos Anti-Virus\sophos_detoured.dll
    • C:\PROGRA~2\Sophos\Sophos Anti-Virus\sophos_detoured_x64.dll
  4. Uninstall the Sophos antivirus. Ensure the Sophos directory under Program Files is also deleted.
  5. Re-install Sophos. The AppInit_DLLs registry keys should now have the following value:

    C:\PROGRA~2\Sophos\Sophos Anti-Virus\SOPHOS~1.DLL,C:\PROGRA~2\Sophos\SOPHOS~1\SOPHOS~1.DLL

CC0002: General license issues (when applying or releasing licenses)

The following is a collection of licensing issues along with their respective resolutions.

Symptom 1

Assume that you uninstalled an iDataAgent from the CommCell (the license used by the agent is consumed). When you try to use the same license on a different server, you might receive the following error in CVCsl_Licensing_Dbg.log:

Not enough licenses

Resolution 1

Perform the following operations to force the CommServe database to update and clean up the license availability.

On the CommCell Console

  1. Run the License Summary Report with most of the options (in the General tab) selected. See License Summary Report for step-by-step instructions.
  2. From the Control Panel, click License Administration and then click the License Details tab. Compare the licenses shown in the report and the License Details tab to check that they match.

On the CommServe

  1. Log off from the CommCell Console.
  2. Restart the SnapProtect services. See Controlling Services on Windows for step-by-step instructions.
  3. Log on to the CommCell Console, and generate a new License Summary Report.

Symptom 2

Evaluation licenses can be applied only once on a specific server, unless the server is rebuilt from the operating system.

Once an Evaluation license expires at the end of 30 days, the CommCell does not let you re-apply a new Evaluation license to the same server. Also, if you uninstall an Evaluation license, it cannot be used on another server. These errors can be found in CVCsl_Licensing_Dbg.log.

Resolution 2

Use the following steps to resolve this issue:

  1. Rebuild the server from the operating system
  2. Request  a Permanent License from your Account Manager
  3. Request additional Evaluation Licenses from your Account Manager

CC0003: Impact of Daylight Savings Time on backup schedules, Disaster Recovery testing, and Microsoft SQL Server

Symptom 1

There is a known Microsoft SQL issue with an incorrect value in the TIME_ZONE column of the BACKUPSET table. This issue affects CommServes setup with SQL 2008, SQL 2008 R2, and SQL 2012 when log shipping is used and Daylight Saving Time changes are implemented.

Resolution 1

CommServes Setup with Microsoft SQL 2008

  1. Check for a Microsoft SQL Server agent installed on the CommServe.
    • If the Microsoft SQL Server agent is not installed, log shipping is not used and the CommServe is not affected.
  2. If there is a Microsoft SQL Server agent installed on the CommServe, check to see if the agent has regularly scheduled backups (database and logs) as well as out of place restores of the logs to the Disaster Recovery CommServe.
    • If this setup is not used, log shipping is not used and the CommServe is not affected.
  3. If your setup meets the criteria above, please contact customer support to open a support case.

CommServes Setup with Microsoft SQL 2008 R2 or SQL 2012

To resolve this issue, refer to Microsoft KB article 2697983.

Symptom 2

The continuous world-wide changes to the Daylight Saving Time (DST) start and stop times affect jobs scheduled during the hour of the Time Change Event, as follows:

  • For any jobs scheduled when the clock's time goes 1 hour ahead

    All Storage Policy schedules and client jobs scheduled at this time will be automatically moved to the next hour beyond the time change hour.

    For example, the US EST to EDT time change is between 2:00AM and 2:59AM on Sunday March 13th, 2011. This means all schedule jobs will be moved to 3:00AM to 3:59AM on Sunday March 13th, 2011.

  • For any jobs scheduled when the clock's time goes 1 hour back

    All Storage Policy schedules and client jobs scheduled at this time will continue to run, and no duplicate jobs will be launched.

    For example, the US EDT to EST time change is between 1:00AM and 1:59AM on Sunday November 6th, 2011 (time regresses to 1:00AM at 2:00AM). This means all schedule jobs started between 1:00AM to 1:59AM EDT will remain running, and duplicates will not be kicked off at 1:00AM EST.

The schedules return to normal on the next day's schedules.

Resolution 2

This issue needs has no resolution as this is a normal function of the CommCell.

  • For Standard Time to Daylight Time

    The time between 2:00AM and 2:59AM does not exist for this day. Note also that any jobs already running on this time frame will seem to take an additional hour to complete due to the time advance.

  • For Daylight Time to Standard Time

    The time between 1:00:00AM and 1:59:59AM is repeated. Note also that any jobs already running on this time frame will seem to take one hour less to complete due to the time regression.

For additional information on DST changes and to make sure your systems are set up properly, see the following details on Microsoft and Java DST:

Additional Information on Day Light Savings Time

Microsoft Daylight Savings Time

There have been cases where the Microsoft DST patches were not applied completely by some customers. Due to this, it is required that you audit your servers and confirm if the patches for your servers and desktops have correctly updated the Windows registries.

The DST change impacts SQL Sever Log Shipping, Log Replication and SQL Mirroring where servers are located in different time zones. It is critical that all DR testing scenarios include the Microsoft and Java DST updates in order to match the changes in the current production CommServe/MediaAgent/Client.

Use the following registry locations to confirm if the latest time zones have been applied:

  • HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows NT\CurrentVersion\Time Zones
  • HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Control\TimeZoneInformation

The following are helpful links for Microsoft Time Zone details:

Java Daylight Savings Time

Java uses its own Olson Time Zone database and not the Microsoft registry entries as Java offers support for multiple platforms. If you are using an older version of Java during the Time Zone change period (depending on the country/time zone the CommServe is located), the CommCell Console may display incorrect times compared to the operating system time.

If the latest Java version is not installed, make sure to follow the standard Java recommended practices and download the latest version of Java from the Java site. Also, if you have multiple versions of Java installed (including the latest version), then from the Java console located under the Control Panel, disable all older versions as long as they are not needed.

The following are helpful links for Java Time Zone details:

CC0004: Job killed or suspended by the system

Symptom

When a job is killed or suspended by the system, the message displayed on Reason for Delay reflects the action taken by the system. This message conveys that the disk space resources on the CommServe are running low or below the minimum space requirements. One of the following event codes may be displayed in the message:

  • Error Code [19:861]

    Killed by the System

  • Error Code [19:862]

    Suspended by the System

  • Error Code [19:1595]

    Killed and marked as failed by System

  • Error Code [19:1111]

    The job has exceeded the total running time

There are additional scenarios that can produce the same messages listed above

Resolution

The Job Manager terminates a job when:

  • Free space is less than 25MB in the drive where the CommServe installation directory is located
  • The number of job retries exceeds the value set in the Job Management (Job Restarts) dialog box
  • Jobs overlap with each other (known as conflicting jobs). For example, when a new backup job is initiated for the same subclient where another backup job is currently running.

    The Job Manager kills the earlier job and replaces it with the newer job if the earlier job has not yet transferred any data to the media.

Do the following to resolve the issues:

  • Perform a Space Check

    Before you determine if there is enough space in the drive where the CommServe is installed, you should make sure that the space requirements for your Windows server computer are met. Microsoft has published best practices for their operating system requirements in the following KB articles:

    In the KB articles below, navigate to the Important Supportability Information section, which contains the information you need.

    When you start to check the space for the CommServe installation directory, you also need to take into account the CommServe folder under the SQL Server directory, for example, C:\Program Files\MSSQL2008\MSSQL10.COMMVAULT\Data.

  • Fix Conflicting Jobs

    Configure the Job Manager to deal with conflicting jobs by killing the current job and replacing it with the newer job if the earlier job has not yet transferred any data to the media. You must enable the JMKillPreviousBackupJobForSameSubclient additional setting to accomplish this. See Terminating a Conflicting Job for more information.

  • Check Job Restartability

    Check the configuration for CommCell jobs from the Job Management option in the CommCell Console's Control Panel. See Job Restartability for more information.

CC0005: Jobs pending with error "Please check the log files for this job for more details"

Symptom

The following are Reason for Job Delay errors that may show for a pending job:

  • Error Code [19:1109]

    Please check the log files for this job for more details

    This error is recorded in the JobManager.log located on the <Software Installation Directory>/Log Files folder.

  • Error Code [19:1310]

    Please check the log files for this job for more details

    This error is recorded in the scheduleReport.log located on the <Software Installation Directory>/Log Files folder.

    This error message indicates that an unknown error has occurred, and the log files must be examined to determine the cause of the problem. Depending on the cause of the error, you might find the following error phrases in the logs:

    • Media Manager related Errors

      62:1 Undefined software error occurred.
      62:441 Internal software error occurred.

    • Archive Manager related Error

      80:6 Undefined software error occurred.

One of the common triggers for these errors is a change to one of the job's components, such as:

  • IP Address, where the hosts file or DIPS no longer matches actual information.
  • Communications between Job components has failed (for example, CommServe from/to MediaAgent).
  • After an OS upgrade, the user account no longer has correct Read/Write permissions to all locations used during a job.

Resolution

We recommend that a support representative examines the log files to resolve this issue. Please contact Customer Support to continue troubleshooting this issue.

In most scenarios, the support representative will require a full set of logs from the CommServe and MediaAgents associated with the job(s). To provide the logs, you must perform the Send Log operation from the CommCell Console. See Sending Log Files for more information. When performing the send log operation, consider the following:

  • Before sending the logs:
    • Run a manual Disaster Recovery backup (as the scheduled DR backup may not contain the data when the error occurred).
    • In the <Software Installation Directory>/Log Files folder, check if there are any logs larger than 50MB. If large log files are found, work with the support representative to determine what triggered the abnormal growth of the logs.

      Also, look for any .CAB files. These files do not normally belong to the Log Files folder, and will cause issues when trying to send log files. If you find these files, report them to the support representative.

  • Do not select Log fragments for this job only and Crash Dump are not selected in the Machine Information tab.
  • Do not to set any time range in the send log operation as the issue might fall outside the time criteria that you set.

CC0006: Jobs pending with error code [19:1161] "The number of running jobs has exceeded the limit on the Maximum Number [#] allowed"

Symptom

Job Pending Reason error code [19:1161]:

The number of running jobs has exceeded the limit on the maximum number [# value] allowed.

This error message is delivered because the High Watermark Level for CommCell jobs was reached or exceeded.

Resolution

To address this issue, the High Watermark setting must be updated as it may too low for the number of job streams concurrently running.

Use the following steps to update the high watermark setting from the CommCell Console:

  1. Suspend all jobs and kill the jobs that cannot be suspended.

    When changing the high watermark setting, no jobs should be running in the Job Controller window or in the command line. We recommend to take screen captures of the jobs in the Job Controller to easily restart the ones that will be killed.

  2. From the CommCell Console toolbar menu, click Control Panel.
  3. Under the System section, click Job Management.
  4. In the High Watermark Level box, add 25 to the current value and click OK.

    Monitor the CommCell and check if the issue subsided. If the issue persists, increase the High Watermark Level option again by 25. Do this until the issue subsides.

  5. Restart all services. See Restarting a Service for more information.
  6. Resume all suspended jobs and restart the jobs that you killed.
  7. Monitor the CommCell and check if the issue subsided.

    If the issue persists, increase the High Watermark Level option again by 25. Do this until the issue subsides.

    Keep in mind that if the high watermark value is too high, it may introduce performance issues on the CommServe, depending on the available resources.

    If you experience that the CommServe performance decreased, reduce the High Watermark Level option by 10 and monitor the CommServe.

CC0008: Unable to run or modify the CommCell Console or scheduled reports after updating the Java version

Symptom

Java version 7 (1.7.0_xx) is installed with the CommCell Console. The latest versions of Java 7 are installed in the <Java Installation Path>\jre7 directory.

When the CommCell Console runs, it may point to the old Java directory (Java version 6), for example, <Java Installation Path>\jre6 directory. This may cause the following issues:

  • The CommCell Console does not open (hangs in a loading state). Also, when the console runs as a web-based application, it tries to load the wrong Java version (local version versus the CommServe version)

    You may find a mismatch of the installed Java versions between your local browser and the CommServe, as seen below:

    • Local Java version is 1.7.0_xx for 32-bit, CommServe has 1.6.0_xx Installed in Program Files (x86).
    • Local Java version is 1.7.0_xx for 64-bit, CommServe has 1.6.0_xx Installed in Program Files.

  • The shortcut to launch the CommCell Console are reported as missing

    Windows is searching for the javaw.exe file. This file does no longer reside under the <Java Installation Path>\jre6\bin directory.

  • Manually running a scheduled report in the console fails

  • Modifying a report in the console fails

Resolution

Depending on the agents and features installed on the CommCell, there are several items to review and correct when installing or updating the Java version. The CommCell Console needs that the Java version matches on the CommServe and on the local computer.

To resolve this issue, review the following checklist:

  • Check the browser and Java versions installed on the CommServe and the local computer (32-bit and/or 64-bit).
  • Check for older Java versions prior to version 7 that are installed on the CommServe and local computer (32-bit and/or 64-bit).
  • Fix the CommCell Console shortcuts to point to the new Java version 7 path.
  • Update SnapProtect registry keys to point to the new Java version 7.

The following solution steps assume that you upgraded from Java version 6 to version 7.

Fix the Console not Opening and/or Loading Wrong Java Version

  1. Check if there is an older Java version installed on your computer using Add or Remove Programs.

    If found, uninstall all the older versions.

  2. Verify that both 32-bit and 64-bit versions of Java 7 are installed.

    Review the following link to check if you have both 32-bit and 64-bit versions of Java:

    Which Java download should I choose for my 64-bit Windows operating system?

Fix the Missing CommCell Console Shortcuts Issue

  1. Right-click the CommCell Console shortcut file and then click Properties.
  2. In the Target box, change jre6 to jre7.
  3. Click OK.

You can fix all SnapProtect shortcuts by fixing the main CommCell Console launcher:

  1. Click the Start button on the Windows task bar and then click All Programs.
  2. Navigate to ##_DOC_OEM_INSTALL_DESTINATION_COMPANY_FOLDER_## and then right-click SnapProtect Administrative Console.
  3. In the Target box, change jre6 to jre7 and then click OK.

Fix the Java Data Paths in the Registry

  1. Open the Registry Editor window.
  2. Navigate to HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance001\JRE.
  3. Update the value of the sINSTALLPATH key to point to the new Java path.

Depending on the agents and features installed, you may need to update other registry keys as well.

For example, if you have the Web Console installed, navigate to HKEY_LOCAL_MACHINE\SOFTWARE\CommVault Systems\Galaxy\Instance001\WebConsole. and update the value of the sZJAVAHOME key to point to the new Java path.

CC0009: Users have access to files and folders belonging to others users

Symptom

Users are able to Browse, Find and Restore files and folders associated with other users from all user-interfaces such as the CommCell Console, Web Console, etc.

Cause

  • ACLs (Access Control Lists) are not included in the backup, and/or
  • The user has Browse capability instead of only the End User Access capability.

Resolution

  1. Make sure that only End User Access capability is configured on the client computer. For instructions, see Configuring End-User Operations on Client Computers.

    Specifically ensure that Browse capability is not assigned to these users.

    Assigning End User Access capability helps to maintain multiple user profiles on the same laptop (or desktop) and ensures that each user has the ability to browse and restore only the data to which he or she has access.

  2. Make sure that the Catalog ACL (end user access control list) option is enabled in the subclients before performing a backup as described in:

Once enabled, this option ensures that ACL s are included in the backup, which in turn, allows users to access only those files and folders for which they have permissions.

After enabling this option, make sure to run a Full backup subsequently. This will ensure that ACLs are available in the backup data.

Conversely, if you run a Differential or Incremental backup after enabling this option, only the newer data will include ACLs.