Data Pipe

Table of Contents

Overview

The Data Pipe is the logical mechanism through which data is transferred between a client and a MediaAgent. You configure and optimize the Data Pipe using the Data Pipe Buffers, Network Agents and Network Bandwidth Throttling, and Application Read Size parameters described on this page. All these parameters should be set so that data transfer is not only fast and lossless, but also does not use more resources than necessary.

Data Pipe Buffers

The Data Pipe Buffers setting controls the amount of shared memory allocated for data pipes on each client or MediaAgent computer. By default, the system allocates 30 pipeline buffers of 64KB size on each client computer. (30 buffers of 64KB each allocates about 2 MB of shared memory.)

You can configure additional buffers for transferring data between the client and the MediaAgent. Having more pipeline buffers may translate into faster dedicated (non-multiplexed) backups to tape devices, and therefore improve data transfer performance. Of course, additional buffers consume more shared memory.

Configuring Data Pipe Buffers

This is an advanced feature. Please use caution when modifying this setting.

Important: You can set the number of data pipe buffers on the MediaAgent or the client. If the value is set both on the client and the MediaAgent, the setting on the client is used.

Change the default data pipe buffers value (30) as follows:

  1. In the CommCell Browser, expand Client Computers.
  2. Right-click the client, then click Properties.
  3. Click Advanced.
  4. On the Additional Settings tab, click Add.
  5. In Name, enter nNumPipelineBuffers. Press TAB.
  6. In Value, enter a number between 30 to 1024.
  7. Click OK to save the setting.
  8. Click OK to close the dialog box.

Network Agents

Network Agents establish the data pipes. By default, the system uses two network agents, which may be suitable for 100-megabit networks. Increasing this value may increase the data transfer throughput from the client. Other network features, such as NIC speed and network switch speed, also impact network throughput. You may need to experiment in your environment to determine whether you can realize faster data throughput.

Setting the Network Bandwidth and Network Agents for a Data Protection Operation

All subclients except those using NAS iDataAgents support network bandwidth throttling.

To set a limit for network bandwidth and the number of concurrent network agents for a subclient backup operation, perform these steps:

  1. In the CommCell Browser, expand Client Computers > client_name > agent.
  2. Click a backup_set_name, then on the backup_set tab, right-click a subclient and click Properties.
    • If your iDataAgent is DB2, DB2 MultiNode, Informix, Oracle, Oracle RAC, SAP, or Sybase, right-click an instance, then click Properties.
  3. Click the Storage Device tab, then the Data Transfer Option tab.
    • Set the maximum number of Network Agents to run concurrently when performing a subclient or instance backup.
    • Select the Throttle Network Bandwidth (MB/HR) option, then enter the throughput limit as needed. Note that the SnapProtect software throttles the transfer on a per-network-agent basis.
  4. Click OK to save your changes.

Setting the Network Bandwidth and Network Agents for Multiple Subclients Using Subclient Policy

For iDataAgents that support subclient policy, network agents can be established for creating multiple subclients using an existing subclient policy.

Use these steps to set the Network Bandwidth and Network Agents while creating multiple subclients:

  1. From the CommCell Browser, expand Policies > Subclient Policies.
  2. Right-click Subclient Policies, then click New Subclient Policy. The New Subclient Policy dialog box opens.
  3. Click in the Name box, then enter a name for your subclient policy.
  4. Click the iDataAgent box and select Windows File System.
  5. Click once in the Storage Policy column cell and select a storage policy for the default subclient.
  6. Click the Association tab and select the backup set on the client that you want to associate with this subclient policy.
  7. Click the General tab, then select the default subclient row and click Edit.
  8. Click the Content tab and enter the contents for the subclient by browsing to them or by entering their path.
  9. Click the Storage Device tab and select the storage policy.
  10. Click the Data Transfer Option tab.
    1. Enter the number of Network Agents that must be used to perform backup operations on the subclient/instance.
    2. Click the Throttle Network Bandwidth (MB/HR) option, then enter a throughput value. Note that throttling is done on a per-Network-Agent basis.
  11. Click OK to save the changes.
  12. Click OK to complete this operation.

The default subclient is automatically created in all the clients with which the subclient policy is associated.

Application Read Size

Application read size is the size of the application data read from the clients for data transfer during backup operations. Values for Application Read Size must be in the power of 2; minimum value is 64 KB, maximum value is 4096 KB (4 MB).

Note that each application internally allocates a buffer size that is suitable for handling the application data. When the size of the application data read during backup operations matches the source application's internal buffer allocation then the overhead is minimized. So to achieve optimal rate of data transfer during backups, configure this value based on the source application's buffer allocation.

The value of application read size may be increased to reduce the amount of data read from the given application, which in turn reduces the amount of I/O operations performed against it. As a result, overall backup performance may increase. However, backup memory usage may also increase, which may inadvertently consume additional resources from the application. Thus, it is recommended that this value be set at either the default value or match the block size directed by the application.

This configuration is supported for the following agents:

  • Microsoft Windows File System iDataAgent
  • Microsoft SQL Server iDataAgent

Modifying Application Read Size

Use these steps to set the Application Read Size:

  1. In the CommCell Browser, expand Client Computers > client > File System. Click defaultBackupSet.
  2. Right-click a subclient, then click Properties.
  3. Click the Storage Device tab, then Data Transfer Option.
  4. Select Application Read Size, then select its value.
  5. Click OK.