Loading...

Connecting to HTTP Responses with Data Cube

You can use Data Cube to connect to the data in an HTTP response from a RESTful API service.

Before You Begin

  • You must be able to log in to the Web Console to view Data Cube. See Accessing the Web Console.
  • Only users assigned a role with the Data Connectors permission at the MediaAgent level can access Data Cube in the Analytics section of the Web Console. The associated MediaAgent must have been configured with Analytics Engine for Data Cube.
  • You need to obtain the following information to configure the HTTP connection:
    • The type of HTTP request that you want to send, either GET or POST.
    • Any headers, such as security credentials, or other raw data that must be sent with the request to receive the proper HTTP response.
    • The parameters and values for paginated sets of data in the HTTP response.
  • Note: Only JSON responses are supported.

Procedure

  1. In a Web browser, log in to the Web Console and then click Analytics.
  2. In the left navigation next to HTTP, click Add New.
  3. On the New Data Source (HTTP) page, configure the source as follows:
    1. Under Data Source Name:
      • Click the Analytics Engine list and select the Analytics Engine on which to store the crawling data.
      • In Data Source Name, enter a name for the data source. The name can only contain alphanumeric characters, underscores, and dashes.
      • In Data Source Description, enter a description for the data source.
      • Click Next to proceed to the next section.
    2. Under URL Options:
      • In URL, enter the URL to the RESTful API service to which you want to connect.
      • Click the Request Type list and select the type of HTTP request that you want to send.
      • To send raw JSON data with the HTTP request, enter the data in Payload.
      • To send header information with the HTTP request, next to Header enter the data in the Key and Value boxes.

        Note: If you need to add additional headers, click the plus icon. To remove a header row, click the trash icon.

    3. If the URL for the HTTP request contains pagination parameters, click Pagination and proceed as follows:
      • In Pagination Parameter, enter the parameter of the URL that determines which page of the total data set to return in the response in the Name box and the starting position in the Value box.
      • In Parameter that specifies the length of each page, enter the parameter of the URL that determines the number of records to return for each page in the response in the Name box and the value of the parameter in the Value box.
      • In Step pagination parameter by, select the type of pagination used by the API as follows:
        • Offset: Select to increase the pagination parameter by the number of records returned on each page.
        • Page: Select to increase the pagination parameter by one.
      • In Total number of pages/records to crawl, enter a number or a path in the API response to indicate the last page or total number of data records to include when crawling the data source.
    4. Click Next to proceed to the next section.

      Under Preview, a nested view of the fields and data from the HTTP response appears. You can click the links to drill down through the records in the response and view the data.

      Note: By default, all of the records that appear in the HTTP response preview are included in the data connection. To exclude data from crawling, clear the green check mark icon next to the name of the data field. The data field and any records that are nested within the field will be excluded from crawling.

    5. Select Start Crawling Now to start crawling the data source after the data source is saved.
  4. When finished, click Submit.