Script paging in custom connectors

One of the available pagination methods for custom connectors and Flex connectors is Script. Script paging is a powerful scripting language that lets you specify how the connector will fetch and process data. It gives you control over pagination, rate limiting, and API request and response management. When you fetch data using a connector that uses Script paging, the connector fetches and paginates data as follows:

Maia sends your request to the endpoint URI, passing all header, query, and URI parameters.
The first page of results is retrieved, based on the parameters set in the connector configuration.
The pagination script is then executed sequentially, one line at a time. Depending on the statements in the script, this may modify parameters in the request sent, and in the response header and response body that was retrieved.
The next page of results is retrieved, based on the parameters set in the connector configuration, which may have been modified by the pagination script.
The pagination script is executed again.
The previous two steps repeat until the pagination script determines that the connector should stop retrieving data or that there is no more data to retrieve.

Script paging is not a full programming language. It’s a simple scripting language with a limited set of operations that you can use to control pagination, rate limiting, and API request and response management. As a result, Script paging doesn’t support programming constructs such as loops or conditionals. You can only use the syntax explained in this guide in Script pagination scripts.

Using Script paging

To set a custom connector or Flex connector to use Script paging:

If you’re creating a new connector, follow the steps in Creating custom connectors until you reach the Pagination step. If you’re editing an existing connector, click the Pagination tab in the custom connector editor.
Select the Script pagination method.
Write your pagination script in the text field, following the syntax rules in this guide.

Parameters in pagination scripts

Script paging operations can use a connector’s query parameters, URI parameters, and header parameters to control how the response is paginated. For example, the following endpoint URI specifies values for the limit and offset query parameters:

https://api.mydataprovider.com/v3/mydataset?limit=5&offset=0

In your pagination script, you can use these parameter values in expressions and write new values to them. For example:

// Get the current value of the offset parameter
var offset = @request.query.get("offset");

// Increment the offset parameter value by 5
var nextPageOffset = offset + 5;

// Use the new value for the offset parameter
@request.query.put("offset", nextPageOffset);

Script paging syntax

Script pagination scripts are composed of multiple statements. Each statement typically performs a single action, such as setting a variable, modifying a request, or processing a response. Each statement must be on a new line and end with a semicolon ;. Some statements are operations—a specific type of statement that begins with the @ symbol and calls a predefined action, such as @log(...) or @pager.stop(...). For more information about operations, read Script paging operations and JSON manipulation operations.

Comments

You can include comments in your scripts to explain the logic or to temporarily disable code.

To add a single-line comment, write // at the start of the line, followed by your comment.
To add a multi-line comment, enclose the comment in /* and */.

For example, this syntax example contains a single-line comment with a “to-do” note, then a variable definition, then an operation, and then a multi-line comment explaining the operation:

// To-do: check this with the team

var contents = @response.body.get("/0");
@pager.stop(contents == "");

/*
If body is empty:
Stop retrieving pages
*/

Variables

Variables are defined by the keyword var followed by the variable name and the value assigned to it. Variables can have one of four data types:

String: A string of characters. String values must be enclosed in quotes " ".
Boolean: Boolean values must only be true or false.
Number: An integer with or without a decimal point.
Array: An array of strings, booleans, or numbers. An array can only contain values of the same data type.

You don’t have to explicitly assign a data type to a variable. Script paging automatically determines the right data type for a variable based on the value(s) assigned to it. Example variable definitions:

//String variable
var contentType = "application/json";

// Number variable
var pageNumber = 2;

// Boolean variables
var isAuthorized = true;
var isValid = false;

// Array variables
var numberSet = [2, 5, 5];
var nameList = ["john", "mary", "brian"];
var flagList = [true, false, false];

You can declare a variable that has the same value as another variable. In this example, the value of the copyOfContentType variable will always be the same as the value of the contentType variable:

var contentType = "application/json";
var copyOfContentType = contentType;

Arithmetic statements

Script pagination scripts can perform arithmetic directly within variable assignments, for example:

var sum = 10 + 5;
var product = sum * 3;
var quotient = product / 2;
var difference = quotient - 1;

String statements

To concatenate strings or string variables, use the + operator. You can concatenate any number of strings or string variables in a single statement. If a string contains quote marks ", they must be escaped with the \ character when assigned to a string variable: For example:

// Concatenate multiple strings
var fullName = "Joe" + " " + "Bloggs";

// Concatenate a variable and a string
var sentence = fullName + " is a data engineer";

// Concatenate a variable and a string containing quote marks
var myString = fullName + " said \"hello\" to me";

Comparison statements

Comparison statements must be written in the format <value> <operator> <value>. Comparison statements are used in the pager.stop operation, which will evaluate an expression to determine if there are more pages to retrieve, as shown in the example below. Script pagination scripts support the following comparison operators, which you can use to compare literal values and variables:

Equal to ==
Not equal to !=
Less than <
Less than or equal to <=
Greater than >
Greater than or equal to >=

// Comparing literal values and variables
fullName == "Joe Bloggs"
pageNumber < 1000

// Comparing two variables
pageSizeNum == pageSize + 0

// Comparison statement used in the pager.stop operation so that paging will stop when page number 1000 is reached
@pager.stop(pageNumber == 1000);

Increment and decrement statements

Increment statements increase a number. The incremented value should be assigned to a new variable.

var counter = 5;
var incrementedCounter = counter++;

Decrement statements decrease a number. The decremented value should be assigned to a new variable.

var counter = 5;
var decrementedCounter = counter--;

Debug logging

You can add debug logging using the @log(...) operation to help debug your pagination scripts. The arguments passed in the @log(...) operation can include strings and variables. The messages returned by debug logging are shown in different places according to where you’re using the connector:

If you’re configuring or editing a custom connector or Flex connector, the messages are shown in the Logs tab at the bottom of the custom connector editor when you send a test request.
If you run the custom connector or Flex connector in a pipeline in Designer, the messages are shown in the Logging tab when you view details about a task.

Using debug logging in your pagination script helps you identify issues, such as values not being extracted correctly from the response or your @pager.stop rule not returning true when expected, as shown in the following examples.

// Log individual variables and the final result when building a full name from two strings
@log("Build a full name");

var firstName = "Joe";
var surname = "Bloggs";
var fullName = firstName + " " + surname;

@log("Making name from components - ", firstName, ":", surname);
@log("Full Name: ", fullName);

// Log the number of pages returned by the `@pager.pageCount()` operation
var pages = @pager.pageCount();
@log("pages:", pages);

Script paging operations

Script pagination supports operations that allow you to define how responses from endpoints are processed. This is what makes the Script pagination method so flexible. Operations begin with the @ symbol. An operation may return data that you can assign to a variable, or perform an action based on the result of an evaluated expression. The following sub-sections of this guide contain an explanation of what each operation does, a table listing any parameters to configure, and one or more examples showing how to configure the operation.

Returns the number of pages fetched. You can assign the returned number to a variable for use later in the script.

// Get the count of pages and assign it to the variable "pages"

var pages = @pager.pageCount();

Tells the connector not to attempt to fetch the next page if the expression evaluates to true.

Parameter	Type	Description
expression	string	A valid Script pagination comparison statement that will be evaluated to produce a `true` or `false` result.

// Stop fetching pages if the pages variable equals 1000

@pager.stop(pages == 1000);

// Stop fetching pages if the @pager.pageCount() operation returns a number greater than 1000

@pager.stop(@pager.pageCount() > 1000);

response.header.get(key)

Returns the value of the specified key in a key-value pair in the response header.

Parameter	Type	Description
key	string	The key that you want the value of.

// Return the X-Pagination value from the response header

var pagingJson = @response.header.get("X-Pagination");

response.header.getNextLink()

Gets the next page link from the response header of an API that uses link header paging. This works for any API that uses a standard link header paging model, for example the GitHub API illustrated in Example 6.

// Assign the next page link to the variable "nextLink"

var nextLink = @response.header.getNextLink();

response.header.getLastLink()

Gets the last page link from the response header of an API that uses link header paging. This works for any API that uses a standard link header paging model, for example the GitHub API illustrated in Example 6.

// Assign the last page link to the variable "lastLink"

var lastLink = @response.header.getLastLink();

response.body.get(key)

Returns the value of the specified key in a key-value pair in the response body.

Parameter	Type	Description
key	string	The key that you want the value of. You must specify the key location using JSON Pointer notation, as defined in RFC 6901.

// Get the value of "/data/name" from the response body and assign it to the variable "firstName"

var firstName = @response.body.get("/data/name");

response.status.get()

Return the value of the response status code as a number.

// Assign the status code to the variable "status"

var status = @response.status.get();

request.header.put(key, value)

Adds the specified key and value to the request header parameters.

Parameter	Type	Description
key	string	The name of the key that the value will be assigned to.
value	string	The value that will be assigned to the key.

// Assign the value "1" to the key "page"

@request.header.put("page", "1");

request.header.remove(key)

Removes the value of the specified key from request header parameters.

Parameter	Type	Description
key	string	The name of the key that will be removed.

// Remove the value of the key "page" from the request header

@request.header.remove("page");

request.header.clear()

Removes all existing values from the request header parameters.

// Remove all values from the request header

@request.header.clear();

request.header.get(key)

Returns the value of the specified key in a key-value pair in the request header.

Parameter	Type	Description
key	string	The key that you want the value of.

// Get the value of "name" from the request header and assign it to the variable "firstName"

var firstName = @request.header.get("name");

request.query.put(key, value)

Adds the specified key and value to the request query parameters.

Parameter	Type	Description
key	string	The name of the key that the value will be assigned to.
value	string	The value that will be assigned to the key.

// Assigns the string "json" as the value of the "format" key

@request.query.put("format", "json");

request.query.remove(key)

Removes the value of the specified key from request query parameters.

Parameter	Type	Description
key	string	The name of the key that will be removed.

// Remove the value of the key "page" from the request query

@request.query.remove("page");

request.query.clear()

Removes all existing values from the request query parameters.

// Remove all values from the request URI query parameters

@request.query.clear();

request.query.get(key)

Returns the value of the specified key in a key-value pair in the request query parameters.

Parameter	Type	Description
key	string	The key that you want the value of.

// Get the value of "name" from the request query and assign it to the variable "firstName"

var firstName = @request.query.get("name");

request.body.set(jsonString)

Sets the value of a key-value pair in the request body. This is used with JSON body format.

Parameter	Type	Description
jsonString	string	A valid JSON string that contains a key-value pair. You must specify the key location using JSON Pointer notation, as defined in RFC 6901. Quotes in the string must be escaped as `\"`.

// Set the JSON string in a variable, then pass that variable to the operation
var jsonString = "{ \"key\": \"value\" }";
@request.body.set(jsonString);

request.body.put(key, value)

Adds a new key-value pair to the request body. This is used with JSON body format.

Parameter	Type	Description
key	string	The name of the key that you want to add.
value	string	The value assigned to the key.

// Put the key "name" in the request body and assign the value of the variable "firstName" to it

@request.body.put("name", firstName);

request.body.remove(key)

Removes the value of the specified key from the request body. This is used with JSON body format.

Parameter	Type	Description
key	string	The name of the key that you want to remove.

// Remove the value of the key "name" from the request body

@request.body.remove("name");

request.body.clear()

Clears all existing values from the request body. This is used with JSON body format.

// Clear all values from the request body

@request.body.clear();

request.body.get(key)

Returns the value of the specified key in a key-value pair in the request body. This is used with JSON body format.

Parameter	Type	Description
key	string	The key that you want the value of.

// Get the value of "name" from the request body and assign it to the variable "firstName"

var firstName = @request.body.get("name");

request.uri.set(URI)

Sets the value of the URI.

Parameter	Type	Description
URI	string	The URI to set.

// Set the request URI to https://api.mydataprovider.com

@request.uri.set("https://api.mydataprovider.com")

request.uri.append(path)

Appends a string to the request URI. Typically used to append a relative path to a base URI.

Parameter	Type	Description
path	string	The path to append to the request URI.

// Append the string "/pagination/next" to the request URI

@request.uri.append("/pagination/next");

request.uri.replace(parameter, value)

Replaces parameterized values in the URI path. The parameterized values are set as URI parameters.

Parameter	Type	Description
parameter	string	The name of the parameter to be replaced.
value	string or number	The value to replace the parameter with.

// Replace the id parameter with the value 5 in the URI https://api.com/user/{id}

@request.uri.replace("id", 5)

request.ratelimit.set(allowlist, header, wait, retry)

Configures the rate limit for the endpoint. Suggested values for rate limits can usually be found in the API’s documentation.

Parameter	Type	Description
allowlist	array of numbers	For example, `[429, 403]`. These are the status codes for suppressing the rate limit error.
header	string	The response header key whose presence indicates throttled response.
wait	number	Sets the wait in milliseconds, which should be extracted from the response.
retry	number	Sets number of retries.

// Set the ratelimit

@request.ratelimit.set([403, 329], "x-ratelimit-reset", 3600000, 5);

JSON manipulation operations

json.put(key, value, jsonObject)

Adds a key-value pair to a JSON object. The updated JSON object value should be assigned to a new variable.

Parameter	Type	Description
key	string	The key to add or update in the form of a JSON pointer.
value	string	The value to assign.
jsonObject	string	The JSON object to which the key-value pair will be added.

// Add the key "name" to the JSON object "myObject" and assign the value of the variable "firstName" to it

var updatedJson = @json.put("name", firstName, myObject);

json.remove(key, jsonObject)

Removes a key from a JSON object. The updated JSON object value should be assigned to a new variable.

Parameter	Type	Description
key	string	The key to remove in the form of a JSON pointer.
jsonObject	string	The JSON object from which the key will be removed.

// Remove the key "name" from the JSON object "myObject"

var updatedJson = @json.remove("name", myObject);

json.get(path, object)

Extracts a value from a JSON object. You need to know the path to the object in the JSON structure.

Parameter	Type	Description
path	string	The JSON path to the object. Specify the path using `/` to separate the names of nested elements, with `/` by itself indicating a root-level element.
object	string	The JSON object to extract the value from.

Assuming the following JSON structure:

{
  "total_count": 4672,
  "pages": {
    "type": "pages",
    "next": {
      "page": 2,
    },
  }
}

// Assign the JSON object to a variable
var response = @response.header.get("X-Pagination");

// Extract the value of "page"
var page = @json.get("/pages/next/page", response)

// Extract the value of total_count
var count = @json.get("/total_count", response)

Examples

The following examples show some common uses for Script pagination. In each case, we give a real-world API call and the response returned by that call. We then show a suggested script to paginate that response.

Example 1: relative path

In this example, each page of data retrieved from the API endpoint provides us with a relative path which points to the next page we need to retrieve. We can use a script to read that path from each page and use it to retrieve the next page. The following URI retrieves filtered data in JSON format:

https://api.coronavirus.data.gov.uk/v1/data?filters=areaName=England&format=json

This retrieves a page of data with the following structure:

{
  "data": [],
  "pagination": {
    "current": "/v1/data?filters=areaName=England&format=json&page=1",
    "next": "/v1/data?filters=areaName=England&format=json&page=2",
    "previous": null,
    "first": "/v1/data?filters=areaName=England&format=json&page=1",
    "last": "/v1/data?filters=areaName=England&format=json&page=1"
  }
}

We can use the following script to query that data structure and extract from it the path to the next page of data, appending that path to the URI used in the next call to the endpoint. As this script is executed after each page is retrieved, we continue to fetch each subsequent page until we reach the last page.

@request.uri.set("https://api.coronavirus.data.gov.uk");
var nextPage = @response.body.get("/pagination/next");
@request.uri.append(nextPage);
@pager.stop(nextPage == "");

Example 2: full path

This is similar to the last example, in that each page of data contains a pointer to the next page of data. However, the pointer is a full URI, not a relative path. This makes our script simpler, as we don’t have to concatenate the different parts of a URI. Use the following URI to retrieve the first page of data:

https://www.zopim.com/api/v2/chats

This returns data with the following structure:

{
  "data": [],
  "count": 1016,
  "next_url": "https://www.zopim.com/api/v2/chats?cursor=eyJjb3VudCI6IDc"
}

A simple script extracts the next page URI and uses it to fetch the next page of data:

var nextPage = @response.body.get("/next_url");
@request.uri.set(nextPage);
@pager.stop(nextPage == "");

Example 3: page based

An endpoint which uses page-based pagination requires incrementing a page parameter to retrieve each subsequent page. Each page of data contains its page number in the response structure, meaning that we need to read that number, increment it, and query again with the incremented page number. In the following URI, we are telling the endpoint to send us data starting at page 1:

https://randomuser.me/api?page=1&results=10&format=json

The first page of data is returned with the following structure:

{
  "data": [],
  "info": {
    "seed": "faf4b0a59eae416e",
    "results": 10,
    "page": 1,
    "version": "1.4"
  }
}

Our script needs to extract the value of the current page from the response body, add 1 to it, and construct a new query which includes the new page number as a parameter. We will use the script to stop data retrieval after 50 pages:

var currentPage = @response.body.get("/info/page");
var nextPage = currentPage + 1;

@request.query.put("page", nextPage);
@request.query.put("results", "10");
@request.query.put("format", "json");

var pageNumber = @response.body.get("/info/page");
@pager.stop(pageNumber == 50);

Example 4: cursor based

Cursor pagination uses a cursor parameter to navigate between pages. We need a script that reads the value of the next page cursor and puts that into the next query. Use the following URI to retrieve a page of data:

https://api.intercom.io/contacts?per_page=150

The response has the following structure:

{
  "data": [],
  "total_count": 4672,
  "pages": {
    "type": "pages",
    "next": {
      "page": 2,
      "starting_after": "WzE2NjI1MzU0OTgwMDAsIjYyNGFkMDVlOWJkOTg5MWFlYzVlYzI0ZSIsMl0="
    },
    "page": 1,
    "per_page": 150,
    "total_pages": 32
  }
}

The cursor that points to the next page is starting_after. The following script will extract this and put it into the next page query:

var cursor = @response.body.get("/pages/next/starting_after");
@request.query.put("starting_after", cursor);
@request.query.put("per_page", 150);

var totalPages = @response.body.get("/pages/total_pages");
var pageNumber = @response.body.get("/pages/page");
@pager.stop(totalPages == pageNumber);

Example 5: offset

Offset pagination involves paging by incrementing a query parameter. We need to set the parameter in our initial query, and then use a script to increment the parameter for each subsequent page. Use the following URI to retrieve the first page of data:

https://api.spacexdata.com/v3/launches?limit=5&offset=0

This data has the following structure:

[
    {
        "flight_number": 1,
        "mission_name": "FalconSat",
        "mission_id": [],
        "upcoming": false,
        "launch_year": "2006"
    }
]

As there is no paging data in the response, the following script takes the offset value from the original query paramter and increments it by a fixed amount to retrieve the next page of data. Sensible offset numbers can usually be found in the API’s documentation. Note that because nothing in the response tells us when we’ve reached the last page, we’re checking for when an empty response body is returned, as this will tell us we’ve reached the end of the data.

var currentOffset = @request.query.get("offset");
var newOffset = currentOffset + 5;
@request.query.put("offset", newOffset);

// If the body is an empty array, stop paging
var contents = @response.body.get("/0");
@pager.stop(contents == "");

Example 6: link header

Link header pagination uses a field in the response header to point to the next page, as in the following example:

Link=<https://api.github.com/repositories/1300192/issues?page=2>; rel="next", <https://api.github.com/repositories/1300192/issues?page=572>; rel="last"

We can use the following script to retrieve the link to the next page. If we also retrieve the link to the last page, we can then compare the two and stop paging when we reach the last page:

var nextLink = @response.header.getNextLink();
var lastLink = @response.header.getLastLink();
@request.uri.set(nextLink);
@pager.stop(nextLink == lastLink);

GraphQL paging example

API endpoint: https://organizationapi.com/graphql The API integrates pagination directly within the query structure. This requires us to parameterize the page value in the GraphQL query and to pass the page value in the variables. Example query to fetch customers:

query ($page: Int) {
  customers(page: $page) {
    info {
      next
      pages
    }
    results {
      id
      name
    }
  }
}

Example response (truncated):

{
  "data": {
    "customers": {
      "info": {
        "count": 107,
        "next": 2,
        "pages": 6,
      },
      "results": []
    }
  }
}

Solution 1: POST request

A POST request using this JSON POST body containing the query and variables:

{
  "query": "query ($page: Int!) {customers(page: $page, filter: {name: "rick"}) {info {count next pages} results {created gender id}}}",
  "variables": {"page": 1}
}

Implement the paging script:

// Extract current page value
var currentPage = @pager.pageCount();

// Increment value for next page request
var nextPage = currentPage++;

// Update the page value in the request body's variables
@request.body.put("/variables/page", nextPage);

// Get the total number of pages
var totalPages = @response.body.get("/data/customers/info/pages");

// Stop paging when the current page value is the same as total pages
@pager.stop(currentPage == totalPages);

Solution 2: GET request

A GET request using the query and variables as query parameters:

https://organizationapi.com/graphql?query=query($page:Int!){customers(page:$page,filter:{name:"rick"}){info{count next pages}results{created gender id}}}&variables={"page":1}

Implement the paging script:

// Extract current page query parameter value (JSON object)
var currentVariables = @request.query.get("variables");

// Get the current page value from the currentVariables JSON object
var currentPage = @json.get("/page", currentVariables);

// Increment current page value for next request
var nextPage = currentPage++;

// Update the variables in the query parameter
var nextVariables = @json.put("/page", nextPage, currentVariables);

// Replace the variables query parameter value with updated JSON object
@request.query.put("variables", nextVariables);

// Get the total number of pages
var totalPages = @response.body.get("/data/customers/info/pages");

// Stop paging when the current page value is the same as total pages
@pager.stop(currentPage == totalPages);

​Using Script paging

​Parameters in pagination scripts

​Script paging syntax

​Comments

​Variables

​Arithmetic statements

​String statements

​Comparison statements

​Increment and decrement statements

​Debug logging

​Script paging operations

​pager.pageCount()

​pager.stop(expression)

​response.header.get(key)

​response.header.getNextLink()

​response.header.getLastLink()

​response.body.get(key)

​response.status.get()

​request.header.put(key, value)

​request.header.remove(key)

​request.header.clear()

​request.header.get(key)

​request.query.put(key, value)

​request.query.remove(key)

​request.query.clear()

​request.query.get(key)

​request.body.set(jsonString)

​request.body.put(key, value)

​request.body.remove(key)

​request.body.clear()

​request.body.get(key)

​request.uri.set(URI)

​request.uri.append(path)

​request.uri.replace(parameter, value)

​request.ratelimit.set(allowlist, header, wait, retry)

​JSON manipulation operations

​json.put(key, value, jsonObject)

​json.remove(key, jsonObject)

​json.get(path, object)

​Examples

​Example 1: relative path

​Example 2: full path

​Example 3: page based

​Example 4: cursor based

​Example 5: offset

​Example 6: link header

​GraphQL paging example

​Solution 1: POST request

​Solution 2: GET request

Using Script paging

Parameters in pagination scripts

Script paging syntax

Comments

Variables

Arithmetic statements

String statements

Comparison statements

Increment and decrement statements

Debug logging

Script paging operations

pager.pageCount()

pager.stop(expression)

response.header.get(key)

response.header.getNextLink()

response.header.getLastLink()

response.body.get(key)

response.status.get()

request.header.put(key, value)

request.header.remove(key)

request.header.clear()

request.header.get(key)

request.query.put(key, value)

request.query.remove(key)

request.query.clear()

request.query.get(key)

request.body.set(jsonString)

request.body.put(key, value)

request.body.remove(key)

request.body.clear()

request.body.get(key)

request.uri.set(URI)

request.uri.append(path)

request.uri.replace(parameter, value)

request.ratelimit.set(allowlist, header, wait, retry)

JSON manipulation operations

json.put(key, value, jsonObject)

json.remove(key, jsonObject)

json.get(path, object)

Examples

Example 1: relative path

Example 2: full path

Example 3: page based

Example 4: cursor based

Example 5: offset

Example 6: link header

GraphQL paging example

Solution 1: POST request

Solution 2: GET request