Creating custom connectors

This guide explains how to add and configure a new custom connector so that you can fetch the data you need from the a data source of your choice. The steps described in the Configure endpoints section of this guide also apply when editing the endpoints included in a custom connector or Flex connector. Custom connectors are connectors that you create to fetch data from external data sources that are not available as pre-built connectors in Maia. Custom connectors can only fetch data from REST APIs that return data in JSON format. They cannot fetch data in XML format, and cannot connect to self-signed HTTPS API endpoints. Flex connectors are pre-built, editable custom connectors provided in Maia that let you fetch data from frequently used external data sources. They include a curated selection of endpoints, authentication options, and parameters for third-party REST API endpoints that return data in JSON format. If a Flex connector is available for the data source you want to connect to, we recommend using this Flex connector instead of a new custom connector, which requires you to add and configure all the endpoints you want to use. You can add, edit, and remove the endpoints included in a Flex connector in the custom connector editor. For more information about custom connectors and Flex connectors, read the Custom connector overview and Flex connector overview. For instructions about configuring a custom connector in an orchestration pipeline in Designer, read Custom connectors in Designer.

---

​

Prerequisites

To create a custom connector, you’ll need:

• The base URL for the API you’re connecting to
• Credentials for the authentication method you want to use
• Information about the endpoints you want to add

---

​

Add a custom connector

1. In the left navigation, click , then Custom Connectors.
2. In the top right of the Custom connectors dashboard, click Add connector to open the custom connector editor.
3. Click the Rename connector pencil icon to the right of the custom connector’s default name, and enter a unique, descriptive name.
4. (Optional) Click the </> icon to the left of the custom connector’s name to upload a logo image for this connector. The image is used as the custom connector’s component icon on the Designer canvas. In the Update connector logo dialog, enter the URL of the image, then click Apply.

---

​

Configure endpoints

In the custom connector editor, the endpoints that have already been added to the selected custom connector are listed on the left. You can manage these endpoints as follows:

• To rename an endpoint, click the Rename endpoint pencil icon next to its name and enter a new name.
• To add another endpoint, click Add an endpoint at the bottom of the Endpoints list on the left side of the custom connector editor.
• To clone or delete an endpoint, click the three dots … next to its name to access these options.

The right side of the custom connector editor contains two top-level tabs. Complete the required fields in these tabs for each endpoint in your custom connector.

• Request, where you configure the request to send. This tab contains four sub-tabs:
  • Authentication
  • Parameters
  • Headers
  • Body
• Pagination, where you configure how the returned data is paginated.

​

Request tab

To create a request, provide the request method and URL of the endpoint you want to configure:

1. Select the request method of the endpoint you’re configuring:
   • The default request method is GET. To create a GET request, do not change the request method.
   • To create a POST request, click the GET drop-down and select POST.
2. In the URL field next to the request method, enter the URL of the endpoint to configure.

​

Authentication tab

For more information about the authentication options for custom connectors, the credentials required for each authentication method, and creating and using OAuth connections, read Custom connector authentication.

1. Click the Authentication tab.
2. In the Authentication type drop-down, select the authentication method you want to use to connect to the API. The table below provides an overview of the available authentication methods.
3. Enter the required credentials for the selected authentication method, or select the OAuth connection that you want to use to authenticate your custom connector.

​

Parameters and Headers tabs

In the Parameters tab, you can add and configure query parameters and URI parameters. In the Headers tab, you can add and configure request headers that define the HTTP(S) interaction. Each parameter type serves a different purpose:

• Query parameters are added to the end of the URI as key-value pairs. Available parameter keys and values can be obtained from the API provider—for example, adding ?currency=USD to the end of a URI queries for data where currency is USD. Query parameters can be constant or configurable.
• URI parameters are included in the endpoint URI itself, before any query parameters. They are always configurable, because they essentially let you retrieve data from different endpoint URLs using one custom connector. The request will fail if no value is set for a URI parameter.
• Header parameters are included in the request header and are used to provide additional information that doesn’t belong in the URI, such as metadata, authentication tokens, or data that shouldn’t be included in the URI for security or versioning reasons. Header parameters can be constant or configurable.

When configuring the custom connector properties in Designer, you must provide a value for all URI parameters and all configurable query and header parameters that you add to the custom connector. For more information, read Custom connectors in Designer. To add and configure a parameter:

1. Click the Parameters or Headers tab, depending on the type of parameter you want to add.
2. Click Add a query parameter, Add a URI parameter or Add a header parameter to add the corresponding parameter type to this endpoint.
3. For each parameter, complete the following fields:
   1. In Name, enter the parameter name.
   2. In Value, enter the parameter value.
   3. For query parameters and header parameters, use the Configurability toggle to set the parameter as Constant or Configurable.

To delete a parameter, click the Remove icon on the right side of its row.

​

Configurability

Each query parameter and header parameter can be Constant or Configurable. URI parameters are always Configurable. This setting affects how the value of the parameter is sent in the request:

• Constant: Always use the value set in the custom connector editor for this parameter. For example, if you set an aggregation period parameter to Constant, you will always fetch data for the specified time period using this custom connector.
• Configurable: Set the value of this parameter when the custom connector is used in a pipeline. This lets you use a variable to set the value of the configurable parameter each time the custom connector is used in a pipeline. For example, if you set an account ID parameter to Configurable, you can use the custom connector to fetch data from different accounts.

​

Body tab

This field is only enabled for POST requests. Enter any data to be sent from the client to the API. Like URI and Header parameters, the Body field supports configurable values in the form of pipeline or project variables using the standard ${variable} syntax. For more information, read Variables. A request body is only required for POST requests. If you’re configuring a GET request, the field isn’t editable and you don’t need to enter a request body.

​

Pagination tab

If you’re fetching large datasets using a custom connector or Flex connector, we recommend using pagination to make sure that your request fetches the full dataset and to prevent errors that may occur if the connector tries to fetch the whole dataset at once. For example, an API might limit responses to 100 records per request. Without any pagination, your connector only retrieves the first 100 records. With pagination, the connector automatically retrieves each subsequent page until the full dataset is returned. In the Pagination tab, you can set your pagination method. The pagination method determines how multiple pages of data are retrieved from the API. The custom connector editor provides pre-built pagination methods that you can configure. You can also select Script pagination, which lets you script your own custom pagination method with control over pagination, rate limiting, and API request and response management.

1. Click the Pagination tab.
2. Click the pagination method you want to use:
   • No Pagination: No pagination—fetches all data without splitting it into pages.
   • Relative Path: Navigates to the next page using the relative URI provided in the response.
   • Full Path: Navigates to the next page using the complete URI provided in the response.
   • Page Based: Navigates to the next page by incrementing a page parameter, for example ?page=1.
   • Link Header: Navigates between pages using the link header in the response headers.
   • Offset: Navigates to the next page by incrementing a query parameter, for example ?offset=0&limit=10. This method is often used with SQL-like databases.
   • Cursor: Navigates between pages using a cursor parameter, for example ?page_size=10&cursor=id001.
   • Script: Lets you script your own custom pagination method.
3. Configure your selected pagination method as explained in the corresponding tab below.
   • No pagination
   • Relative path
   • Full path
   • Page based
   • Link header
   • Offset
   • Cursor
   • Script

   This is the default setting for new custom connectors and doesn’t require any configuration.When No Pagination is selected, all data is fetched, without splitting it into pages. This is suitable for small datasets that can be fetched with a single request.

   ​

   Base URI

   required

   Enter the base URI to exclude from the relative path.

   ​

   Next page URI

   required

   Select the property in the response that indicates the next page. The drop-down contains available properties in the response schema.

   ​

   Page size parameter

   Enter the query parameter that specifies the page size.

   ​

   Page size

   required

   Enter the number of records to return per page.

   ​

   Next page URI

   required

   Select the property in the response that indicates the next page. The drop-down contains available properties in the response schema.

   ​

   Page size parameter

   Enter the query parameter that specifies the page size.

   ​

   Page size

   required

   Enter the number of records to return per page.

   ​

   Page size parameter

   Enter the query parameter that specifies the page size.

   ​

   Page size

   required

   Enter the number of records to return per page.

   ​

   Page number parameter

   required

   Enter the query parameter that specifies the page number.

   ​

   Last page

   Select the property in the response that indicates the last page. The drop-down contains available properties in the response schema.

   ​

   Page size parameter

   Enter the query parameter that specifies the page size.

   ​

   Page size

   required

   Enter the number of records to return per page.

   Offset pagination parameters must be configured in the Pagination tab as described here. Do not configure them in the Request tab—this will cause test requests for this custom connector to fail and prevent the paging mechanism from working correctly.

   ​

   Limit parameter

   Enter the query parameter that specifies the page limit.

   ​

   Limit

   required

   Enter the number of records to return per page.

   ​

   Record count

   required

   Select the property in the response that counts the total records. The drop-down contains available properties in the response schema.Alternatively, use a boolean to indicate whether there are more records, such as has_more = true or false.

   ​

   Offset parameter

   Enter the query parameter that specifies the page offset.

   ​

   Pagination method

   required

   Select the pagination method to use:

   • Query parameter: (Default) Extracts the cursor from the JSON response and sends it as a query parameter in the request.
   • Header: Extracts the cursor from a header in the response and sends it as a header in the request.

   • Header
   • Query parameter

   ​

   Cursor source header name

   required

   Enter the name of the header parameter to extract the cursor from.

   ​

   Cursor header name

   required

   Enter the name of the cursor header parameter to send in the request.

   ​

   Page size header

   Enter the header parameter that specifies the page size.

   ​

   Page size

   required

   Enter the number of records to return per page.

   ​

   Cursor

   required

   Select the property in the response to extract the cursor from. The drop-down contains available properties in the response schema.

   ​

   More pages indicator key

   Select the property in the response that indicates whether there are more pages. The drop-down contains available properties in the response schema.

   ​

   Cursor parameter name

   required

   Enter the name of the cursor query parameter to send in the request.

   ​

   Page size parameter

   Enter the query parameter that specifies the page size.

   ​

   Page size

   required

   Enter the number of records to return per page.

   Script pagination lets you use a scripting language to write your own pagination method, giving you full control over pagination, rate limiting, and API request and response management. For more information, read Script paging in custom connectors.
4. (Optional) Click Test pagination to test your pagination settings.

---

​

Send a test request

Click Send to make a test request. After making a test request, the status code of the request is shown in the bottom right of the custom connector editor, for example 200 (success). Click the tabs at the bottom of the custom connector editor to view and configure the following details.

​

Response

Click the Response tab to view the response, structure, and headers. You can’t edit the raw response or headers, but you can edit the structure to configure the API’s response. This structure is used when loading your data into your destination. Make sure that the structure shown here accurately represents the API’s response to ensure that your data is loaded correctly. To edit the structure:

1. In the Response tab, click the Configure structure icon in the top right of the Structure section.
   If you’re editing an existing custom connector or Flex connector, you may need to unlock the structure to edit it. To do this, click the Structure is locked icon in the top right of the Structure section.
2. The Configure the structure dialog shows the API response as a tree structure containing each column’s name and data type. Use the arrow icons to expand and collapse nested parts of the structure. Click the three dots next to any element in the structure to make changes:
   • When editing an array or nested element:
     • Click Set as selected data to set this array or nested element as the part of the schema the connector will fetch data from.
     • Click Add element to add a new element inside this array or nested element.
   • When editing any element:
     • Click Edit element to change the element’s name or data type.
     • Click Delete element to remove this element from the API response.
3. Click Apply.
4. (Optional) Click the Structure is unlocked icon in the top right of the Structure section to lock the structure.

---

​

Save your connector

Click Save in the top right of the custom connector editor to save your custom connector. It will appear in the list of available custom connectors in the Custom connectors dashboard, and can be added to orchestration pipelines in Designer like any other connector. For information about configuring custom connector properties in orchestration pipelines, read Custom connectors in Designer.