This page describes how to configure the Datadog connector component as part of a data pipeline within . The Datadog component uses the Connect and Configure parameters to create a table of Datadog data, which is then stored in your preferred storage location (Snowflake, Databricks, Amazon Redshift, or cloud storage). You do not need to use the Create Table component when using this connector, as the Datadog component will create a new table or replace an existing table for you using the Destination parameters you define.
The Datadog connector is a Flex connector. In , Flex connectors let you connect to a curated set of endpoints to load data.
You can use the Datadog connector in its preconfigured state, or you can edit the connector by adding or amending available Datadog endpoints as per your use case. You can edit Flex connectors in the Custom Connector user interface.
For detailed information about authentication, specific endpoint parameters, pagination, and more aspects of the Datadog API, read the Datadog API documentation.
Properties
Reference material is provided below for the Connect, Configure, Destination, and Advanced Settings properties.
A human-readable name for the component.
Connect
The data source to load data from in this pipeline. The drop-down menu lists the Datadog API endpoints available in the connector. For detailed information about specific endpoints, read the Datadog API documentation. The authentication method to authorize access to your Datadog data. Currently supports API Key.
The key of a working API key:value pair.Enter DD-API-KEY in the Key field.All endpoints (listed below), also need an Application key, which uses the header DD-APPLICATION-KEY. For more information, read Getting started. Use the drop-down menu to select the corresponding secret definition that denotes the value of a working API key:value pair.Read Secrets and secret definitions to learn how to create a new secret definition.Read the Datadog API documentation to learn how to acquire an API key.
- Parameter Name: The name of a URI parameter.
- Parameter Value: The value of the corresponding parameter.
| Required parameter | Endpoints | Description |
|---|
| VERSION | Get All Dashboard Lists, Get All Downtimes, Search Monitors, Get All Monitor Details, Get All Synthetics Tests, Get All Dashboards, Get List of Events, Get All Hosts, Get All SLOs | v1 |
| VERSION | Get All Teams, List All Users, Get All API Keys, Get All Application Keys | v2 |
- Parameter Name: The name of a query parameter.
- Parameter Value: The value of the corresponding parameter.
| Required parameter | Endpoints | Description |
|---|
| query | Search Monitors | After entering a search query in your Manage Monitor page, use the query parameter value in the URL of the page as value for this parameter. Consult the dedicated manage monitor documentation page to learn more. The query can contain any number of space-separated monitor attributes, for instance query="type:metric status:alert". |
| name | Get All Monitor Details | A string to filter monitors by name. |
| tags | Get All Monitor Details | A comma-separated list indicating what tags, if any, should be used to filter the list of monitors by scope. Enter env:prod. |
| page | Get All Monitor Details | The page to start paginating from. If this argument is not specified, the request returns all monitors without pagination. |
| page | Get List of Events | By default, 1000 results are returned per request. Set page to the number of the page to return with 0 being the first page. The page parameter can only be used when either unaggregated or exclude_aggregate is set to true. |
| page_size | Get All Monitor Details | The number of monitors to return per page. If the page argument is not specified, the default behavior returns all monitors without a page_size limit. However, if page is specified and page_size is not, the argument defaults to 100. For example, enter 200. |
| start | Get List of Events | Enter a POSIX timestamp for the start of the list, for example 1688601600. |
| end | Get List of Events | Enter a POSIX timestamp for the end of the list, for example 1688601600. |
| exclude_aggregate | Get List of Events | Set exclude_aggregate to true to only return unaggregated events with is_aggregate=false in the response. If the exclude_aggregate parameter is set to true, then the unaggregated parameter is ignored and will be true by default. Enter true. |
| filter | Get All Hosts | String to filter search results. |
- Parameter Name: The name of a header parameter.
- Parameter Value: The value of the corresponding parameter.
| Required parameter | Endpoints | Description |
|---|
| DD-APPLICATION-KEY | All endpoints | Use this application key to authenticate to the Datadog API. For more information, read Application keys. |
| Content-Type | Search Logs | application/json |
A JSON body to include as part of a POST request. Use Custom Connector to test your endpoints work as expected before moving to pipelines.You should also consult the developer documentation for the API you’re connecting to—as the developer portal may provide additional information about endpoints and requests.For the Search Logs endpoint, include the following POST Body:{
"filter": {
"query": "env:prod service:myservice \"some search text\" || \"a different search text\" ",
"indexes": [
"main"
],
"from": "2023-07-21T10:00:36+01:00",
"to": "2023-08-21T10:45:36+01:00"
},
"sort": "timestamp",
"page": {
"limit": 150
}
}
A numeric value to limit the maximum number of records per page.
Destination
Select your cloud data warehouse.
Snowflake
Databricks
Amazon Redshift
- Snowflake: Load your data into Snowflake. You’ll need to set a cloud storage location for temporary staging of the data.
- Cloud Storage: Load your data directly into your preferred cloud storage location.
Click either the Snowflake or Cloud Storage tab on this page for documentation applicable to that destination type.The Snowflake warehouse used to run the queries. The special value [Environment Default] uses the warehouse defined in the environment. Read Overview of Warehouses to learn more. The Snowflake schema. The special value [Environment Default] uses the schema defined in the environment. Read Database, Schema, and Share DDL to learn more. The name of the table to be created.
- Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
- Truncate and Insert: If the specified table name already exists, all rows within the table will be removed and new rows will be inserted per the next run of this pipeline.
- Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
- Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
- Yes: Staged files will be destroyed after data is loaded. This is the default setting.
- No: Staged files are retained in the staging area after data is loaded.
Select the stage access strategy. The strategies available depend on the cloud platform you select in Stage Platform.
- Credentials: Connects to the external stage (AWS, Azure) using your configured cloud provider credentials. Not available for Google Cloud Storage.
- Storage Integration: Use a Snowflake storage integration to grant access to Snowflake to read data from and write to a cloud storage location. This will reveal the Storage Integration property, through which you can select any of your existing Snowflake storage integrations.
Choose a data staging platform using the drop-down menu.
- Amazon S3: Stage your data on an AWS S3 bucket.
- Snowflake: Stage your data on a Snowflake internal stage.
- Azure Storage: Stage your data in an Azure Blob Storage container.
- Google Cloud Storage: Stage your data in a Google Cloud Storage bucket.
Click one of the tabs below for documentation applicable to that staging platform. Amazon S3
Snowflake
Azure Storage
Google Cloud Storage
Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.
Select the Snowflake internal stage type. Use the Snowflake links provided to learn more about each type of stage.
- User: Each Snowflake user has a user stage allocated to them by default for file storage. You may find the user stage convenient if your files will only be accessed by a single user, but need to be copied into multiple tables.
- Named: A named stage provides high flexibility for data loading. Users with the appropriate privileges on the stage can load data into any table. Furthermore, because the stage is a database object, any security or access rules that apply to all objects will apply to the named stage.
Named stages can be altered and dropped. User stages cannot. Select your named stage. Read Creating a named stage to learn how to create a new named stage.There is a known issue where named stages that include special characters or spaces are not supported.
Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.
Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview. Select the stage access strategy. The strategies available depend on the cloud platform you select in Stage Platform.
- Storage Integration: Use a Snowflake storage integration to grant access to Snowflake to read data from and write to a cloud storage location. This will reveal the Storage Integration property, through which you can select any of your existing Snowflake storage integrations.
Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.
Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.
- Append Files in Folder: Appends files to storage folder. This is the default setting.
- Overwrite Files in Folder: Overwrite existing files with matching structure.
See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:| Configuration | Description |
|---|
| Append files in folder with defined folder path and file prefix. | Files will be stored under the structure uniqueID/timestamp-partX where X is the part number, starting from 1. |
| Append files in folder without defined folder path and file prefix. | Files will be stored under the structure folder/prefix-timestamp-partX where X is the part number, starting from 1. |
| Overwrite files in folder with defined folder path and file prefix. | Files will be stored under the structure folder/prefix-partX where X is the part number, starting from 1. All files with matching structures will be overwritten. |
| Overwrite files in folder without defined folder path and file prefix. | Validation will fail. Folder path and file prefix must be supplied for this load strategy. |
The folder path of the written files.
A string of characters to include at the beginning of the written files. Often used for organizing database objects.
A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.Click the tab that corresponds to your chosen cloud storage service.
Amazon S3
Azure Storage
Google Cloud Storage
Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview. Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.
- Databricks: Load your data into Databricks. You’ll need to set a cloud storage location for temporary staging of the data.
- Cloud Storage: Load your data directly into your preferred cloud storage location.
Click either the Databricks or Cloud Storage tab on this page for documentation applicable to that destination type.Select a Databricks Unity Catalog. The special value [Environment Default] uses the catalog defined in the environment. Selecting a catalog will determine which databases are available in the next parameter. The Databricks schema. The special value [Environment Default] uses the schema defined in the environment. Read Create and manage schemas to learn more. The name of the table to be created.
- Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
- Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
- Truncate and Insert: If the specified table name already exists, all rows within the table will be removed and new rows will be inserted per the next run of this pipeline.
- Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
- Yes: Staged files will be destroyed after data is loaded. This is the default setting.
- No: Staged files are retained in the staging area after data is loaded.
Choose a data staging platform using the drop-down menu.
- Amazon S3: Stage your data on an AWS S3 bucket.
- Azure Storage: Stage your data in an Azure Blob Storage container.
Click one of the tabs below for documentation applicable to that staging platform. Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.
- Append Files in Folder: Appends files to storage folder. This is the default setting.
- Overwrite Files in Folder: Overwrite existing files with matching structure.
See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:| Configuration | Description |
|---|
| Append files in folder with defined folder path and file prefix. | Files will be stored under the structure uniqueID/timestamp-partX where X is the part number, starting from 1. |
| Append files in folder without defined folder path and file prefix. | Files will be stored under the structure folder/prefix-timestamp-partX where X is the part number, starting from 1. |
| Overwrite files in folder with defined folder path and file prefix. | Files will be stored under the structure folder/prefix-partX where X is the part number, starting from 1. All files with matching structures will be overwritten. |
| Overwrite files in folder without defined folder path and file prefix. | Validation will fail. Folder path and file prefix must be supplied for this load strategy. |
The folder path of the written files.
A string of characters to include at the beginning of the written files. Often used for organizing database objects.
A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.Click the tab that corresponds to your chosen cloud storage service.
Amazon S3
Azure Storage
Google Cloud Storage
Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview. Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.
- Redshift: Load your data into Amazon Redshift. You’ll need to set a cloud storage location for temporary staging of the data.
- Cloud Storage: Load your data directly into your preferred cloud storage location.
Click either the Amazon Redshift or Cloud Storage tab on this page for documentation applicable to that destination type. Amazon Redshift
Cloud Storage
The Amazon Redshift schema. The special value [Environment Default] uses the schema defined in the environment. Read Schemas to learn more.For more information on using multiple schemas, read Schemas. The name of the table to be created.
- Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
- Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
- Truncate and Insert: If the specified table name already exists, all rows within the table will be removed and new rows will be inserted per the next run of this pipeline.
- Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
- Yes: Staged files will be destroyed after data is loaded. This is the default setting.
- No: Staged files are retained in the staging area after data is loaded.
- Append Files in Folder: Appends files to storage folder. This is the default setting.
- Overwrite Files in Folder: Overwrite existing files with matching structure.
See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:| Configuration | Description |
|---|
| Append files in folder with defined folder path and file prefix. | Files will be stored under the structure uniqueID/timestamp-partX where X is the part number, starting from 1. |
| Append files in folder without defined folder path and file prefix. | Files will be stored under the structure folder/prefix-timestamp-partX where X is the part number, starting from 1. |
| Overwrite files in folder with defined folder path and file prefix. | Files will be stored under the structure folder/prefix-partX where X is the part number, starting from 1. All files with matching structures will be overwritten. |
| Overwrite files in folder without defined folder path and file prefix. | Validation will fail. Folder path and file prefix must be supplied for this load strategy. |
The folder path of the written files.
A string of characters to include at the beginning of the written files. Often used for organizing database objects.
A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.Click the tab that corresponds to your chosen cloud storage service.
Amazon S3
Azure Storage
Google Cloud Storage
Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview. Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.
Advanced Settings
Set the severity level of logging. Choose from Error, Warn, Info, Debug, or Trace. Logs can be found in the Message field of the task details after the pipeline has been run. Choose whether to return the entire payload or only selected data objects. Read Structure to learn how to select which data objects to include in your API response.
- No: Will return the entire payload. This is the default setting.
- Yes: Will return only the objects in Custom Connector that are marked as Selected Data in the Structure setting.
Deactivate soft delete for Azure blobs (Databricks)
If you intend to set your destination as Databricks and your stage platform as Azure Storage, you must turn off the “Enable soft delete for blobs” setting in your Azure account for your pipeline to run successfully. To do this:
- In the Azure portal, navigate to your storage account.
- In the menu, under Data management, click Data protection.
- Clear the Enable soft delete for blobs checkbox. For more information, read Soft delete for blobs.
