Custom connectors are connectors that you create to fetch data from external data sources that are not available as pre-built connectors in Maia.Custom connectors can only fetch data from REST APIs that return data in JSON format. They cannot fetch data in XML format, and cannot connect to self-signed HTTPS API endpoints.This page describes how to configure a custom connector component as part of an orchestration pipeline in Designer. Custom connectors use the Connect and Configure parameters to create a table of data from the service your custom connector connects to, which is then stored in your chosen data warehouse or cloud storage location. You do not need to use the Create Table component when using a custom connector, because the custom connector component will create a new table or replace an existing table for you using the Destination parameters you define.
Maia provides a number of pre-built Flex connectors, each with a curated set of common endpoints for those services. If the service you want to connect to is available as a Flex connector, we recommend using the Flex connector as a starting point for configuring your custom connector. Editing Flex connectors is the same as creating a custom connector.
Prerequisites
Before configuring these properties for your custom connector on the Designer canvas, create your custom connector and set up the endpoints you want to connect to. For more information, read Creating custom connectors.For detailed information about authentication, specific endpoint parameters, pagination, and more, consult the API documentation for the service you’re connecting to.
Properties
Reference material is provided below for the Connect, Configure, Destination, and Advanced Settings properties.
string
required
A human-readable name for the component.
Connect
drop-down
required
Select the API endpoint to retrieve data from.The drop-down lists all the API endpoints that you added and configured when creating the custom connector.
drop-down
required
Select how to authorize access to your data.The drop-down lists the authentication method that you selected when configuring the endpoint you selected. The authentication properties you need to configure depend on the authentication type, as shown in the tabs below.
Basic authentication
Bearer token
API key
OAuth
string
required
Enter your username for the service you’re connecting to.
drop-down
required
Use the drop-down menu to select the secret definition that represents your password.Read Secrets and secret definitions to learn how to create a new secret definition.
drop-down
required
Use the drop-down menu to select the secret definition that represents your bearer token.Read Secrets and secret definitions to learn how to create a new secret definition.
string
required
Enter the key of a working API key-value pair.
drop-down
required
Select the secret definition that denotes the value of the key you selected.Read Secrets and secret definitions to learn how to create a new secret definition.
drop-down
required
Select your authentication profile.To create a new custom connector OAuth connection, read the OAuth section of the custom connector authentication guide.
Configure
column editor
required
Select and set values for the URI parameters to send in the request. You must set a value for all the URI parameters that you added when creating the custom connector.
Parameter name: Select a URI parameter. The drop-down lists all URI parameters that you added when creating the custom connector.
Parameter value: Enter a value for the parameter.
column editor
required
Select and set values for the query parameters to send in the request. You must set a value for all the configurable query parameters that you added when creating the custom connector.
Parameter name: Select a query parameter.
Parameter value: Enter a value for the parameter.
The drop-down lists all configurable query parameters that you added when creating the custom connector. Constant query parameters aren’t listed in the drop-down because you set a value for these parameters when creating the custom connector.
column editor
required
Select and set values for the header parameters to send in the request. You must set a value for all the configurable header parameters that you added when creating the custom connector.
Parameter name: Select a header parameter.
Parameter value: Enter a value for the parameter.
The drop-down lists all configurable header parameters that you added when creating the custom connector. Constant header parameters aren’t listed in the drop-down because you set a value for these parameters when creating the custom connector.
JSON
If you’re sending a POST request, enter a JSON body to include in the request.We recommend using the custom connector editor to send a test request to make sure your JSON body works as expected. For additional information about endpoints and requests, consult the documentation for the API you’re connecting to.
integer
Enter a numeric value to use as the maximum number of records per page.
Destination
Select your cloud data warehouse.
Snowflake
Databricks
Amazon Redshift
Google BigQuery
drop-down
required
Snowflake: Load your data into Snowflake. You’ll need to set a cloud storage location for temporary staging of the data.
Cloud Storage: Load your data directly into your preferred cloud storage location.
Click either the Snowflake or Cloud Storage tab on this page for documentation applicable to that destination type.
Snowflake
Cloud Storage
drop-down
required
The Snowflake warehouse used to run the queries. The special value [Environment Default] uses the warehouse defined in the environment. Read Overview of Warehouses to learn more.
drop-down
required
The Snowflake database. The special value [Environment Default] uses the database defined in the environment. Read Databases, Tables and Views - Overview to learn more.
drop-down
required
The Snowflake schema. The special value [Environment Default] uses the schema defined in the environment. Read Database, Schema, and Share DDL to learn more.
string
required
The name of the table to be created.
drop-down
required
Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
boolean
required
Yes: Staged files will be destroyed after data is loaded. This is the default setting.
No: Staged files are retained in the staging area after data is loaded.
drop-down
Select the stage access strategy. The strategies available depend on the cloud platform you select in Stage Platform.
Credentials: Connects to the external stage (AWS, Azure) using your configured cloud provider credentials. Not available for Google Cloud Storage.
Storage Integration: Use a Snowflake storage integration to grant access to Snowflake to read data from and write to a cloud storage location. This will reveal the Storage Integration property, through which you can select any of your existing Snowflake storage integrations.
drop-down
required
Choose a data staging platform using the drop-down menu.
Amazon S3: Stage your data on an AWS S3 bucket.
Snowflake: Stage your data on a Snowflake internal stage.
Azure Storage: Stage your data in an Azure Blob Storage container.
Google Cloud Storage: Stage your data in a Google Cloud Storage bucket.
Click one of the tabs below for documentation applicable to that staging platform.
Amazon S3
Snowflake
Azure Storage
Google Cloud Storage
drop-down
required
Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.
drop-down
required
An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.
drop-down
required
Select the Snowflake internal stage type. Use the Snowflake links provided to learn more about each type of stage.
User: Each Snowflake user has a user stage allocated to them by default for file storage. You may find the user stage convenient if your files will only be accessed by a single user, but need to be copied into multiple tables.
Named: A named stage provides high flexibility for data loading. Users with the appropriate privileges on the stage can load data into any table. Furthermore, because the stage is a database object, any security or access rules that apply to all objects will apply to the named stage.
Named stages can be altered and dropped. User stages cannot.
drop-down
required
Select your named stage. Read Creating a named stage to learn how to create a new named stage.
You can’t select a named stage if its name includes special characters or spaces.
drop-down
required
Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.
drop-down
required
Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.
Select the stage access strategy. The strategies available depend on the cloud platform you select in Stage Platform.
Storage Integration: Use a Snowflake storage integration to grant access to Snowflake to read data from and write to a cloud storage location. This will reveal the Storage Integration property, through which you can select any of your existing Snowflake storage integrations.
drop-down
required
Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.
drop-down
required
The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.
boolean
required
Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.
drop-down
Append Files in Folder: Appends files to storage folder. This is the default setting.
Overwrite Files in Folder: Overwrite existing files with matching structure.
See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:
Configuration
Description
Append files in folder with defined folder path and file prefix.
Files will be stored under the structure uniqueID/timestamp-partX where X is the part number, starting from 1.
Append files in folder without defined folder path and file prefix.
Files will be stored under the structure folder/prefix-timestamp-partX where X is the part number, starting from 1.
Overwrite files in folder with defined folder path and file prefix.
Files will be stored under the structure folder/prefix-partX where X is the part number, starting from 1. All files with matching structures will be overwritten.
Overwrite files in folder without defined folder path and file prefix.
Validation will fail. Folder path and file prefix must be supplied for this load strategy.
string
The folder path of the written files.
string
A string of characters to include at the beginning of the written files. Often used for organizing database objects.
drop-down
required
A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.Click the tab that corresponds to your chosen cloud storage service.
Amazon S3
Azure Storage
Google Cloud Storage
drop-down
required
An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.
drop-down
required
Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.
The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.
boolean
required
Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.
drop-down
required
Databricks: Load your data into Databricks. You’ll need to set a cloud storage location for temporary staging of the data.
Cloud Storage: Load your data directly into your preferred cloud storage location.
Click either the Databricks or Cloud Storage tab on this page for documentation applicable to that destination type.
Databricks
Cloud Storage
drop-down
required
Select a Databricks Unity Catalog. The special value [Environment Default] uses the catalog defined in the environment. Selecting a catalog will determine which databases are available in the next parameter.
drop-down
required
The Databricks schema. The special value [Environment Default] uses the schema defined in the environment. Read Create and manage schemas to learn more.
string
required
The name of the table to be created.
drop-down
required
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
column editor
In the tree structure representing the source data, select the checkboxes next to the fields that you want to output from this connector, then click Save.Unselected fields will not be included in the output from this connector.For each field you select, you can click the Edit Alias icon to change the name used for this field in all downstream components in this pipeline.
boolean
Determines what happens to a row when a mapped column comes from a field nested inside an array that’s empty or missing for that row. This property only appears once your Column Mapping includes a field nested inside an array.
Yes: Keep the row, and populate the mapped column with NULL. This is the default setting.
No: Omit the row from the output entirely.
object selector
Select one or more columns to use as the table’s primary key. If you select multiple columns, the component creates a composite primary key using the combination of the selected columns.Leave this blank to create the table without a primary key.
boolean
required
Yes: Staged files will be destroyed after data is loaded. This is the default setting.
No: Staged files are retained in the staging area after data is loaded.
drop-down
required
Choose a data staging platform using the drop-down menu.
Amazon S3: Stage your data on an AWS S3 bucket.
Azure Storage: Stage your data in an Azure Blob Storage container.
Click one of the tabs below for documentation applicable to that staging platform.
Amazon S3
Azure Storage
drop-down
required
An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.
drop-down
required
Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.
Append Files in Folder: Appends files to storage folder. This is the default setting.
Overwrite Files in Folder: Overwrite existing files with matching structure.
See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:
Configuration
Description
Append files in folder with defined folder path and file prefix.
Files will be stored under the structure uniqueID/timestamp-partX where X is the part number, starting from 1.
Append files in folder without defined folder path and file prefix.
Files will be stored under the structure folder/prefix-timestamp-partX where X is the part number, starting from 1.
Overwrite files in folder with defined folder path and file prefix.
Files will be stored under the structure folder/prefix-partX where X is the part number, starting from 1. All files with matching structures will be overwritten.
Overwrite files in folder without defined folder path and file prefix.
Validation will fail. Folder path and file prefix must be supplied for this load strategy.
string
The folder path of the written files.
string
A string of characters to include at the beginning of the written files. Often used for organizing database objects.
drop-down
required
A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.Click the tab that corresponds to your chosen cloud storage service.
Amazon S3
Azure Storage
Google Cloud Storage
drop-down
required
An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.
drop-down
required
Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.
The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.
boolean
required
Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.
drop-down
required
Redshift: Load your data into Amazon Redshift. You’ll need to set a cloud storage location for temporary staging of the data.
Cloud Storage: Load your data directly into your preferred cloud storage location.
Click either the Amazon Redshift or Cloud Storage tab on this page for documentation applicable to that destination type.
Amazon Redshift
Cloud Storage
drop-down
required
The Amazon Redshift schema. The special value [Environment Default] uses the schema defined in the environment. Read Schemas to learn more.For more information on using multiple schemas, read Schemas.
string
required
The name of the table to be created.
drop-down
required
Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
boolean
required
Yes: Staged files will be destroyed after data is loaded. This is the default setting.
No: Staged files are retained in the staging area after data is loaded.
drop-down
required
An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.
drop-down
Append Files in Folder: Appends files to storage folder. This is the default setting.
Overwrite Files in Folder: Overwrite existing files with matching structure.
See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:
Configuration
Description
Append files in folder with defined folder path and file prefix.
Files will be stored under the structure uniqueID/timestamp-partX where X is the part number, starting from 1.
Append files in folder without defined folder path and file prefix.
Files will be stored under the structure folder/prefix-timestamp-partX where X is the part number, starting from 1.
Overwrite files in folder with defined folder path and file prefix.
Files will be stored under the structure folder/prefix-partX where X is the part number, starting from 1. All files with matching structures will be overwritten.
Overwrite files in folder without defined folder path and file prefix.
Validation will fail. Folder path and file prefix must be supplied for this load strategy.
string
The folder path of the written files.
string
A string of characters to include at the beginning of the written files. Often used for organizing database objects.
drop-down
required
A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.Click the tab that corresponds to your chosen cloud storage service.
Amazon S3
Azure Storage
Google Cloud Storage
drop-down
required
An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the cloud provider credentials that you have associated with your environment.
drop-down
required
Select a storage account linked to your desired blob container to be used for staging the data. For more information, read Storage account overview.
The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the cloud provider credentials that you have associated with your environment.
boolean
required
Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.
drop-down
required
Select the destination for your data:
Google BigQuery: Load your data into a table in Google BigQuery.
Cloud Storage: Load your data directly into files in your preferred cloud storage location. The format of these files can differ between source systems and will not have a file extension. Check the output to determine the format of the data.
Google BigQuery
Cloud Storage
drop-down
required
Select the Google BigQuery project to load data into. The special value [Environment Default] uses the project defined in the environment.
drop-down
required
Select the Google BigQuery dataset to load data into. The special value [Environment Default] uses the dataset defined in the environment.
string
required
The name of the table to be created in your Google BigQuery project. You can use a Table Input component in a transformation pipeline to access and transform this data after it has been loaded.
drop-down
required
Replace: If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.
Truncate and Insert: Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.
Fail if Exists: If the specified table name already exists, this pipeline will fail to run.
Append: If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.
column editor
In the tree structure representing the source data, select the checkboxes next to the fields that you want to output from this connector, then click Save.Unselected fields will not be included in the output from this connector.For each field you select, you can click the Edit Alias icon to change the name used for this field in all downstream components in this pipeline.
object selector
Select one or more columns to use as the table’s primary key. If you select multiple columns, the component creates a composite primary key using the combination of the selected columns.Leave this blank to create the table without a primary key.
boolean
required
Yes: Staged files will be destroyed after data is loaded. This is the default setting.
No: Staged files are retained in the staging area after data is loaded.
drop-down
Select the time period to use to partition the data loaded into your table, for example Day or Month.
drop-down
required
Select the Google Cloud Storage (GCS) bucket linked to your Google Cloud Platform account.
boolean
required
Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.
drop-down
Append Files in Folder: Appends files to storage folder. This is the default setting.
Overwrite Files in Folder: Overwrite existing files with matching structure.
string
The folder path for the files to be written to. Note that this path follows, but does not include, the bucket or container name.
string
A string of characters that precedes the name of the written files. This can be useful for organizing database objects.
drop-down
required
A cloud storage location to load your data into files for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.
Advanced Settings
drop-down
Select the severity level of logging. Choose from Error, Warn, Info, Debug, or Trace. The default setting is Error.Logs can be found in the Logging tab in when viewing details about a task after the pipeline has been run.
boolean
Choose whether to return the entire payload or only selected data objects. Read Response to learn how to select the data objects to include in your API response.
No: (Default) Return the entire payload.
Yes: Only return the objects that have been Set as selected data when configuring the response structure in the custom connector editor.
Deactivate soft delete for Azure blobs (Databricks)
If you intend to set your destination as Databricks and your stage platform as Azure Storage, you must turn off the “Enable soft delete for blobs” setting in your Azure account for your pipeline to run successfully. To do this:
In the Azure portal, navigate to your storage account.
In the menu, under Data management, click Data protection.
Clear the Enable soft delete for blobs checkbox. For more information, read Soft delete for blobs.
Was this page helpful?
{"docsConfig":{"theme":"aspen","$schema":"https://mintlify.com/docs.json","name":"Maia Documentation","colors":{"primary":"#001f1d","light":"#20514e","dark":"#001f1d"},"logo":{"light":"https://mintcdn.com/matillion/GyuNBHAx-GXnVlKX/images/maia-by-matillion-light.svg?fit=max\u0026auto=format\u0026n=GyuNBHAx-GXnVlKX\u0026q=85\u0026s=2c48b854f858c5a835450b3f34360db7","dark":"https://mintcdn.com/matillion/GyuNBHAx-GXnVlKX/images/maia-by-matillion-dark.svg?fit=max\u0026auto=format\u0026n=GyuNBHAx-GXnVlKX\u0026q=85\u0026s=88f8d1e1b1931cefd35c233a56cafe0a","href":"https://docs.maia.ai/"},"favicon":{"light":"https://mintcdn.com/matillion/0xM0A92Owv1nZXX1/images/maia-favicon-dark-green.png?fit=max\u0026auto=format\u0026n=0xM0A92Owv1nZXX1\u0026q=85\u0026s=356295fc83c5c6d1ed4aac2573ae45b1","dark":"https://mintcdn.com/matillion/0xM0A92Owv1nZXX1/images/maia-favicon-white.png?fit=max\u0026auto=format\u0026n=0xM0A92Owv1nZXX1\u0026q=85\u0026s=8709fc6d370155352cb6a4b5bbdd59d6"},"appearance":{"default":"light","strict":true},"background":{"decoration":"gradient"},"navbar":{"links":[{"href":"https://support.matillion.com/s/","label":"Support"}],"primary":{"type":"button","label":"Book your Maia demo","href":"https://www.maia.ai/book-a-demo"}},"footer":{"socials":{"x":"https://x.com/matillion","facebook":"https://www.facebook.com/matillion","youtube":"https://www.youtube.com/user/MatillionVideos","linkedin":"https://www.linkedin.com/company/matillion-limited","instagram":"https://www.instagram.com/matillion_"},"links":[{"header":"Maia","items":[{"label":"Maia.ai","href":"https://maia.ai"},{"label":"Academy","href":"https://academy.matillion.com"},{"label":"Exchange","href":"https://exchange.matillion.com/"},{"label":"Roadmap","href":"https://roadmap.matillion.com/"},{"label":"Support","href":"https://support.matillion.com/s"},{"label":"YouTube","href":"https://www.youtube.com/@Matillion"}]},{"header":"Matillion","items":[{"label":"Matillion.com","href":"https://matillion.com"},{"label":"Status","href":"https://status.matillion.com/"},{"label":"News","href":"https://www.matillion.com/news"},{"label":"Legal","href":"https://www.matillion.com/legal"},{"label":"Trust Center","href":"https://www.matillion.com/trust-center"}]}]},"seo":{"metatags":{"google-site-verification":"0VLXGRESICGHnU66TdhEgUMBRRubPnW8JTkzOpiiNn0"}},"fonts":{"heading":{"family":"Inter, system-ui"},"body":{"family":"Inter, system-ui"}},"integrations":{"ga4":{"measurementId":"G-6T2W93NTXW"},"posthog":{"apiKey":"phc_ky6YtuQApuXUkdApnVTY5PrjvnHx7EQGtYyRVSd9G535","apiHost":"https://us.i.posthog.com"}},"contextual":{"options":["claude","chatgpt","perplexity","copy","view"],"display":"header"},"styling":{"latex":false},"navigation":{"pages":[]}},"deployment":{"deploymentMetadataValue":{"subdomain":"matillion","hasAudioNarration":false,"actualSubdomain":"matillion","customDomains":["docs.maia.ai"],"gitSource":{"type":"github","owner":"matillion","repo":"maia-docs","deployBranch":"main","contentDirectory":"","isPrivate":true,"mountPath":"","label":""},"feedback":{"thumbs":true},"entitlements":{"AI_CHAT":{"status":"ENABLED"},"REMOVE_BRANDING":{"status":"DISABLED"},"CONTEXTUAL_FEEDBACK":{"status":"DISABLED"},"CODE_SNIPPET_FEEDBACK":{"status":"DISABLED"},"WEB_MCP":{"status":"ENABLED"},"NARRATION":{"status":"ENABLED"}},"buildId":"6af73146bd8b0362cded8904","clientVersion":"0.0.4143","searchFilterCounts":[{"language":"default","tag":"Components","count":356},{"language":"default","tag":"Guides","count":210},{"language":"default","tag":"Connectors","count":188},{"language":"default","tag":"API Reference","count":128},{"language":"default","tag":"Endpoints","count":113},{"language":"default","tag":"Orchestration","count":94},{"language":"default","tag":"Admin","count":70},{"language":"default","tag":"Transformation","count":69},{"language":"default","tag":"Runners","count":51},{"language":"default","tag":"Streaming","count":34}],"searchSettings":{"showContentSnippets":false}},"assistantConfig":{},"basePath":"","hasChatPermissions":false,"hasWebMcpPermissions":true,"toggles":[],"googleFontsInlined":false},"customCss":[{"filePath":"styles.css","content":"#navbar .bg-background-light,#navbar .bg-background-dark{background-color:#001f1d!important}#navbar a,#navbar button,#navbar span,#navbar p{color:white!important}#navbar svg{color:white!important;stroke:white!important}#navbar #topbar-cta-button a,#navbar #topbar-cta-button a span,#navbar #topbar-cta-button svg{color:#001f1d!important}#navbar #topbar-cta-button a{font-weight:600!important}:root:not(.dark) #search-bar-entry,:root:not(.dark) #nav-anchor,:root:not(.dark) #navbar .navbar-link,:root:not(.dark) .nav-anchor{background-color:#0a3d39!important;border-radius:9999px!important}#topbar-cta-button a span.bg-primary-dark{background:#ffc700!important;border-radius:9999px!important}#navbar svg.bg-gray-500{background-color:white!important}#navbar button.lg\\:hidden div.font-semibold{color:white!important}#navbar button.lg\\:hidden div.text-gray-500 svg{color:white!important;stroke:white!important}#navbar img.nav-logo{content:url('/images/maia-by-matillion-light.svg');height:50px!important;width:auto!important}#footer img.nav-logo{content:url('/images/maia-by-matillion-dark.svg');height:80px!important;width:auto!important}#pagination{display:none}#navigation-items>div{padding-bottom:4rem!important}#navbar a[data-active=\"true\"] div.h-px{height:4px!important;background-color:#ffc700!important;border-radius:9999px!important}[id^=\"headlessui-dialog-panel\"] .navbar-link a{color:white!important}[id^=\"headlessui-dialog-panel\"] nav{padding-top:2.5rem}.accordion{width:100%!important;overflow-x:auto!important}.accordion>div{padding-left:1.375rem!important;padding-right:0!important}.accordion>div[class*=\"overflow\"],.accordion>div>div[class*=\"overflow\"]{overflow:visible!important}@media (min-width: 1920px){body:has(#navbar.is-not-custom) #body-content{max-width:2000px!important}body:has(#navbar.is-not-custom) #content-container{max-width:1650px!important}body:has(#navbar.is-not-custom) #navbar .max-w-8xl{max-width:2000px!important}}body:has(#navbar.is-not-custom) #content{zoom:var(--maia-content-zoom, 100%)}body:has(#navbar.is-not-custom) #content-area>#content+div:has(>p+a){zoom:var(--maia-content-zoom, 100%)}#content-area>#content+div:has(>p+a)>p{font-size:1.5rem;line-height:2rem;font-weight:600;letter-spacing:normal;color:#101828;margin-bottom:0.5rem}.dark #content-area>#content+div:has(>p+a)>p{color:inherit}#content-area>#content+div:has(>p+a)>a{font-size:1rem;line-height:1.75rem}"},{"filePath":"new-features-blog/new-features-blog.css","content":".fos-background-container{position:fixed;top:0;left:0;height:100vh;width:100vw;background:#fff3d1;z-index:-1;background-size:cover}.fos-post-page-container{background-color:#fff;border-radius:20px;padding:20px!important}.fos-roadmap-page-container{background-color:#fff;border-radius:20px;width:95%}@media (min-width: 1024px){.fos-roadmap-page-container{width:70%;margin:0 auto}}.fos-changelog-page-container{background-color:#fff;border-radius:20px}.banner-title,.fos-header-banner-title{color:#00524E;font-size:4rem;font-weight:700;font-family:'Inter', sans-serif}.banner-subtitle,.fos-header-banner-subtitle{color:#001F1D;font-size:1.1487em;margin-top:0.5em;margin-bottom:2em}.fos-header-boards-trigger,.fos-header-roadmap-trigger,.fos-header-changelog-trigger{display:none}.fos-user-side-header{color:white;font-family:'Inter', sans-serif}.fos-header-container{background-color:#001f1d;box-shadow:0px 1px 13px #bfbfbf}.fos-header-boards-dropdown-content span{color:white}.fos-header-links-container{justify-content:center;gap:1em}.fos-header-link{color:white;font-size:14px!important;font-weight:bold;padding:10px 0}.fos-header-logo-container{display:flex}.fos-header-logo-image{width:100px}.fos-header-boards-link{color:white;font-size:16px;padding:10px 0;font-weight:bold}.fos-header-link-icon{color:white}"}],"page":{"pagePath":"docs/components/custom-connector","slug":"/docs/components/custom-connector","pageMetadata":{"title":"Custom connectors in Designer","description":null,"href":"/docs/components/custom-connector"},"pageType":"default","mdx":{"mdxSource":{"compiledSource":"\"use strict\";const{Fragment:_Fragment,jsx:_jsx,jsxs:_jsxs}=arguments[0];const{useMDXComponents:_provideComponents}=arguments[0];const designer_0=undefined;const designer=`Designer`;const maia=`Maia`;const ComponentMetadata=({warehouses,unsupportedWarehouses=[],componentType,connectionInputs,connectionOutputs,driverVersion,driverVersionUrl})=>{const _components={a:`a`,..._provideComponents()};const allWarehouses=[...warehouses.map(w=>({name:w,supported:true})),...unsupportedWarehouses.map(w=>({name:w,supported:false}))];return _jsx(`div`,{style:{background:`var(--colors-background-light, #f9fafb)`,border:`1px solid var(--colors-border-default, #e5e7eb)`,borderRadius:`12px`,padding:`20px 28px`,marginBottom:`28px`,boxShadow:`0 1px 4px rgba(0,0,0,0.10)`},children:_jsx(`table`,{style:{width:`100%`,borderCollapse:`collapse`},children:_jsxs(`tbody`,{children:[_jsxs(`tr`,{children:[_jsx(`td`,{style:{fontWeight:`600`,paddingRight:`32px`,paddingBottom:`14px`,whiteSpace:`nowrap`,verticalAlign:`middle`,width:`180px`},children:`Project Availability`}),_jsx(`td`,{style:{paddingBottom:`14px`,verticalAlign:`middle`},children:_jsx(`div`,{style:{display:`flex`,flexWrap:`wrap`,gap:`8px`},children:allWarehouses.map((w,i)=>_jsxs(`span`,{style:{background:w.supported?`#dcfce7`:`#fee2e2`,color:w.supported?`#15803d`:`#b91c1c`,border:`1px solid ${w.supported?`#bbf7d0`:`#fca5a5`}`,borderRadius:`9999px`,padding:`3px 12px`,fontSize:`0.85rem`,fontWeight:`500`,whiteSpace:`nowrap`},children:[w.name,` `,w.supported?`✅`:`❌`]},i))})})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{fontWeight:`600`,paddingRight:`32px`,paddingBottom:`14px`,whiteSpace:`nowrap`,verticalAlign:`middle`},children:`Component Type`}),_jsx(`td`,{style:{paddingBottom:`14px`,verticalAlign:`middle`},children:componentType})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{fontWeight:`600`,paddingRight:`32px`,paddingBottom:`14px`,whiteSpace:`nowrap`,verticalAlign:`middle`},children:`Connection Inputs`}),_jsx(`td`,{style:{paddingBottom:`14px`,verticalAlign:`middle`},children:connectionInputs})]}),_jsxs(`tr`,{children:[_jsx(`td`,{style:{fontWeight:`600`,paddingRight:`32px`,paddingBottom:driverVersion?`14px`:`0`,whiteSpace:`nowrap`,verticalAlign:`middle`},children:`Connection Outputs`}),_jsx(`td`,{style:{paddingBottom:driverVersion?`14px`:`0`,verticalAlign:`middle`},children:connectionOutputs})]}),driverVersion\u0026\u0026_jsxs(`tr`,{children:[_jsx(`td`,{style:{fontWeight:`600`,paddingRight:`32px`,whiteSpace:`nowrap`,verticalAlign:`middle`},children:`Driver Version`}),_jsx(`td`,{style:{verticalAlign:`middle`},children:driverVersionUrl?_jsx(_components.a,{href:driverVersionUrl,target:`_blank`,rel:`noopener noreferrer`,children:driverVersion}):driverVersion})]})]})})})};function _createMdxContent(props){const _components={a:`a`,code:`code`,em:`em`,hr:`hr`,li:`li`,ol:`ol`,p:`p`,strong:`strong`,tbody:`tbody`,td:`td`,th:`th`,thead:`thead`,tr:`tr`,ul:`ul`,..._provideComponents(),...props.components},{Heading,Note,ResponseField,Tab,Table,Tabs,_MdxComponentBoundary}=_components;if(!Heading)_missingMdxReference(`Heading`,true);if(!Note)_missingMdxReference(`Note`,true);if(!ResponseField)_missingMdxReference(`ResponseField`,true);if(!Tab)_missingMdxReference(`Tab`,true);if(!Table)_missingMdxReference(`Table`,true);if(!Tabs)_missingMdxReference(`Tabs`,true);if(!_MdxComponentBoundary)_missingMdxReference(`_MdxComponentBoundary`,true);return _jsxs(_Fragment,{children:[_jsx(_MdxComponentBoundary,{name:`ComponentMetadata`,children:_jsx(ComponentMetadata,{warehouses:[`Snowflake`,`Databricks`,`Amazon Redshift`,`Google BigQuery`],unsupportedWarehouses:[],componentType:`Connector, Orchestration`,connectionInputs:`One`,connectionOutputs:`Unlimited`})}),`\n`,_jsx(_components.p,{children:`Custom connectors are connectors that you create to fetch data from external data sources that are not available as pre-built connectors in Maia.`}),`\n`,_jsx(_components.p,{children:`Custom connectors can only fetch data from REST APIs that return data in JSON format. They cannot fetch data in XML format, and cannot connect to self-signed HTTPS API endpoints.`}),`\n`,_jsxs(_components.p,{children:[`This page describes how to configure a custom connector component as part of an orchestration pipeline in `,designer,`. Custom connectors use the Connect and Configure parameters to create a table of data from the service your custom connector connects to, which is then stored in your chosen data warehouse or cloud storage location. You `,_jsx(_components.em,{children:`do not`}),` need to use the Create Table component when using a custom connector, because the custom connector component will create a new table or replace an existing table for you using the Destination parameters you define.`]}),`\n`,_jsx(Note,{children:_jsxs(_components.p,{children:[maia,` provides a number of pre-built Flex connectors, each with a curated set of common endpoints for those services. If the service you want to connect to is available as a Flex connector, we recommend using the Flex connector as a starting point for configuring your custom connector. Editing Flex connectors is the same as `,_jsx(_components.a,{href:`/docs/guides/custom-connector-setup`,children:`creating a custom connector`}),`.`]})}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`prerequisites`,children:`Prerequisites`}),`\n`,_jsxs(_components.p,{children:[`Before configuring these properties for your custom connector on the `,designer,` canvas, create your custom connector and set up the endpoints you want to connect to. For more information, read `,_jsx(_components.a,{href:`/docs/guides/custom-connector-setup`,children:`Creating custom connectors`}),`.`]}),`\n`,_jsx(_components.p,{children:`For detailed information about authentication, specific endpoint parameters, pagination, and more, consult the API documentation for the service you’re connecting to.`}),`\n`,_jsx(_components.hr,{}),`\n`,_jsx(Heading,{level:`2`,id:`properties`,children:`Properties`}),`\n`,_jsx(_components.p,{children:`Reference material is provided below for the Connect, Configure, Destination, and Advanced Settings properties.`}),`\n`,_jsx(ResponseField,{name:`Name`,type:`string`,required:true,id:`param-name`,children:_jsx(_components.p,{children:`A human-readable name for the component.`})}),`\n`,_jsx(Heading,{level:`3`,id:`connect`,children:`Connect`}),`\n`,_jsxs(ResponseField,{name:`Data Source`,type:`drop-down`,required:true,id:`param-data-source`,children:[_jsx(_components.p,{children:`Select the API endpoint to retrieve data from.`}),_jsx(_components.p,{children:`The drop-down lists all the API endpoints that you added and configured when creating the custom connector.`})]}),`\n`,_jsxs(ResponseField,{name:`Authentication Type`,type:`drop-down`,required:true,id:`param-authentication-type`,children:[_jsx(_components.p,{children:`Select how to authorize access to your data.`}),_jsx(_components.p,{children:`The drop-down lists the authentication method that you selected when configuring the endpoint you selected. The authentication properties you need to configure depend on the authentication type, as shown in the tabs below.`})]}),`\n`,_jsxs(Tabs,{children:[_jsxs(Tab,{title:`Basic authentication`,id:`basic-authentication`,children:[_jsx(ResponseField,{name:`Username`,type:`string`,required:true,id:`param-username`,children:_jsx(_components.p,{children:`Enter your username for the service you’re connecting to.`})}),_jsxs(ResponseField,{name:`Password`,type:`drop-down`,required:true,id:`param-password`,children:[_jsx(_components.p,{children:`Use the drop-down menu to select the secret definition that represents your password.`}),_jsxs(_components.p,{children:[`Read `,_jsx(_components.a,{href:`/docs/guides/secrets-and-secret-definitions`,children:`Secrets and secret definitions`}),` to learn how to create a new secret definition.`]})]})]}),_jsx(Tab,{title:`Bearer token`,id:`bearer-token`,children:_jsxs(ResponseField,{name:`Token`,type:`drop-down`,required:true,id:`param-token`,children:[_jsx(_components.p,{children:`Use the drop-down menu to select the secret definition that represents your bearer token.`}),_jsxs(_components.p,{children:[`Read `,_jsx(_components.a,{href:`/docs/guides/secrets-and-secret-definitions`,children:`Secrets and secret definitions`}),` to learn how to create a new secret definition.`]})]})}),_jsxs(Tab,{title:`API key`,id:`api-key`,children:[_jsx(ResponseField,{name:`Key`,type:`string`,required:true,id:`param-key`,children:_jsx(_components.p,{children:`Enter the key of a working API key-value pair.`})}),_jsxs(ResponseField,{name:`Value`,type:`drop-down`,required:true,id:`param-value`,children:[_jsx(_components.p,{children:`Select the secret definition that denotes the value of the key you selected.`}),_jsxs(_components.p,{children:[`Read `,_jsx(_components.a,{href:`/docs/guides/secrets-and-secret-definitions`,children:`Secrets and secret definitions`}),` to learn how to create a new secret definition.`]})]})]}),_jsx(Tab,{title:`OAuth`,id:`oauth`,children:_jsxs(ResponseField,{name:`Authentication`,type:`drop-down`,required:true,id:`param-authentication`,children:[_jsx(_components.p,{children:`Select your authentication profile.`}),_jsxs(_components.p,{children:[`To create a new custom connector OAuth connection, read the `,_jsx(_components.a,{href:`/docs/guides/custom-connector-authentication#oauth`,children:`OAuth`}),` section of the custom connector authentication guide.`]})]})})]}),`\n`,_jsx(Heading,{level:`3`,id:`configure`,children:`Configure`}),`\n`,_jsxs(ResponseField,{name:`URI parameters`,type:`column editor`,required:true,id:`param-uri-parameters`,children:[_jsx(_components.p,{children:`Select and set values for the URI parameters to send in the request. You must set a value for all the URI parameters that you added when creating the custom connector.`}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Parameter name:`}),` Select a URI parameter. The drop-down lists all URI parameters that you added when creating the custom connector.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Parameter value:`}),` Enter a value for the parameter.`]}),`\n`]})]}),`\n`,_jsxs(ResponseField,{name:`Query parameters`,type:`column editor`,required:true,id:`param-query-parameters`,children:[_jsx(_components.p,{children:`Select and set values for the query parameters to send in the request. You must set a value for all the configurable query parameters that you added when creating the custom connector.`}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Parameter name:`}),` Select a query parameter.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Parameter value:`}),` Enter a value for the parameter.`]}),`\n`]}),_jsxs(_components.p,{children:[`The drop-down lists all `,_jsx(_components.a,{href:`/docs/guides/custom-connector-setup#configurability`,children:`configurable`}),` query parameters that you added when creating the custom connector. `,_jsx(_components.a,{href:`/docs/guides/custom-connector-setup#configurability`,children:`Constant`}),` query parameters aren’t listed in the drop-down because you set a value for these parameters when creating the custom connector.`]})]}),`\n`,_jsxs(ResponseField,{name:`Header parameters`,type:`column editor`,required:true,id:`param-header-parameters`,children:[_jsx(_components.p,{children:`Select and set values for the header parameters to send in the request. You must set a value for all the configurable header parameters that you added when creating the custom connector.`}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Parameter name:`}),` Select a header parameter.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Parameter value:`}),` Enter a value for the parameter.`]}),`\n`]}),_jsxs(_components.p,{children:[`The drop-down lists all `,_jsx(_components.a,{href:`/docs/guides/custom-connector-setup#configurability`,children:`configurable`}),` header parameters that you added when creating the custom connector. `,_jsx(_components.a,{href:`/docs/guides/custom-connector-setup#configurability`,children:`Constant`}),` header parameters aren’t listed in the drop-down because you set a value for these parameters when creating the custom connector.`]})]}),`\n`,_jsxs(ResponseField,{name:`Post Body`,type:`JSON`,id:`param-post-body`,children:[_jsxs(_components.p,{children:[`If you’re sending a `,_jsx(_components.code,{children:`POST`}),` request, enter a JSON body to include in the request.`]}),_jsxs(_components.p,{children:[`We recommend using the custom connector editor to `,_jsx(_components.a,{href:`/docs/guides/custom-connector-setup#send-a-test-request`,children:`send a test request`}),` to make sure your JSON body works as expected. For additional information about endpoints and requests, consult the documentation for the API you’re connecting to.`]})]}),`\n`,_jsx(ResponseField,{name:`Page Limit`,type:`integer`,id:`param-page-limit`,children:_jsx(_components.p,{children:`Enter a numeric value to use as the maximum number of records per page.`})}),`\n`,_jsx(Heading,{level:`3`,id:`destination`,children:`Destination`}),`\n`,_jsx(_components.p,{children:`Select your cloud data warehouse.`}),`\n`,_jsxs(Tabs,{children:[_jsxs(Tab,{title:`Snowflake`,id:`snowflake`,\"data-child-tab-ids\":`[\"snowflake-2\",\"cloud-storage\"]`,children:[_jsxs(ResponseField,{name:`Destination`,type:`drop-down`,required:true,id:`param-destination`,children:[_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Snowflake:`}),` Load your data into Snowflake. You’ll need to set a cloud storage location for temporary staging of the data.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Cloud Storage:`}),` Load your data directly into your preferred cloud storage location.`]}),`\n`]}),_jsxs(_components.p,{children:[`Click either the `,_jsx(_components.strong,{children:`Snowflake`}),` or `,_jsx(_components.strong,{children:`Cloud Storage`}),` tab on this page for documentation applicable to that destination type.`]})]}),_jsxs(Tabs,{children:[_jsxs(Tab,{title:`Snowflake`,id:`snowflake-2`,\"data-child-tab-ids\":`[\"amazon-s3\",\"snowflake-3\",\"azure-storage\",\"google-cloud-storage\"]`,children:[_jsx(ResponseField,{name:`Warehouse`,type:`drop-down`,required:true,id:`param-warehouse`,children:_jsxs(_components.p,{children:[`The Snowflake warehouse used to run the queries. The special value `,_jsx(_components.code,{children:`[Environment Default]`}),` uses the warehouse defined in the environment. Read `,_jsx(_components.a,{href:`https://docs.snowflake.com/en/user-guide/warehouses-overview.html`,children:`Overview of Warehouses`}),` to learn more.`]})}),_jsx(ResponseField,{name:`Database`,type:`drop-down`,required:true,id:`param-database`,children:_jsxs(_components.p,{children:[`The Snowflake database. The special value `,_jsx(_components.code,{children:`[Environment Default]`}),` uses the database defined in the environment. Read `,_jsx(_components.a,{href:`https://docs.snowflake.com/en/guides-overview-db`,children:`Databases, Tables and Views - Overview`}),` to learn more.`]})}),_jsx(ResponseField,{name:`Schema`,type:`drop-down`,required:true,id:`param-schema`,children:_jsxs(_components.p,{children:[`The Snowflake schema. The special value `,_jsx(_components.code,{children:`[Environment Default]`}),` uses the schema defined in the environment. Read `,_jsx(_components.a,{href:`https://docs.snowflake.com/en/sql-reference/ddl-database.html`,children:`Database, Schema, and Share DDL`}),` to learn more.`]})}),_jsx(ResponseField,{name:`Table Name`,type:`string`,required:true,id:`param-table-name`,children:_jsx(_components.p,{children:`The name of the table to be created.`})}),_jsx(ResponseField,{name:`Load Strategy`,type:`drop-down`,required:true,id:`param-load-strategy`,children:_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Replace:`}),` If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Truncate and Insert:`}),` Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Fail if Exists:`}),` If the specified table name already exists, this pipeline will fail to run.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Append:`}),` If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.`]}),`\n`]})}),_jsx(ResponseField,{name:`Clean Staged files`,type:`boolean`,required:true,id:`param-clean-staged-files`,children:_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Yes:`}),` Staged files will be destroyed after data is loaded. This is the default setting.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`No:`}),` Staged files are retained in the staging area after data is loaded.`]}),`\n`]})}),_jsxs(ResponseField,{name:`Stage Access Strategy`,type:`drop-down`,id:`param-stage-access-strategy`,children:[_jsxs(_components.p,{children:[`Select the stage access strategy. The strategies available depend on the cloud platform you select in `,_jsx(_components.strong,{children:`Stage Platform`}),`.`]}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Credentials:`}),` Connects to the external stage (AWS, Azure) using your configured `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),`. Not available for Google Cloud Storage.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Storage Integration:`}),` Use a Snowflake storage integration to grant access to Snowflake to read data from and write to a cloud storage location. This will reveal the `,_jsx(_components.strong,{children:`Storage Integration`}),` property, through which you can select any of your existing Snowflake storage integrations.`]}),`\n`]})]}),_jsxs(ResponseField,{name:`Stage Platform`,type:`drop-down`,required:true,id:`param-stage-platform`,children:[_jsx(_components.p,{children:`Choose a data staging platform using the drop-down menu.`}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Amazon S3:`}),` Stage your data on an AWS S3 bucket.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Snowflake:`}),` Stage your data on a Snowflake internal stage.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Azure Storage:`}),` Stage your data in an Azure Blob Storage container.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Google Cloud Storage:`}),` Stage your data in a Google Cloud Storage bucket.`]}),`\n`]}),_jsx(_components.p,{children:`Click one of the tabs below for documentation applicable to that staging platform.`})]}),_jsxs(Tabs,{children:[_jsxs(Tab,{title:`Amazon S3`,id:`amazon-s3`,children:[_jsx(ResponseField,{name:`Storage Integration`,type:`drop-down`,required:true,id:`param-storage-integration`,children:_jsx(_components.p,{children:`Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.`})}),_jsx(ResponseField,{name:`Amazon S3 Bucket`,type:`drop-down`,required:true,id:`param-amazon-s3-bucket`,children:_jsxs(_components.p,{children:[`An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),` that you have associated with your `,_jsx(_components.a,{href:`/docs/guides/environments`,children:`environment`}),`.`]})})]}),_jsxs(Tab,{title:`Snowflake`,id:`snowflake-3`,children:[_jsxs(ResponseField,{name:`Internal Stage Type`,type:`drop-down`,required:true,id:`param-internal-stage-type`,children:[_jsx(_components.p,{children:`Select the Snowflake internal stage type. Use the Snowflake links provided to learn more about each type of stage.`}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`User:`}),` Each Snowflake user has a `,_jsx(_components.a,{href:`https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage#user-stages`,children:`user stage`}),` allocated to them by default for file storage. You may find the user stage convenient if your files will only be accessed by a single user, but need to be copied into multiple tables.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Named:`}),` A `,_jsx(_components.a,{href:`https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage#named-stages`,children:`named stage`}),` provides high flexibility for data loading. Users with the appropriate privileges on the stage can load data into any table. Furthermore, because the stage is a database object, any security or access rules that apply to all objects will apply to the named stage.`]}),`\n`]}),_jsx(_components.p,{children:`Named stages can be altered and dropped. User stages cannot.`})]}),_jsxs(ResponseField,{name:`Named Stage`,type:`drop-down`,required:true,id:`param-named-stage`,children:[_jsxs(_components.p,{children:[`Select your named stage. Read `,_jsx(_components.a,{href:`https://docs.snowflake.com/en/user-guide/data-load-local-file-system-create-stage#creating-a-named-stage`,children:`Creating a named stage`}),` to learn how to create a new named stage.`]}),_jsx(Note,{children:_jsx(_components.p,{children:`You can’t select a named stage if its name includes special characters or spaces.`})})]})]}),_jsxs(Tab,{title:`Azure Storage`,id:`azure-storage`,children:[_jsx(ResponseField,{name:`Storage Integration`,type:`drop-down`,required:true,id:`param-storage-integration-1`,children:_jsx(_components.p,{children:`Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.`})}),_jsx(ResponseField,{name:`Storage Account`,type:`drop-down`,required:true,id:`param-storage-account`,children:_jsxs(_components.p,{children:[`Select a storage account linked to your desired blob container to be used for staging the data. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview`,children:`Storage account overview`}),`.`]})}),_jsx(ResponseField,{name:`Container`,type:`drop-down`,required:true,id:`param-container`,children:_jsxs(_components.p,{children:[`Select a Blob container to be used for staging the data. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction`,children:`Introduction to Azure Blob storage`}),`.`]})})]}),_jsxs(Tab,{title:`Google Cloud Storage`,id:`google-cloud-storage`,children:[_jsxs(ResponseField,{name:`Stage Access Strategy`,type:`drop-down`,id:`param-stage-access-strategy-1`,children:[_jsxs(_components.p,{children:[`Select the stage access strategy. The strategies available depend on the cloud platform you select in `,_jsx(_components.strong,{children:`Stage Platform`}),`.`]}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Storage Integration:`}),` Use a Snowflake storage integration to grant access to Snowflake to read data from and write to a cloud storage location. This will reveal the `,_jsx(_components.strong,{children:`Storage Integration`}),` property, through which you can select any of your existing Snowflake storage integrations.`]}),`\n`]})]}),_jsx(ResponseField,{name:`Storage Integration`,type:`drop-down`,required:true,id:`param-storage-integration-2`,children:_jsx(_components.p,{children:`Select the storage integration. Storage integrations are required to permit Snowflake to read data from and write to a cloud storage location. Integrations must be set up in advance of selecting them. Storage integrations can be configured to support Amazon S3, Google Cloud Storage, or Microsoft Azure Blob Storage, regardless of the cloud provider that hosts your Snowflake account.`})}),_jsx(ResponseField,{name:`GCS Bucket`,type:`drop-down`,required:true,id:`param-gcs-bucket`,children:_jsxs(_components.p,{children:[`The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),` that you have associated with your `,_jsx(_components.a,{href:`/docs/guides/environments`,children:`environment`}),`.`]})}),_jsx(ResponseField,{name:`Overwrite`,type:`boolean`,required:true,id:`param-overwrite`,children:_jsx(_components.p,{children:`Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.`})})]})]})]}),_jsxs(Tab,{title:`Cloud Storage`,id:`cloud-storage`,\"data-child-tab-ids\":`[\"amazon-s3-2\",\"azure-storage-2\",\"google-cloud-storage-2\"]`,children:[_jsxs(ResponseField,{name:`Load Strategy`,type:`drop-down`,id:`param-load-strategy-1`,children:[_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Append Files in Folder:`}),` Appends files to storage folder. This is the default setting.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Overwrite Files in Folder:`}),` Overwrite existing files with matching structure.`]}),`\n`]}),_jsx(_components.p,{children:`See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:`}),_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Configuration`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Append files in folder with defined folder path and file prefix.`}),_jsxs(_components.td,{children:[`Files will be stored under the structure `,_jsx(_components.code,{children:`uniqueID/timestamp-partX`}),` where X is the part number, starting from 1.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Append files in folder without defined folder path and file prefix.`}),_jsxs(_components.td,{children:[`Files will be stored under the structure `,_jsx(_components.code,{children:`folder/prefix-timestamp-partX`}),` where X is the part number, starting from 1.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Overwrite files in folder with defined folder path and file prefix.`}),_jsxs(_components.td,{children:[`Files will be stored under the structure `,_jsx(_components.code,{children:`folder/prefix-partX`}),` where X is the part number, starting from 1. All files with matching structures will be overwritten.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Overwrite files in folder without defined folder path and file prefix.`}),_jsx(_components.td,{children:`Validation will fail. Folder path and file prefix must be supplied for this load strategy.`})]})]})]})]}),_jsx(ResponseField,{name:`Folder Path`,type:`string`,id:`param-folder-path`,children:_jsx(_components.p,{children:`The folder path of the written files.`})}),_jsx(ResponseField,{name:`File Prefix`,type:`string`,id:`param-file-prefix`,children:_jsx(_components.p,{children:`A string of characters to include at the beginning of the written files. Often used for organizing database objects.`})}),_jsxs(ResponseField,{name:`Storage`,type:`drop-down`,required:true,id:`param-storage`,children:[_jsx(_components.p,{children:`A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.`}),_jsx(_components.p,{children:`Click the tab that corresponds to your chosen cloud storage service.`})]}),_jsxs(Tabs,{children:[_jsx(Tab,{title:`Amazon S3`,id:`amazon-s3-2`,children:_jsx(ResponseField,{name:`Amazon S3 Bucket`,type:`drop-down`,required:true,id:`param-amazon-s3-bucket-1`,children:_jsxs(_components.p,{children:[`An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),` that you have associated with your `,_jsx(_components.a,{href:`/docs/guides/environments`,children:`environment`}),`.`]})})}),_jsxs(Tab,{title:`Azure Storage`,id:`azure-storage-2`,children:[_jsx(ResponseField,{name:`Storage Account`,type:`drop-down`,required:true,id:`param-storage-account-1`,children:_jsxs(_components.p,{children:[`Select a storage account linked to your desired blob container to be used for staging the data. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview`,children:`Storage account overview`}),`.`]})}),_jsx(ResponseField,{name:`Container`,type:`drop-down`,required:true,id:`param-container-1`,children:_jsxs(_components.p,{children:[`Select a Blob container to be used for staging the data. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction`,children:`Introduction to Azure Blob storage`}),`.`]})})]}),_jsxs(Tab,{title:`Google Cloud Storage`,id:`google-cloud-storage-2`,children:[_jsx(ResponseField,{name:`GCS Bucket`,type:`drop-down`,required:true,id:`param-gcs-bucket-1`,children:_jsxs(_components.p,{children:[`The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),` that you have associated with your `,_jsx(_components.a,{href:`/docs/guides/environments`,children:`environment`}),`.`]})}),_jsx(ResponseField,{name:`Overwrite`,type:`boolean`,required:true,id:`param-overwrite-1`,children:_jsx(_components.p,{children:`Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.`})})]})]})]})]})]}),_jsxs(Tab,{title:`Databricks`,id:`databricks`,\"data-child-tab-ids\":`[\"databricks-2\",\"cloud-storage-2\"]`,children:[_jsxs(ResponseField,{name:`Destination`,type:`drop-down`,required:true,id:`param-destination-1`,children:[_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Databricks:`}),` Load your data into Databricks. You’ll need to set a cloud storage location for temporary staging of the data.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Cloud Storage:`}),` Load your data directly into your preferred cloud storage location.`]}),`\n`]}),_jsxs(_components.p,{children:[`Click either the `,_jsx(_components.strong,{children:`Databricks`}),` or `,_jsx(_components.strong,{children:`Cloud Storage`}),` tab on this page for documentation applicable to that destination type.`]})]}),_jsxs(Tabs,{children:[_jsxs(Tab,{title:`Databricks`,id:`databricks-2`,\"data-child-tab-ids\":`[\"amazon-s3-3\",\"azure-storage-3\"]`,children:[_jsx(ResponseField,{name:`Catalog`,type:`drop-down`,required:true,id:`param-catalog`,children:_jsxs(_components.p,{children:[`Select a `,_jsx(_components.a,{href:`https://docs.databricks.com/en/data-governance/unity-catalog/index.html`,children:`Databricks Unity Catalog`}),`. The special value `,_jsx(_components.code,{children:`[Environment Default]`}),` uses the catalog defined in the environment. Selecting a catalog will determine which databases are available in the next parameter.`]})}),_jsx(ResponseField,{name:`Schema`,type:`drop-down`,required:true,id:`param-schema-1`,children:_jsxs(_components.p,{children:[`The Databricks schema. The special value `,_jsx(_components.code,{children:`[Environment Default]`}),` uses the schema defined in the environment. Read `,_jsx(_components.a,{href:`https://docs.databricks.com/en/data-governance/unity-catalog/create-schemas.html`,children:`Create and manage schemas`}),` to learn more.`]})}),_jsx(ResponseField,{name:`Table Name`,type:`string`,required:true,id:`param-table-name-1`,children:_jsx(_components.p,{children:`The name of the table to be created.`})}),_jsx(ResponseField,{name:`Load Strategy`,type:`drop-down`,required:true,id:`param-load-strategy-2`,children:_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Fail if Exists:`}),` If the specified table name already exists, this pipeline will fail to run.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Replace:`}),` If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Truncate and Insert:`}),` Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Append:`}),` If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.`]}),`\n`]})}),_jsxs(ResponseField,{name:`Column Mapping`,type:`column editor`,id:`param-column-mapping`,children:[_jsxs(_components.p,{children:[`In the tree structure representing the source data, select the checkboxes next to the fields that you want to output from this connector, then click `,_jsx(_components.strong,{children:`Save`}),`.`]}),_jsx(_components.p,{children:`Unselected fields will not be included in the output from this connector.`}),_jsxs(_components.p,{children:[`For each field you select, you can click the `,_jsx(_components.strong,{children:`Edit Alias`}),` icon to change the name used for this field in all downstream components in this pipeline.`]})]}),_jsxs(ResponseField,{name:`Outer Join`,type:`boolean`,id:`param-outer-join`,children:[_jsxs(_components.p,{children:[`Determines what happens to a row when a mapped column comes from a field nested inside an array that’s empty or missing for that row. This property only appears once your `,_jsx(_components.strong,{children:`Column Mapping`}),` includes a field nested inside an array.`]}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Yes:`}),` Keep the row, and populate the mapped column with `,_jsx(_components.code,{children:`NULL`}),`. This is the default setting.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`No:`}),` Omit the row from the output entirely.`]}),`\n`]})]}),_jsxs(ResponseField,{name:`Primary Keys`,type:`object selector`,id:`param-primary-keys`,children:[_jsx(_components.p,{children:`Select one or more columns to use as the table’s primary key. If you select multiple columns, the component creates a composite primary key using the combination of the selected columns.`}),_jsx(_components.p,{children:`Leave this blank to create the table without a primary key.`})]}),_jsx(ResponseField,{name:`Clean Staged Files`,type:`boolean`,required:true,id:`param-clean-staged-files-1`,children:_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Yes:`}),` Staged files will be destroyed after data is loaded. This is the default setting.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`No:`}),` Staged files are retained in the staging area after data is loaded.`]}),`\n`]})}),_jsxs(ResponseField,{name:`Stage Platform`,type:`drop-down`,required:true,id:`param-stage-platform-1`,children:[_jsx(_components.p,{children:`Choose a data staging platform using the drop-down menu.`}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Amazon S3:`}),` Stage your data on an AWS S3 bucket.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Azure Storage:`}),` Stage your data in an Azure Blob Storage container.`]}),`\n`]}),_jsx(_components.p,{children:`Click one of the tabs below for documentation applicable to that staging platform.`})]}),_jsxs(Tabs,{children:[_jsx(Tab,{title:`Amazon S3`,id:`amazon-s3-3`,children:_jsx(ResponseField,{name:`Amazon S3 Bucket`,type:`drop-down`,required:true,id:`param-amazon-s3-bucket-2`,children:_jsxs(_components.p,{children:[`An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),` that you have associated with your `,_jsx(_components.a,{href:`/docs/guides/environments`,children:`environment`}),`.`]})})}),_jsxs(Tab,{title:`Azure Storage`,id:`azure-storage-3`,children:[_jsx(ResponseField,{name:`Storage Account`,type:`drop-down`,required:true,id:`param-storage-account-2`,children:_jsxs(_components.p,{children:[`Select a storage account linked to your desired blob container to be used for staging the data. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview`,children:`Storage account overview`}),`.`]})}),_jsx(ResponseField,{name:`Container`,type:`drop-down`,required:true,id:`param-container-2`,children:_jsxs(_components.p,{children:[`Select a Blob container to be used for staging the data. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction`,children:`Introduction to Azure Blob storage`}),`.`]})})]})]})]}),_jsxs(Tab,{title:`Cloud Storage`,id:`cloud-storage-2`,\"data-child-tab-ids\":`[\"amazon-s3-4\",\"azure-storage-4\",\"google-cloud-storage-3\"]`,children:[_jsxs(ResponseField,{name:`Load Strategy`,type:`drop-down`,id:`param-load-strategy-3`,children:[_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Append Files in Folder:`}),` Appends files to storage folder. This is the default setting.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Overwrite Files in Folder:`}),` Overwrite existing files with matching structure.`]}),`\n`]}),_jsx(_components.p,{children:`See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:`}),_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Configuration`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Append files in folder with defined folder path and file prefix.`}),_jsxs(_components.td,{children:[`Files will be stored under the structure `,_jsx(_components.code,{children:`uniqueID/timestamp-partX`}),` where X is the part number, starting from 1.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Append files in folder without defined folder path and file prefix.`}),_jsxs(_components.td,{children:[`Files will be stored under the structure `,_jsx(_components.code,{children:`folder/prefix-timestamp-partX`}),` where X is the part number, starting from 1.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Overwrite files in folder with defined folder path and file prefix.`}),_jsxs(_components.td,{children:[`Files will be stored under the structure `,_jsx(_components.code,{children:`folder/prefix-partX`}),` where X is the part number, starting from 1. All files with matching structures will be overwritten.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Overwrite files in folder without defined folder path and file prefix.`}),_jsx(_components.td,{children:`Validation will fail. Folder path and file prefix must be supplied for this load strategy.`})]})]})]})]}),_jsx(ResponseField,{name:`Folder Path`,type:`string`,id:`param-folder-path-1`,children:_jsx(_components.p,{children:`The folder path of the written files.`})}),_jsx(ResponseField,{name:`File Prefix`,type:`string`,id:`param-file-prefix-1`,children:_jsx(_components.p,{children:`A string of characters to include at the beginning of the written files. Often used for organizing database objects.`})}),_jsxs(ResponseField,{name:`Storage`,type:`drop-down`,required:true,id:`param-storage-1`,children:[_jsx(_components.p,{children:`A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.`}),_jsx(_components.p,{children:`Click the tab that corresponds to your chosen cloud storage service.`})]}),_jsxs(Tabs,{children:[_jsx(Tab,{title:`Amazon S3`,id:`amazon-s3-4`,children:_jsx(ResponseField,{name:`Amazon S3 Bucket`,type:`drop-down`,required:true,id:`param-amazon-s3-bucket-3`,children:_jsxs(_components.p,{children:[`An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),` that you have associated with your `,_jsx(_components.a,{href:`/docs/guides/environments`,children:`environment`}),`.`]})})}),_jsxs(Tab,{title:`Azure Storage`,id:`azure-storage-4`,children:[_jsx(ResponseField,{name:`Storage Account`,type:`drop-down`,required:true,id:`param-storage-account-3`,children:_jsxs(_components.p,{children:[`Select a storage account linked to your desired blob container to be used for staging the data. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview`,children:`Storage account overview`}),`.`]})}),_jsx(ResponseField,{name:`Container`,type:`drop-down`,required:true,id:`param-container-3`,children:_jsxs(_components.p,{children:[`Select a Blob container to be used for staging the data. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction`,children:`Introduction to Azure Blob storage`}),`.`]})})]}),_jsxs(Tab,{title:`Google Cloud Storage`,id:`google-cloud-storage-3`,children:[_jsx(ResponseField,{name:`GCS Bucket`,type:`drop-down`,required:true,id:`param-gcs-bucket-2`,children:_jsxs(_components.p,{children:[`The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),` that you have associated with your `,_jsx(_components.a,{href:`/docs/guides/environments`,children:`environment`}),`.`]})}),_jsx(ResponseField,{name:`Overwrite`,type:`boolean`,required:true,id:`param-overwrite-2`,children:_jsx(_components.p,{children:`Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.`})})]})]})]})]})]}),_jsxs(Tab,{title:`Amazon Redshift`,id:`amazon-redshift`,\"data-child-tab-ids\":`[\"amazon-redshift-2\",\"cloud-storage-3\"]`,children:[_jsxs(ResponseField,{name:`Destination`,type:`drop-down`,required:true,id:`param-destination-2`,children:[_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Redshift:`}),` Load your data into Amazon Redshift. You’ll need to set a cloud storage location for temporary staging of the data.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Cloud Storage:`}),` Load your data directly into your preferred cloud storage location.`]}),`\n`]}),_jsxs(_components.p,{children:[`Click either the `,_jsx(_components.strong,{children:`Amazon Redshift`}),` or `,_jsx(_components.strong,{children:`Cloud Storage`}),` tab on this page for documentation applicable to that destination type.`]})]}),_jsxs(Tabs,{children:[_jsxs(Tab,{title:`Amazon Redshift`,id:`amazon-redshift-2`,children:[_jsxs(ResponseField,{name:`Schema`,type:`drop-down`,required:true,id:`param-schema-2`,children:[_jsxs(_components.p,{children:[`The Amazon Redshift schema. The special value `,_jsx(_components.code,{children:`[Environment Default]`}),` uses the schema defined in the environment. Read `,_jsx(_components.a,{href:`https://docs.aws.amazon.com/redshift/latest/dg/r_Schemas_and_tables.html`,children:`Schemas`}),` to learn more.`]}),_jsxs(_components.p,{children:[`For more information on using multiple schemas, read `,_jsx(_components.a,{href:`https://docs.aws.amazon.com/redshift/latest/dg/r_Schemas_and_tables.html`,children:`Schemas`}),`.`]})]}),_jsx(ResponseField,{name:`Table Name`,type:`string`,required:true,id:`param-table-name-2`,children:_jsx(_components.p,{children:`The name of the table to be created.`})}),_jsx(ResponseField,{name:`Load Strategy`,type:`drop-down`,required:true,id:`param-load-strategy-4`,children:_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Replace:`}),` If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Fail if Exists:`}),` If the specified table name already exists, this pipeline will fail to run.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Truncate and Insert:`}),` Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Append:`}),` If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.`]}),`\n`]})}),_jsx(ResponseField,{name:`Clean Staged Files`,type:`boolean`,required:true,id:`param-clean-staged-files-2`,children:_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Yes:`}),` Staged files will be destroyed after data is loaded. This is the default setting.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`No:`}),` Staged files are retained in the staging area after data is loaded.`]}),`\n`]})}),_jsx(ResponseField,{name:`Amazon S3 Bucket`,type:`drop-down`,required:true,id:`param-amazon-s3-bucket-4`,children:_jsxs(_components.p,{children:[`An AWS S3 bucket to stage data into. The drop-down menu will include buckets tied to the `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),` that you have associated with your `,_jsx(_components.a,{href:`/docs/guides/environments`,children:`environment`}),`.`]})})]}),_jsxs(Tab,{title:`Cloud Storage`,id:`cloud-storage-3`,\"data-child-tab-ids\":`[\"amazon-s3-5\",\"azure-storage-5\",\"google-cloud-storage-4\"]`,children:[_jsxs(ResponseField,{name:`Load Strategy`,type:`drop-down`,id:`param-load-strategy-5`,children:[_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Append Files in Folder:`}),` Appends files to storage folder. This is the default setting.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Overwrite Files in Folder:`}),` Overwrite existing files with matching structure.`]}),`\n`]}),_jsx(_components.p,{children:`See the configuration table for how this parameter works with the Folder Path and File Prefix parameters:`}),_jsxs(Table,{children:[_jsx(_components.thead,{children:_jsxs(_components.tr,{children:[_jsx(_components.th,{children:`Configuration`}),_jsx(_components.th,{children:`Description`})]})}),_jsxs(_components.tbody,{children:[_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Append files in folder with defined folder path and file prefix.`}),_jsxs(_components.td,{children:[`Files will be stored under the structure `,_jsx(_components.code,{children:`uniqueID/timestamp-partX`}),` where X is the part number, starting from 1.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Append files in folder without defined folder path and file prefix.`}),_jsxs(_components.td,{children:[`Files will be stored under the structure `,_jsx(_components.code,{children:`folder/prefix-timestamp-partX`}),` where X is the part number, starting from 1.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Overwrite files in folder with defined folder path and file prefix.`}),_jsxs(_components.td,{children:[`Files will be stored under the structure `,_jsx(_components.code,{children:`folder/prefix-partX`}),` where X is the part number, starting from 1. All files with matching structures will be overwritten.`]})]}),_jsxs(_components.tr,{children:[_jsx(_components.td,{children:`Overwrite files in folder without defined folder path and file prefix.`}),_jsx(_components.td,{children:`Validation will fail. Folder path and file prefix must be supplied for this load strategy.`})]})]})]})]}),_jsx(ResponseField,{name:`Folder Path`,type:`string`,id:`param-folder-path-2`,children:_jsx(_components.p,{children:`The folder path of the written files.`})}),_jsx(ResponseField,{name:`File Prefix`,type:`string`,id:`param-file-prefix-2`,children:_jsx(_components.p,{children:`A string of characters to include at the beginning of the written files. Often used for organizing database objects.`})}),_jsxs(ResponseField,{name:`Storage`,type:`drop-down`,required:true,id:`param-storage-2`,children:[_jsx(_components.p,{children:`A cloud storage location to load your data into for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.`}),_jsx(_components.p,{children:`Click the tab that corresponds to your chosen cloud storage service.`})]}),_jsxs(Tabs,{children:[_jsx(Tab,{title:`Amazon S3`,id:`amazon-s3-5`,children:_jsx(ResponseField,{name:`Amazon S3 Bucket`,type:`drop-down`,required:true,id:`param-amazon-s3-bucket-5`,children:_jsxs(_components.p,{children:[`An AWS S3 bucket to load data into. The drop-down menu will include buckets tied to the `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),` that you have associated with your `,_jsx(_components.a,{href:`/docs/guides/environments`,children:`environment`}),`.`]})})}),_jsxs(Tab,{title:`Azure Storage`,id:`azure-storage-5`,children:[_jsx(ResponseField,{name:`Storage Account`,type:`drop-down`,required:true,id:`param-storage-account-4`,children:_jsxs(_components.p,{children:[`Select a storage account linked to your desired blob container to be used for staging the data. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-us/azure/storage/common/storage-account-overview`,children:`Storage account overview`}),`.`]})}),_jsx(ResponseField,{name:`Container`,type:`drop-down`,required:true,id:`param-container-4`,children:_jsxs(_components.p,{children:[`Select a Blob container to be used for staging the data. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-us/azure/storage/blobs/storage-blobs-introduction`,children:`Introduction to Azure Blob storage`}),`.`]})})]}),_jsxs(Tab,{title:`Google Cloud Storage`,id:`google-cloud-storage-4`,children:[_jsx(ResponseField,{name:`GCS Bucket`,type:`drop-down`,required:true,id:`param-gcs-bucket-3`,children:_jsxs(_components.p,{children:[`The drop-down menu will include Google Cloud Storage (GCS) buckets tied to the `,_jsx(_components.a,{href:`/docs/guides/cloud-credentials`,children:`cloud provider credentials`}),` that you have associated with your `,_jsx(_components.a,{href:`/docs/guides/environments`,children:`environment`}),`.`]})}),_jsx(ResponseField,{name:`Overwrite`,type:`boolean`,required:true,id:`param-overwrite-3`,children:_jsx(_components.p,{children:`Select whether to overwrite files of the same name when this pipeline runs. Default is Yes.`})})]})]})]})]})]}),_jsxs(Tab,{title:`Google BigQuery`,id:`google-bigquery`,\"data-child-tab-ids\":`[\"google-bigquery-2\",\"cloud-storage-4\"]`,children:[_jsxs(ResponseField,{name:`Destination`,type:`drop-down`,required:true,id:`param-destination-3`,children:[_jsx(_components.p,{children:`Select the destination for your data:`}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Google BigQuery:`}),` Load your data into a table in Google BigQuery.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Cloud Storage:`}),` Load your data directly into files in your preferred cloud storage location. The format of these files can differ between source systems and will not have a file extension. Check the output to determine the format of the data.`]}),`\n`]})]}),_jsxs(Tabs,{children:[_jsxs(Tab,{title:`Google BigQuery`,id:`google-bigquery-2`,children:[_jsx(ResponseField,{name:`Project`,type:`drop-down`,required:true,id:`param-project`,children:_jsxs(_components.p,{children:[`Select the Google BigQuery project to load data into. The special value `,_jsx(_components.code,{children:`[Environment Default]`}),` uses the project defined in the environment.`]})}),_jsx(ResponseField,{name:`Dataset`,type:`drop-down`,required:true,id:`param-dataset`,children:_jsxs(_components.p,{children:[`Select the Google BigQuery dataset to load data into. The special value `,_jsx(_components.code,{children:`[Environment Default]`}),` uses the dataset defined in the environment.`]})}),_jsx(ResponseField,{name:`Table Name`,type:`string`,required:true,id:`param-table-name-3`,children:_jsxs(_components.p,{children:[`The name of the table to be created in your Google BigQuery project. You can use a `,_jsx(_components.a,{href:`/docs/components/table-input`,children:`Table Input`}),` component in a transformation pipeline to access and transform this data after it has been loaded.`]})}),_jsx(ResponseField,{name:`Load Strategy`,type:`drop-down`,required:true,id:`param-load-strategy-6`,children:_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Replace:`}),` If the specified table name already exists, that table will be destroyed and replaced by the table created during this pipeline run.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Truncate and Insert:`}),` Each time the pipeline runs, two operations are performed: first, the table is truncated, meaning all existing rows are deleted. Then, your new rows are inserted. The table itself is never destroyed and recreated.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Fail if Exists:`}),` If the specified table name already exists, this pipeline will fail to run.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Append:`}),` If the specified table name already exists, then the data is inserted without altering or deleting the existing data in the table. It’s appended onto the end of the existing data in the table. If the specified table name doesn’t exist, then the table will be created, and your data will be inserted into the table.`]}),`\n`]})}),_jsxs(ResponseField,{name:`Column Mapping`,type:`column editor`,id:`param-column-mapping-1`,children:[_jsxs(_components.p,{children:[`In the tree structure representing the source data, select the checkboxes next to the fields that you want to output from this connector, then click `,_jsx(_components.strong,{children:`Save`}),`.`]}),_jsx(_components.p,{children:`Unselected fields will not be included in the output from this connector.`}),_jsxs(_components.p,{children:[`For each field you select, you can click the `,_jsx(_components.strong,{children:`Edit Alias`}),` icon to change the name used for this field in all downstream components in this pipeline.`]})]}),_jsxs(ResponseField,{name:`Primary Keys`,type:`object selector`,id:`param-primary-keys-1`,children:[_jsx(_components.p,{children:`Select one or more columns to use as the table’s primary key. If you select multiple columns, the component creates a composite primary key using the combination of the selected columns.`}),_jsx(_components.p,{children:`Leave this blank to create the table without a primary key.`})]}),_jsx(ResponseField,{name:`Clean Staged Files`,type:`boolean`,required:true,id:`param-clean-staged-files-3`,children:_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Yes:`}),` Staged files will be destroyed after data is loaded. This is the default setting.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`No:`}),` Staged files are retained in the staging area after data is loaded.`]}),`\n`]})}),_jsx(ResponseField,{name:`Partition Type`,type:`drop-down`,id:`param-partition-type`,children:_jsxs(_components.p,{children:[`Select the time period to use to partition the data loaded into your table, for example `,_jsx(_components.strong,{children:`Day`}),` or `,_jsx(_components.strong,{children:`Month`}),`.`]})}),_jsx(ResponseField,{name:`GCS Bucket`,type:`drop-down`,required:true,id:`param-gcs-bucket-4`,children:_jsx(_components.p,{children:`Select the Google Cloud Storage (GCS) bucket linked to your Google Cloud Platform account.`})}),_jsx(ResponseField,{name:`Overwrite`,type:`boolean`,required:true,id:`param-overwrite-4`,children:_jsxs(_components.p,{children:[`Select whether to overwrite files of the same name when this pipeline runs. Default is `,_jsx(_components.strong,{children:`Yes`}),`.`]})})]}),_jsxs(Tab,{title:`Cloud Storage`,id:`cloud-storage-4`,children:[_jsx(ResponseField,{name:`Load Strategy`,type:`drop-down`,id:`param-load-strategy-7`,children:_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Append Files in Folder:`}),` Appends files to storage folder. This is the default setting.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Overwrite Files in Folder:`}),` Overwrite existing files with matching structure.`]}),`\n`]})}),_jsx(ResponseField,{name:`Folder Path`,type:`string`,id:`param-folder-path-3`,children:_jsx(_components.p,{children:`The folder path for the files to be written to. Note that this path follows, but does not include, the bucket or container name.`})}),_jsx(ResponseField,{name:`File Prefix`,type:`string`,id:`param-file-prefix-3`,children:_jsx(_components.p,{children:`A string of characters that precedes the name of the written files. This can be useful for organizing database objects.`})}),_jsx(ResponseField,{name:`Storage`,type:`drop-down`,required:true,id:`param-storage-3`,children:_jsx(_components.p,{children:`A cloud storage location to load your data into files for storage. Choose either Amazon S3, Azure Storage, or Google Cloud Storage.`})})]})]})]})]}),`\n`,_jsx(Heading,{level:`3`,id:`advanced-settings`,children:`Advanced Settings`}),`\n`,_jsxs(ResponseField,{name:`Log Level`,type:`drop-down`,id:`param-log-level`,children:[_jsxs(_components.p,{children:[`Select the severity level of logging. Choose from `,_jsx(_components.strong,{children:`Error`}),`, `,_jsx(_components.strong,{children:`Warn`}),`, `,_jsx(_components.strong,{children:`Info`}),`, `,_jsx(_components.strong,{children:`Debug`}),`, or `,_jsx(_components.strong,{children:`Trace`}),`. The default setting is `,_jsx(_components.strong,{children:`Error`}),`.`]}),_jsxs(_components.p,{children:[`Logs can be found in the `,_jsx(_components.a,{href:`/docs/guides/using-designer#logging`,children:`Logging`}),` tab in `,` when viewing details about a task after the pipeline has been run.`]})]}),`\n`,_jsxs(ResponseField,{name:`Load Selected Data`,type:`boolean`,id:`param-load-selected-data`,children:[_jsxs(_components.p,{children:[`Choose whether to return the entire payload or only selected data objects. Read `,_jsx(_components.a,{href:`/docs/guides/custom-connector-setup#response`,children:`Response`}),` to learn how to select the data objects to include in your API response.`]}),_jsxs(_components.ul,{children:[`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`No:`}),` (Default) Return the entire payload.`]}),`\n`,_jsxs(_components.li,{children:[_jsx(_components.strong,{children:`Yes:`}),` Only return the objects that have been `,_jsx(_components.strong,{children:`Set as selected data`}),` when configuring the response structure in the custom connector editor.`]}),`\n`]})]}),`\n`,_jsx(Heading,{level:`2`,id:`deactivate-soft-delete-for-azure-blobs-databricks`,children:`Deactivate soft delete for Azure blobs (Databricks)`}),`\n`,_jsxs(_components.p,{children:[`If you intend to set your destination as Databricks `,_jsx(_components.em,{children:`and`}),` your stage platform as Azure Storage, you must turn off the “Enable soft delete for blobs” setting in your Azure account for your pipeline to run successfully. To do this:`]}),`\n`,_jsxs(_components.ol,{children:[`\n`,_jsxs(_components.li,{children:[`In the `,_jsx(_components.a,{href:`https://portal.azure.com/`,children:`Azure portal`}),`, navigate to your storage account.`]}),`\n`,_jsxs(_components.li,{children:[`In the menu, under `,_jsx(_components.strong,{children:`Data management`}),`, click `,_jsx(_components.strong,{children:`Data protection`}),`.`]}),`\n`,_jsxs(_components.li,{children:[`Clear the `,_jsx(_components.strong,{children:`Enable soft delete for blobs`}),` checkbox. For more information, read `,_jsx(_components.a,{href:`https://learn.microsoft.com/en-gb/azure/storage/blobs/soft-delete-blob-overview`,children:`Soft delete for blobs`}),`.`]}),`\n`]})]})}function MDXContent(props={}){const{wrapper:MDXLayout}={..._provideComponents(),...props.components};return MDXLayout?_jsx(MDXLayout,{...props,children:_jsx(_createMdxContent,{...props})}):_createMdxContent(props)}return{designer_0,designer,maia,ComponentMetadata,default:MDXContent};function _missingMdxReference(id,component){throw new Error(`Expected `+(component?`component`:`object`)+\" `\"+id+\"` to be defined: you likely forgot to import, pass, or provide it.\")}","frontmatter":{},"scope":{"config":{},"pageMetadata":{"title":"Custom connectors in Designer","description":null,"href":"/docs/components/custom-connector"}}},"mdxSourceWithNoJs":null,"noJsSplices":[[161,2740,"ComponentMetadata=()=>null;const designer=`Designer`;const maia=`Maia`;function "],[4258,4276,"`Designer`,`. Custom"],[4808,4826,"`Maia`,` provides a "],[5459,5477,"`Designer`,` canvas,"],[63330,63370,"ComponentMetadata,designer,maia,default:"]]},"mdxExtracts":{"tableOfContents":[{"title":"Prerequisites","slug":"prerequisites","depth":2,"children":[]},{"title":"Properties","slug":"properties","depth":2,"children":[{"title":"Connect","slug":"connect","depth":3,"children":[]},{"title":"Configure","slug":"configure","depth":3,"children":[]},{"title":"Destination","slug":"destination","depth":3,"children":[]},{"title":"Advanced Settings","slug":"advanced-settings","depth":3,"children":[]}]},{"title":"Deactivate soft delete for Azure blobs (Databricks)","slug":"deactivate-soft-delete-for-azure-blobs-databricks","depth":2,"children":[]}],"codeExamples":{}},"jsLinks":["/_mintlify/custom/matillion/1eecfe13cb8a36b9/scripts/feedback-widget.js","/_mintlify/custom/matillion/b8b092d8d394af0f/scripts/ga4-events.js","/_mintlify/custom/matillion/5baf5b38fe34f41f/scripts/iubenda.js","/_mintlify/custom/matillion/573540f1e9142d9c/scripts/kapa-widget.js","/_mintlify/custom/matillion/a52101786f42c23b/scripts/ph-pageleaves.js","/_mintlify/custom/matillion/b8f0431aaf9ef855/scripts/cookie-preferences-button.js","/_mintlify/custom/matillion/a3d31738f4c89949/scripts/status-badge-nav.js","/_mintlify/custom/matillion/b444958fcc79163d/scripts/text-size-slider.js","/_mintlify/custom/matillion/f7016c1e98f77080/scripts/posthog-events.js"],"subdomain":"matillion","siteName":"Maia Documentation","title":"Custom connectors in Designer - Maia Documentation","relatedPagesConfig":{"enabled":true,"limit":5,"mode":"manual"},"navMaps":{"firstHrefInVersion":{},"firstHrefInLanguage":{}},"lastModified":"2026-11-12T18:15:27.566Z","originalFileLocation":"docs/components/custom-connector.mdx"},"headHtml":"\u003cscript data-mint-head>(function(){var d=document.documentElement,p=\"light\";if(!p){try{p=localStorage.getItem(\"isDarkMode\")}catch(e){}if(p==='true')p='dark';else if(p==='false')p='light';if(p!=='light'\u0026\u0026p!=='dark'\u0026\u0026p!=='system')p=\"light\"}var r=p==='system'?(matchMedia(\"(prefers-color-scheme: dark)\").matches?'dark':'light'):p;d.classList.remove(r==='dark'?'light':'dark');d.classList.add(r);d.setAttribute(\"data-theme-preference\",p)})();\u003c/script>\u003cmeta data-mint-head name=\"application-name\" content=\"Maia Documentation\">\u003cmeta data-mint-head name=\"generator\" content=\"Mintlify\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/matillion/eb7R3mt9l7P1vThW/_generated/favicon/android-chrome-192x192.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"apple-touch-icon\" type=\"image/png\" sizes=\"180x180\" href=\"/mintlify-assets/_mintlify/favicons/matillion/eb7R3mt9l7P1vThW/_generated/favicon/apple-touch-icon.png\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/matillion/eb7R3mt9l7P1vThW/_generated/favicon/favicon-16x16.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/matillion/eb7R3mt9l7P1vThW/_generated/favicon/favicon-32x32.png\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/matillion/eb7R3mt9l7P1vThW/_generated/favicon/favicon.ico\" media=\"(prefers-color-scheme: light)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"192x192\" href=\"/mintlify-assets/_mintlify/favicons/matillion/eb7R3mt9l7P1vThW/_generated/favicon-dark/android-chrome-192x192.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"16x16\" href=\"/mintlify-assets/_mintlify/favicons/matillion/eb7R3mt9l7P1vThW/_generated/favicon-dark/favicon-16x16.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"icon\" type=\"image/png\" sizes=\"32x32\" href=\"/mintlify-assets/_mintlify/favicons/matillion/eb7R3mt9l7P1vThW/_generated/favicon-dark/favicon-32x32.png\" media=\"(prefers-color-scheme: dark)\">\u003clink data-mint-head rel=\"shortcut icon\" type=\"image/x-icon\" href=\"/mintlify-assets/_mintlify/favicons/matillion/eb7R3mt9l7P1vThW/_generated/favicon-dark/favicon.ico\" media=\"(prefers-color-scheme: dark)\">\u003cmeta data-mint-head property=\"og:title\" content=\"Custom connectors in Designer - Maia Documentation\">\u003cmeta data-mint-head property=\"og:site_name\" content=\"Maia Documentation\">\u003cmeta data-mint-head property=\"og:url\" content=\"https://docs.maia.ai/docs/components/custom-connector\">\u003cmeta data-mint-head property=\"og:type\" content=\"website\">\u003cmeta data-mint-head property=\"og:image\" content=\"https://matillion.mintlify.site/_mintlify/api/og/v2?division=Connectors\u0026amp;title=Custom+connectors+in+Designer\u0026amp;theme=ce57a3e3dfd574362d42229a\u0026amp;signature=0b8ff999113ad7f9cfddbade519a150fd1aa2943811e9b79fdde63a95de333b7\">\u003cmeta data-mint-head property=\"og:image:width\" content=\"1200\">\u003cmeta data-mint-head property=\"og:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"twitter:card\" content=\"summary_large_image\">\u003cmeta data-mint-head name=\"twitter:title\" content=\"Custom connectors in Designer - Maia Documentation\">\u003cmeta data-mint-head name=\"twitter:image\" content=\"https://matillion.mintlify.site/_mintlify/api/og/v2?division=Connectors\u0026amp;title=Custom+connectors+in+Designer\u0026amp;theme=ce57a3e3dfd574362d42229a\u0026amp;signature=0b8ff999113ad7f9cfddbade519a150fd1aa2943811e9b79fdde63a95de333b7\">\u003cmeta data-mint-head name=\"twitter:image:width\" content=\"1200\">\u003cmeta data-mint-head name=\"twitter:image:height\" content=\"630\">\u003cmeta data-mint-head name=\"msapplication-config\" content=\"/mintlify-assets/_mintlify/favicons/matillion/eb7R3mt9l7P1vThW/_generated/favicon/browserconfig.xml\">\u003cmeta data-mint-head name=\"apple-mobile-web-app-title\" content=\"Maia Documentation\">\u003cmeta data-mint-head name=\"msapplication-TileColor\" content=\"#001f1d\">\u003cmeta data-mint-head name=\"google-site-verification\" content=\"0VLXGRESICGHnU66TdhEgUMBRRubPnW8JTkzOpiiNn0\">\u003clink data-mint-head rel=\"canonical\" href=\"https://docs.maia.ai/docs/components/custom-connector\">\u003clink data-mint-head rel=\"alternate\" type=\"application/xml\" href=\"/sitemap.xml\">\u003clink data-mint-head rel=\"alternate\" type=\"text/markdown\" href=\"/docs/components/custom-connector.md\">\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@graph\":[{\"@type\":\"Organization\",\"@id\":\"https://docs.maia.ai/#organization\",\"name\":\"Maia Documentation\",\"url\":\"https://docs.maia.ai\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https://mintcdn.com/matillion/GyuNBHAx-GXnVlKX/images/maia-by-matillion-light.svg?fit=max\u0026auto=format\u0026n=GyuNBHAx-GXnVlKX\u0026q=85\u0026s=2c48b854f858c5a835450b3f34360db7\"}},{\"@type\":\"WebSite\",\"@id\":\"https://docs.maia.ai#website\",\"name\":\"Maia Documentation\",\"url\":\"https://docs.maia.ai\",\"publisher\":{\"@id\":\"https://docs.maia.ai/#organization\"}},{\"@type\":\"WebPage\",\"@id\":\"https://docs.maia.ai/docs/components/custom-connector#webpage\",\"url\":\"https://docs.maia.ai/docs/components/custom-connector\",\"name\":\"Custom connectors in Designer\",\"dateModified\":\"2026-11-12T18:15:27.566Z\",\"isPartOf\":{\"@id\":\"https://docs.maia.ai#website\"},\"breadcrumb\":{\"@id\":\"https://docs.maia.ai/docs/components/custom-connector#breadcrumb\"}},{\"@type\":\"BreadcrumbList\",\"@id\":\"https://docs.maia.ai/docs/components/custom-connector#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Connectors\",\"item\":\"https://docs.maia.ai/docs/components/connectors-overview\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Custom connectors in Designer\",\"item\":\"https://docs.maia.ai/docs/components/custom-connector\"}]},{\"@type\":[\"Article\",\"TechArticle\"],\"@id\":\"https://docs.maia.ai/docs/components/custom-connector#article\",\"headline\":\"Custom connectors in Designer\",\"name\":\"Custom connectors in Designer\",\"url\":\"https://docs.maia.ai/docs/components/custom-connector\",\"mainEntityOfPage\":{\"@id\":\"https://docs.maia.ai/docs/components/custom-connector#webpage\"},\"dateModified\":\"2026-11-12T18:15:27.566Z\",\"publisher\":{\"@id\":\"https://docs.maia.ai/#organization\"},\"isPartOf\":{\"@id\":\"https://docs.maia.ai#website\"}}]}\u003c/script>\u003cscript data-mint-head type=\"application/ld+json\">{\"@context\":\"https://schema.org\",\"@type\":\"WebSite\",\"name\":\"Maia Documentation\",\"publisher\":{\"@type\":\"Organization\",\"name\":\"Maia Documentation\"},\"provider\":{\"@type\":\"Organization\",\"name\":\"Mintlify\",\"url\":\"https://mintlify.com\"}}\u003c/script>","agentPointerHref":"/docs/components/custom-connector.md","siteName":"Maia Documentation","nav":{"global":null,"tabs":[{"tab":"Home","pages":[{"title":"Home","href":""}]},{"tab":"Admin","pages":[{"title":"Admin","href":"/docs/administration/administration-overview"}]},{"tab":"Components","pages":[{"title":"Components overview","href":"/docs/components/components-overview"},{"group":"Connectors","root":{"title":"Connectors overview","href":"/docs/components/connectors-overview"},"pages":[{"title":"Connectors overview","description":null,"href":"/docs/components/connectors-overview"},{"title":"Custom connectors in Designer","href":"/docs/components/custom-connector"},{"title":"System tables","description":null,"href":"/docs/guides/system-tables-example"},{"group":"Connector authentication guides","root":{"title":"Connector authentication","href":"/docs/guides/connector-authentication-overview"},"pages":[{"title":"Connector authentication","href":"/docs/guides/connector-authentication-overview"},{"title":"HubSpot authentication guide","href":"/docs/guides/hubspot-authentication-guide"},{"title":"Jira authentication guide","href":"/docs/guides/jira-authentication-guide"},{"title":"LinkedIn Ads authentication guide","href":"/docs/guides/linkedin-ads-authentication-guide"},{"title":"Kafka authentication guide","href":"/docs/guides/kafka-authentication-guide"},{"title":"Marketo authentication guide","href":"/docs/guides/marketo-authentication-guide"},{"title":"Microsoft Fabric Lakehouse authentication guide","href":"/docs/guides/microsoft-fabric-lakehouse-authentication-guide"},{"title":"NetSuite Query authentication guide","href":"/docs/guides/netsuite-query-authentication-guide"},{"title":"NetSuite SuiteAnalytics authentication guide","href":"/docs/guides/netsuite-suiteanalytics-authentication-guide"},{"title":"Oracle Autonomous Database authentication guide","href":"/docs/guides/oracle-autonomous-database-authentication-guide"},{"title":"Salesforce authentication guide","href":"/docs/guides/salesforce-authentication-guide"},{"title":"ServiceNow authentication guide","href":"/docs/guides/servicenow-authentication-guide"},{"title":"Workday authentication guide","href":"/docs/guides/workday-authentication-guide"}]},{"title":"ActiveCampaign","href":"/docs/components/activecampaign"},{"title":"Amazon DynamoDB Load","href":"/docs/components/amazon-dynamodb-load"},{"title":"Amazon Redshift Load","href":"/docs/components/amazon-redshift-load"},{"title":"Amplitude","href":"/docs/components/amplitude"},{"title":"Anaplan","href":"/docs/components/anaplan"},{"title":"Asana","href":"/docs/components/asana"},{"title":"Attio","href":"/docs/components/attio"},{"title":"AVRO Load","href":"/docs/components/avro-load"},{"title":"Azure Blob Storage Load","href":"/docs/components/azure-blob-storage-load"},{"title":"Azure Blob Storage Unload","href":"/docs/components/azure-blob-storage-unload"},{"title":"Azure Cosmos DB for NoSQL","href":"/docs/components/azure-cosmos-db-for-nosql"},{"title":"Azure SQL","href":"/docs/components/azure-sql"},{"title":"Bing Ads Load","href":"/docs/components/bing-ads-load"},{"title":"Bing Ads Query (deprecated)","href":"/docs/components/bing-ads-query"},{"title":"Box","href":"/docs/components/box"},{"title":"Braze","href":"/docs/components/braze"},{"title":"Brevo","href":"/docs/components/brevo"},{"title":"Bugcrowd","href":"/docs/components/bugcrowd"},{"title":"Buildkite","href":"/docs/components/buildkite"},{"title":"Chargebee","href":"/docs/components/chargebee"},{"title":"CircleCI","href":"/docs/components/circleci"},{"title":"ClickUp","href":"/docs/components/clickup"},{"title":"Coalesce","href":"/docs/components/coalesce"},{"title":"Concord","href":"/docs/components/concord"},{"title":"Confluence","href":"/docs/components/confluence"},{"title":"CSV Load","href":"/docs/components/csv-load"},{"title":"Data Transfer","href":"/docs/components/data-transfer"},{"title":"Database Query","href":"/docs/components/database-query"},{"title":"Databricks","href":"/docs/components/databricks"},{"title":"Datadog","href":"/docs/components/datadog"},{"title":"dbt Cloud","href":"/docs/components/dbt-cloud"},{"title":"Delighted","href":"/docs/components/delighted"},{"title":"Dixa","href":"/docs/components/dixa"},{"title":"Dropbox","href":"/docs/components/dropbox"},{"title":"Dynamics 365 Business Central","href":"/docs/components/dynamics-365-business-central"},{"title":"Elasticsearch Load","href":"/docs/components/elasticsearch-load"},{"title":"Email Load","href":"/docs/components/email-load"},{"title":"Eventbrite","href":"/docs/components/eventbrite"},{"title":"Excel Load","href":"/docs/components/excel-load"},{"title":"Excel Query","href":"/docs/components/excel-query"},{"title":"Facebook Ads","href":"/docs/components/facebook-ads"},{"title":"Facebook Ads Query (deprecated)","href":"/docs/components/facebook-ads-query"},{"title":"Facebook Ads Query example configurations","href":"/docs/guides/facebook-ads-query-example"},{"title":"Facebook Load","href":"/docs/components/facebook-load"},{"title":"Facebook Query (deprecated)","href":"/docs/components/facebook-query"},{"title":"Fanatics","href":"/docs/components/fanatics"},{"title":"FireHydrant","href":"/docs/components/firehydrant"},{"title":"Float","href":"/docs/components/flex-float"},{"title":"Freshdesk","href":"/docs/components/freshdesk"},{"title":"GitHub","href":"/docs/components/github"},{"title":"Gmail","href":"/docs/components/gmail"},{"title":"Gong","href":"/docs/components/gong"},{"title":"Google Ads Load","href":"/docs/components/google-ads-load"},{"title":"Google Analytics","href":"/docs/components/google-analytics"},{"title":"Google Analytics Query (deprecated)","href":"/docs/components/google-analytics-query"},{"title":"Google BigQuery (connector)","href":"/docs/components/google-bigquery"},{"title":"Google BigQuery Load","href":"/docs/components/google-bigquery-load"},{"title":"Google Cloud Storage Unload","href":"/docs/components/google-cloud-storage-unload"},{"title":"Google Sheets","href":"/docs/components/google-sheets"},{"title":"HubSpot Load","href":"/docs/components/hubspot-load"},{"title":"HubSpot Query (deprecated)","href":"/docs/components/hubspot-query"},{"title":"IBM Db2 for i Load","href":"/docs/components/ibm-db2-load"},{"title":"Infobip","href":"/docs/components/infobip"},{"title":"Intercom","href":"/docs/components/intercom"},{"title":"JDBC Load","href":"/docs/components/jdbc-load"},{"title":"Jira Load","href":"/docs/components/jira-load"},{"title":"Jotform","href":"/docs/components/jotform"},{"title":"JSON Load","href":"/docs/components/json-load"},{"title":"Kafka","href":"/docs/components/kafka"},{"title":"Klaviyo","href":"/docs/components/klaviyo"},{"title":"LaunchDarkly","href":"/docs/components/launchdarkly"},{"title":"LDAP Load","href":"/docs/components/ldap-load"},{"title":"LDAP Query (deprecated)","href":"/docs/components/ldap-query"},{"title":"LinkedIn Ads","href":"/docs/components/linkedin-ads"},{"title":"Lob","href":"/docs/components/lob"},{"title":"Maia Flex connector","href":"/docs/components/maia-flex-connector"},{"title":"Mailchimp","href":"/docs/components/mailchimp"},{"title":"Mailgun","href":"/docs/components/mailgun"},{"title":"Mandrill","href":"/docs/components/mandrill"},{"title":"MariaDB Load","href":"/docs/components/mariadb-load"},{"title":"MariaDB Unload","href":"/docs/components/mariadb-unload"},{"title":"Marketo Load","href":"/docs/components/marketo-load"},{"title":"Microsoft Dataverse","href":"/docs/components/microsoft-dataverse"},{"title":"Microsoft Dynamics 365 Load","href":"/docs/components/microsoft-dynamics-365-load"},{"title":"Microsoft Dynamics CRM Load","href":"/docs/components/microsoft-dynamics-crm-load"},{"title":"Microsoft Exchange","href":"/docs/components/microsoft-exchange"},{"title":"Microsoft SQL Server Load","href":"/docs/components/microsoft-sql-server-load"},{"title":"Mixpanel","href":"/docs/components/mixpanel"},{"title":"MongoDB Load","href":"/docs/components/mongodb-load"},{"title":"NetSuite Query","href":"/docs/components/netsuite-query"},{"title":"NetSuite SuiteAnalytics (deprecated)","href":"/docs/components/netsuite-suiteanalytics"},{"title":"NetSuite SuiteAnalytics Load","href":"/docs/components/netsuite-suiteanalytics-load"},{"title":"New Relic","href":"/docs/components/new-relic"},{"title":"Notion","href":"/docs/components/notion"},{"title":"OData","href":"/docs/components/odata"},{"title":"OData Query","href":"/docs/components/odata-query"},{"title":"Ongoing WMS","href":"/docs/components/ongoing-wms"},{"title":"Open Exchange Rates","href":"/docs/components/open-exchange-rates"},{"title":"Oracle Eloqua Load","href":"/docs/components/oracle-eloqua-load"},{"title":"Oracle Fusion Cloud Financials Load","href":"/docs/components/oracle-fcf-load"},{"title":"Oracle Fusion Cloud HCM Load","href":"/docs/components/oracle-hcm-load"},{"title":"Oracle Fusion Cloud Procurement Load","href":"/docs/components/oracle-fcp-load"},{"title":"Oracle Fusion Cloud Project Management Load","href":"/docs/components/oracle-fcpm-load"},{"title":"Oracle Load","href":"/docs/components/oracle-load"},{"title":"Oracle Output","href":"/docs/components/oracle-output"},{"title":"Oracle Unload","href":"/docs/components/oracle-unload"},{"title":"Orbit","href":"/docs/components/orbit"},{"title":"ORC Load","href":"/docs/components/orc-load"},{"title":"Ortto","href":"/docs/components/ortto"},{"title":"PagerDuty","href":"/docs/components/pagerduty"},{"title":"Parquet Load","href":"/docs/components/parquet-load"},{"title":"PayPal","href":"/docs/components/paypal"},{"title":"Pendo","href":"/docs/components/pendo"},{"title":"Persona","href":"/docs/components/persona"},{"title":"Pingdom","href":"/docs/components/pingdom"},{"title":"Pipedrive","href":"/docs/components/pipedrive"},{"title":"PostgreSQL Load","href":"/docs/components/postgresql-load"},{"title":"PostgreSQL Unload","href":"/docs/components/postgresql-unload"},{"title":"Productboard","href":"/docs/components/productboard"},{"title":"RDS Query","href":"/docs/components/rds-query"},{"title":"Recurly","href":"/docs/components/recurly"},{"title":"RingCentral","href":"/docs/components/ringcentral"},{"title":"Rootly","href":"/docs/components/rootly"},{"title":"S3 Load","href":"/docs/components/s3-load"},{"title":"S3 Unload","href":"/docs/components/s3-unload"},{"title":"Sage Intacct Load","href":"/docs/components/sage-intacct-load"},{"title":"Salesforce Load","href":"/docs/components/salesforce-load"},{"title":"Salesforce Marketing Cloud Query","href":"/docs/components/salesforce-marketing-cloud-query"},{"title":"Salesforce Pardot","href":"/docs/components/salesforce-pardot"},{"title":"Salesforce Query (deprecated)","href":"/docs/components/salesforce-query"},{"title":"Salesloft","href":"/docs/components/salesloft"},{"title":"SAP NetWeaver (deprecated)","href":"/docs/components/sap-netweaver"},{"title":"SAP NetWeaver Load","href":"/docs/components/sap-netweaver-load"},{"title":"SAP ODP","href":"/docs/components/sap-odp"},{"title":"SAP SuccessFactors","href":"/docs/components/sap-successfactors"},{"title":"SeatGeek","href":"/docs/components/seatgeek"},{"title":"SendGrid","href":"/docs/components/sendgrid"},{"title":"ServiceNow Load","href":"/docs/components/servicenow-load"},{"title":"ServiceNow Query","href":"/docs/components/servicenow-query"},{"title":"SharePoint Load","href":"/docs/components/sharepoint-load"},{"title":"SharePoint Query (deprecated)","href":"/docs/components/sharepoint-query"},{"title":"Shopify","href":"/docs/components/shopify"},{"title":"Shopify Load","href":"/docs/components/shopify-load"},{"title":"Slack","href":"/docs/components/slack"},{"title":"Smartsheet","href":"/docs/components/smartsheet"},{"title":"Snapchat","href":"/docs/components/snapchat"},{"title":"Snowflake Load","href":"/docs/components/snowflake-load"},{"title":"Snyk","href":"/docs/components/snyk"},{"title":"Square","href":"/docs/components/square"},{"title":"Stripe Load","href":"/docs/components/stripe-load"},{"title":"Stripe Query (deprecated)","href":"/docs/components/stripe-query"},{"title":"SugarCRM Load","href":"/docs/components/sugarcrm-load"},{"title":"SurveyMonkey","href":"/docs/components/surveymonkey"},{"title":"Sybase ASE Load","href":"/docs/components/sybase-ase-load"},{"title":"Ticketmaster","href":"/docs/components/ticketmaster"},{"title":"TikTok","href":"/docs/components/tiktok"},{"title":"Toggl","href":"/docs/components/toggl"},{"title":"Twilio","href":"/docs/components/twilio"},{"title":"Unstructured.io","href":"/docs/components/unstructured"},{"title":"Workday","href":"/docs/components/workday"},{"title":"Workday Custom Reports","href":"/docs/components/workday-custom-reports"},{"title":"Workday Load","href":"/docs/components/workday-load"},{"title":"X Ads Load","href":"/docs/components/x-ads-load"},{"title":"Yelp","href":"/docs/components/yelp"},{"title":"Zendesk Talk","href":"/docs/components/zendesk-talk"},{"title":"Zendesk Ticketing","href":"/docs/components/zendesk-ticketing"},{"title":"Zoom","href":"/docs/components/zoom"},{"title":"Zoho CRM Load","href":"/docs/components/zoho-crm-load"},{"title":"Zuora","href":"/docs/components/zuora"},{"title":"Zuora Bulk","href":"/docs/components/zuora-bulk"}]},{"group":"Orchestration","root":{"title":"Orchestration overview","href":"/docs/components/orchestration-pipeline-overview"},"pages":[{"title":"Orchestration overview","href":"/docs/components/orchestration-pipeline-overview"},{"group":"Pipeline control","root":{"title":"Pipeline control components","href":"/docs/components/pipeline-control-components"},"pages":[{"title":"Pipeline control components","href":"/docs/components/pipeline-control-components"},{"title":"And component","href":"/docs/components/and"},{"title":"Begin transaction","href":"/docs/components/begin"},{"title":"Commit transaction","href":"/docs/components/commit"},{"title":"End Failure","href":"/docs/components/end-failure"},{"title":"End Success","href":"/docs/components/end-success"},{"title":"If component","href":"/docs/components/if"},{"title":"Or component","href":"/docs/components/or"},{"title":"Retry","href":"/docs/components/retry"},{"title":"Rollback transaction","href":"/docs/components/rollback"},{"title":"Run Orchestration","href":"/docs/components/run-orchestration"},{"title":"Run Shared Pipeline","href":"/docs/components/run-shared-pipeline"},{"title":"Run Transformation","href":"/docs/components/run-transformation"},{"title":"Start component","href":"/docs/components/start"},{"title":"Start Compute","href":"/docs/components/start-compute"}]},{"group":"DDL","root":{"title":"DDL components","href":"/docs/components/ddl-components"},"pages":[{"title":"DDL components","href":"/docs/components/ddl-components"},{"title":"Add Partition","href":"/docs/components/add-partition"},{"title":"Alter Table","href":"/docs/components/alter-table"},{"title":"Alter Warehouse","href":"/docs/components/alter-warehouse"},{"title":"Analyze Tables","href":"/docs/components/analyze-tables"},{"title":"Create External Table","href":"/docs/components/create-external-table"},{"title":"Create File Format","href":"/docs/components/create-file-format"},{"title":"Create Stream","href":"/docs/components/create-stream"},{"title":"Create Table","href":"/docs/components/create-table"},{"title":"Delete Tables","href":"/docs/components/delete-tables"},{"title":"Google Drive Table","href":"/docs/components/google-drive-table"},{"title":"Optimize","href":"/docs/components/optimize"},{"title":"Refresh External Table","href":"/docs/components/refresh-external-table"},{"title":"Refresh Materialized View","href":"/docs/components/refresh-materialized-view"},{"title":"Refresh Table","href":"/docs/components/refresh-table"},{"title":"Schema Copy","href":"/docs/components/schema-copy"},{"title":"Truncate Table","href":"/docs/components/truncate-table"},{"title":"Vacuum","href":"/docs/components/vacuum"},{"title":"Volume to Delta Table","href":"/docs/components/volume-to-delta-table"}]},{"group":"Scripting","root":{"title":"Scripting components","href":"/docs/components/scripting-components"},"pages":[{"title":"Scripting components","href":"/docs/components/scripting-components"},{"title":"dbt Core","href":"/docs/components/dbt-core"},{"title":"Python Pushdown","href":"/docs/components/python-pushdown"},{"title":"Python Script","href":"/docs/components/python-script"},{"title":"Run Notebook","href":"/docs/components/run-notebook"},{"title":"Script Pushdown","href":"/docs/components/script-pushdown"},{"title":"SQL Script","href":"/docs/components/sql-script"}]},{"group":"AI","root":{"title":"AI components","href":"/docs/components/ai-components"},"pages":[{"title":"AI components","href":"/docs/components/ai-components"},{"title":"Amazon Bedrock Prompt","href":"/docs/components/amazon-bedrock-prompt"},{"title":"Amazon Textract Input","href":"/docs/components/amazon-textract-input"},{"title":"Amazon Transcribe Input","href":"/docs/components/amazon-transcribe-input"},{"title":"Azure Document Intelligence","href":"/docs/components/azure-document-intelligence"},{"title":"Azure OpenAI Prompt","href":"/docs/components/azure-openai-prompt"},{"title":"OpenAI Prompt","href":"/docs/components/openai-prompt"},{"title":"Azure Speech Transcribe","href":"/docs/components/azure-speech-transcribe"},{"title":"Chunk Text","href":"/docs/components/chunk-text"},{"title":"Cortex Finetune","href":"/docs/components/cortex-finetune"},{"title":"Pinecone Vector Query","href":"/docs/components/pinecone-vector-query"},{"title":"Snowpark Container Prompt","tag":"Preview","href":"/docs/components/snowpark-container-prompt"},{"title":"How to create an app in Snowpark Container Services","tag":"Preview","href":"/docs/guides/snowpark-container-app-how-to"},{"title":"TypeSafe Decide","href":"/docs/components/typesafe-decide"},{"title":"Google Vertex AI Prompt","href":"/docs/components/vertex-ai-prompt"}]},{"group":"Variables","root":{"title":"Variables components","href":"/docs/components/variables-components"},"pages":[{"title":"Variables components","href":"/docs/components/variables-components"},{"title":"Append to Grid","href":"/docs/components/append-to-grid"},{"title":"Assert Scalar Variables","href":"/docs/components/assert-scalar-variables"},{"title":"JDBC Table Metadata to Grid","href":"/docs/components/jdbc-table-metadata-to-grid"},{"title":"Print Variables","href":"/docs/components/print-variables"},{"title":"Query Result To Grid","href":"/docs/components/query-result-to-grid"},{"title":"Query Result To Scalar","href":"/docs/components/query-result-to-scalar"},{"title":"Remove from Grid","href":"/docs/components/remove-from-grid"},{"title":"Show to Grid","href":"/docs/components/show-to-grid"},{"title":"Table Metadata to Grid","href":"/docs/components/table-metadata-to-grid"},{"title":"Update Scalar","href":"/docs/components/update-scalar"}]},{"group":"Messaging","root":{"title":"Messaging components","href":"/docs/components/messaging-components"},"pages":[{"title":"Messaging components","href":"/docs/components/messaging-components"},{"title":"Azure Queue Storage Message","href":"/docs/components/azure-queue-storage-message"},{"title":"CloudWatch Publish","href":"/docs/components/cloudwatch-publish"},{"title":"Cloud Pub/Sub","href":"/docs/components/pubsub"},{"title":"Send Email","href":"/docs/components/send-email"},{"title":"SNS Message","href":"/docs/components/sns-message"},{"title":"SQS Message","href":"/docs/components/sqs-message"},{"title":"Webhook Post","href":"/docs/components/webhook-post"}]},{"group":"Assert","root":{"title":"Assert components","href":"/docs/components/assert-components"},"pages":[{"title":"Assert components","href":"/docs/components/assert-components"},{"title":"Assert External Table","href":"/docs/components/assert-external-table"},{"title":"Assert Scalar Variables","href":"/docs/components/assert-scalar-variables"},{"title":"Assert Table","href":"/docs/components/assert-table"}]},{"group":"Metadata","root":{"title":"Metadata components","href":"/docs/components/metadata-components"},"pages":[{"title":"Metadata components","href":"/docs/components/metadata-components"},{"title":"JDBC Table Metadata to Grid","href":"/docs/components/jdbc-table-metadata-to-grid"},{"title":"Table Metadata to Grid","href":"/docs/components/table-metadata-to-grid"}]},{"group":"Iterators","root":{"title":"Iterator components","href":"/docs/guides/iterator-components"},"pages":[{"title":"Iterator components","href":"/docs/guides/iterator-components"},{"title":"File Iterator","href":"/docs/components/file-iterator"},{"title":"Fixed Iterator","href":"/docs/components/fixed-iterator"},{"title":"Grid Iterator","href":"/docs/components/grid-iterator"},{"title":"Loop Iterator","href":"/docs/components/loop-iterator"},{"title":"Table Iterator","href":"/docs/components/table-iterator"}]},{"group":"Output and upsert","root":{"title":"Output and upsert components","href":"/docs/components/output-and-upsert-components"},"pages":[{"title":"Output and upsert components","href":"/docs/components/output-and-upsert-components"},{"title":"Amazon OpenSearch Upsert","href":"/docs/components/amazon-opensearch-upsert"},{"title":"Pinecone Vector Upsert","href":"/docs/components/pinecone-vector-upsert"},{"title":"Postgres Vector Upsert","href":"/docs/components/postgres-vector-upsert"},{"title":"RDS Bulk Output","href":"/docs/components/rds-bulk-output"},{"title":"Microsoft SQL Server Output","href":"/docs/components/sql-server-output"},{"title":"Salesforce Output","href":"/docs/components/salesforce-output"},{"title":"Snowflake Vector Upsert","href":"/docs/components/snowflake-vector-upsert"}]}]},{"group":"Transformation","root":{"title":"Transformation overview","href":"/docs/components/transformation-pipeline-overview"},"pages":[{"title":"Transformation overview","href":"/docs/components/transformation-pipeline-overview"},{"title":"Aggregate","href":"/docs/components/aggregate"},{"title":"Assert View","href":"/docs/components/assert-view"},{"title":"Calculator","href":"/docs/components/calculator"},{"title":"Construct Struct","href":"/docs/components/construct-struct"},{"title":"Construct Variant","href":"/docs/components/construct-variant"},{"title":"Convert String To Struct","href":"/docs/components/convert-string-to-struct"},{"title":"Convert Type","href":"/docs/components/convert-type"},{"group":"Cortex","pages":[{"title":"Cortex Completions","href":"/docs/components/cortex-completions"},{"title":"Cortex Embed","href":"/docs/components/cortex-embed"},{"title":"Cortex Extract Answer","href":"/docs/components/cortex-extract-answer"},{"title":"Cortex Multi Prompt","href":"/docs/components/cortex-multi-prompt"},{"title":"Cortex Parse Document","tag":"Preview","href":"/docs/components/cortex-parse-document"},{"title":"Cortex Sentiment","href":"/docs/components/cortex-sentiment"},{"title":"Cortex Summarize","href":"/docs/components/cortex-summarize"},{"title":"Cortex Translate","href":"/docs/components/cortex-translate"}]},{"title":"Create View","href":"/docs/components/create-view"},{"title":"Data Cleanse","href":"/docs/components/data-cleanse"},{"group":"Databricks AI","pages":[{"title":"AI Analyze Sentiment","href":"/docs/components/databricks-ai-analyze-sentiment"},{"title":"AI Classify","href":"/docs/components/databricks-ai-classify"},{"title":"AI Extract","href":"/docs/components/databricks-ai-extract"},{"title":"AI Fix Grammar","href":"/docs/components/databricks-ai-fix-grammar"},{"title":"AI Mask","href":"/docs/components/databricks-ai-mask"},{"title":"AI Query","href":"/docs/components/databricks-ai-query"},{"title":"AI Similarity","href":"/docs/components/databricks-ai-similarity"},{"title":"AI Summarize","href":"/docs/components/databricks-ai-summarize"},{"title":"AI Translate","href":"/docs/components/databricks-ai-translate"},{"title":"Databricks Vector Search","href":"/docs/components/databricks-vector-search"}]},{"title":"Detect Changes","href":"/docs/components/detect-changes"},{"title":"Distinct","href":"/docs/components/distinct"},{"title":"Document AI Predict","tag":"Preview","href":"/docs/components/document-ai-predict"},{"title":"Except","href":"/docs/components/except"},{"title":"Extract Nested Data","href":"/docs/components/extract-nested-data"},{"title":"Extract Structured Data","href":"/docs/components/extract-structured-data"},{"title":"Filter","href":"/docs/components/filter"},{"title":"First/Last","href":"/docs/components/first-last"},{"title":"Fixed Flow","href":"/docs/components/fixed-flow"},{"title":"Flatten Variant","href":"/docs/components/flatten-variant"},{"title":"Generate Sequence","href":"/docs/components/generate-sequence"},{"group":"Google BigQuery AI","pages":[{"title":"Check Condition","href":"/docs/components/bigquery-check-condition"},{"title":"Classify","href":"/docs/components/bigquery-classify"},{"title":"Embed","href":"/docs/components/bigquery-embed"},{"title":"Generate","href":"/docs/components/bigquery-generate"},{"title":"Score","href":"/docs/components/bigquery-score"},{"title":"Vector Search","href":"/docs/components/bigquery-vector-search"}]},{"title":"Intersect","href":"/docs/components/intersect"},{"title":"Join","href":"/docs/components/join"},{"title":"Lead/Lag","href":"/docs/components/lead-lag"},{"title":"Map Values","href":"/docs/components/map-values"},{"title":"ML Classification","tag":"Preview","href":"/docs/components/ml-classification-sf"},{"title":"Multi Table Input","href":"/docs/components/multi-table-input"},{"title":"Pivot","href":"/docs/components/pivot"},{"title":"Rank","href":"/docs/components/rank"},{"title":"Rename","href":"/docs/components/rename"},{"title":"Rewrite Table","href":"/docs/components/rewrite-table"},{"title":"Split field","href":"/docs/components/split-field"},{"title":"SQL","href":"/docs/components/sql"},{"title":"Stream Input","href":"/docs/components/stream-input"},{"title":"Table Delete Rows","href":"/docs/components/table-delete-rows"},{"title":"Table Input","href":"/docs/components/table-input"},{"title":"Table Output","href":"/docs/components/table-output"},{"title":"Table Update","href":"/docs/components/table-update"},{"title":"Transpose Columns","href":"/docs/components/transpose-columns"},{"title":"Transpose Rows","href":"/docs/components/transpose-rows"},{"title":"Unite","href":"/docs/components/unite"},{"title":"Unpivot","href":"/docs/components/unpivot"},{"title":"Vector Search","href":"/docs/components/vector-search"},{"title":"Wildcard Table Input","href":"/docs/components/wildcard-table-input"},{"title":"Window Calculation","href":"/docs/components/window-calculation"}]},{"group":"Test pipelines","root":{"title":"Test pipelines","href":"/docs/components/test-pipelines"},"pages":[{"title":"Test pipelines","href":"/docs/components/test-pipelines"},{"title":"Assert Table Values","href":"/docs/components/assert-table-values"},{"title":"Compare Table Values","href":"/docs/components/compare-table-values"},{"title":"Create Table From Values","href":"/docs/components/create-table-from-values"},{"title":"Run Pipeline To Test","href":"/docs/components/run-pipeline-to-test"},{"title":"Zero Copy Clone","href":"/docs/components/zero-copy-clone"}]}]},{"tab":"Guides","pages":[{"title":"Guides","href":"/docs/guides/maia-overview"}]},{"tab":"API Reference","pages":[{"title":"API Reference","href":"/docs/api-reference/maia-api-overview"}]},{"tab":"Changelog","pages":[{"title":"Changelog","href":"/docs/changelog/2026-changelog"}]}]},"basePath":"","title":"Custom connectors in Designer - Maia Documentation","buildId":"dpl_Fyv1NbzwP68g2bEH2Roy98uY1zmE","deploymentId":"dpl_Fyv1NbzwP68g2bEH2Roy98uY1zmE","dynamics":[],"chunks":[]}
