Data Transfer - Maia Documentation

Data Transfer is an orchestration component that enables users to transfer files from a chosen source to a chosen target (destination). This component can use a number of common network protocols to transfer data to a variety of sources and supports transferring files of any type. This component copies, not moves, the target file. Setting up this component requires selecting a source type and a target type. The component’s other properties will change to reflect those choices. The Maia runner streams the file between source and target, holding only a small in-memory buffer at any time. The file is never stored on the Maia runner. Currently supported data sources include:

Azure Blob Storage
Box
Dropbox
FTP
Google Cloud Storage
HTTP
HTTPS
Microsoft Exchange
Microsoft SharePoint
S3
SFTP
Windows Fileshare

Currently supported targets include:

Azure Blob Storage
Google Cloud Storage
S3
SFTP
Windows Fileshare

The Data Transfer component does not currently support transferring files to cloud data platforms such as Snowflake or Databricks. However, you can use the Data Transfer component to transfer files to cloud storage locations (Amazon S3 or Azure Blob Storage), and then use the S3 Load or Azure Blob Storage Load components to load the files into your cloud data platform.

To ensure that instance credentials access is managed correctly at all times, we always advise that customers limit scopes (permissions) where applicable.

If the component requires access to a cloud provider (AWS, Azure, or Google Cloud), it will use credentials as follows:

If using Full SaaS: The component will use the cloud credentials associated with your environment to access resources.
If using Hybrid SaaS: By default the component will inherit the agent’s execution role (service account role). However, if there are cloud credentials associated with your environment, these will overwrite the role.

Including credentials in URLs

Some sources and targets require you to configure a Source URL or Target URL to transfer data from or to the specified location. These URLs can contain your source or target username and password, as shown in the examples below, but this requires you to enter your password in plain text. Instead, we recommend using the Source username, Source password, Target username and Target password properties to provide your credentials, because this allows you to provide your password as a secret definition stored in a secure location. For example, if your source type is HTTP, we don’t recommend including your username and password in the Source URL, e.g. http://user:password@www.example.com/products/list. Instead, use https://www.example.com/products/list as the source URL, then enter your username, and select a secret definition that represents your connection password.

Properties

string

required

A human-readable name for the component.

Connect

drop-down

required

Select the data source that contains the files you want to transfer. Use the tabs below to see the properties to configure for each source type.

The service principal or identity that you use to connect to Azure must have the following permissions:

Storage Account Contributor at storage account level.
Storage Blob Data Contributor at storage account or container level.

No SAS tokens are required to connect to Azure Blob Storage. For more information, read Cloud provider credentials.

file explorer

required

Enter the URL that points to the source file you want to transfer.Either use the file explorer to navigate to the file, or enter the URL into the path field at the top of the dialog. This is particularly useful if you want to use variables in the URL. Special characters used in this field must be URL-safe.The URL should be in the following format:

AZURE://<account>/<container>/<path>

Example with variables:

AZURE://${jv_blobStorageAccount}/${jv_containerName}

drop-down

required

Choose your OAuth connection from the drop-down menu.Click Manage to navigate to the OAuth connections list to review and add OAuth connections. Read OAuth to learn how to create an OAuth connection.

drop-down

required

Select the type of file you want to transfer: File or Paper.

string

required

Enter the file path of the file you want to transfer.

boolean

required

Only visible if File type is File.Select whether to download the file to transfer as a ZIP file. Default is No.

drop-down

required

Only visible if File type is Paper.Select the format you want to download the file to transfer in. Default is HTML.

string

required

Enter the URL, including full path and file name, that points to the source file. Special characters used in this field must be URL-safe.We recommend using the Source username and Source password properties to provide credentials. For more information, see Including credentials in URLs.The source URL should be in the following format:

ftp://[username[:password]@]hostname[:port][path]

Example:

ftp://johndoe@example.com/home/johndoe/documents/report.txt

boolean

required

Yes: URLs are relative to the user’s home directory.
No: (Default) URLs are relative to the server root.

string

Enter your URL connection username for the source. This property is optional—your username will only be used if the data source requests it.

drop-down

Select the secret definition that represents your credentials for your chosen source.Click Manage to navigate to the Secret definitions list to review and add secret definitions. Read Secrets and secret definitions to learn how to create a secret definition.This property is optional—your password will only be used if the data source requests it.

This option supports plain FTP only. It does not support FTPS (FTP Secure). An ftps:// URL will not negotiate TLS—the scheme is ignored, and the connection is attempted as plain FTP. The FTP source only supports plain, unencrypted FTP. If the FTP server requires TLS, the connection will fail.If you need an encrypted file transfer protocol, use SFTP instead, which is fully supported.

file explorer

required

Enter the URL that points to the source file.Either use the file explorer to navigate to the file you want to transfer, or enter the URL into the path field at the top of the dialog. This is particularly useful if you want to use variables in the URL. Special characters used in this field must be URL-safe.The source URL should be in the following format:

GS://<bucket>/<folder-path>

Example with variables:

GS://${jv_gcsAccount}/${jv_folderName}

string

required

http://[username[:password]@]hostname[:port][absolute-path]

Example:

http://user:password@www.example.com/products/list

string

Enter your URL connection username for the source. This property is optional—your username will only be used if the data source requests it.

drop-down

string

required

https://[username[:password]@]hostname[:port][absolute-path]

Example:

https://user:password@www.example.com/products/list

drop-down

required

Select whether to check that the SSL certificate for the host is valid before fetching your file. Default is No.

string

Enter your URL connection username for the source. This property is optional—your username will only be used if the data source requests it.

drop-down

The Microsoft Exchange source accepts the following connection types:

OAuth 2.0 Authorization Code: Uses the Matillion-managed app. A user signs in to authorize access.
OAuth 2.0 Client Credentials: Uses your own Microsoft Entra app registration. App-only, with no user sign-in. The Maia runner requests the token from your tenant.
OAuth 2.0 Authorization Code (Bring Your Own App): Uses your own Microsoft Entra app registration. A user signs in with your app, and the Maia runner refreshes the token directly against your tenant.

For setup steps, read Microsoft Exchange.

drop-down

required

string

The mailbox to retrieve attachments from, for example user@example.com. Requests are addressed to this mailbox (Microsoft Graph /users/\{mailbox\}/...).

This property is required with an OAuth 2.0 Client Credentials connection, because an app-only token has no signed-in user.
With the other connection types, leave this property empty to read the signed-in user’s own mailbox. Enter the address of a shared mailbox to read that mailbox, if the signed-in user has access to it.

drop-down

required

Select the type of Exchange item to retrieve attachments from:

Message: Retrieves attachments from an email message.
Event: Retrieves attachments from a calendar event.

With an OAuth 2.0 Client Credentials connection that only has the Mail.Read application permission, Event isn’t available. Retrieving event attachments also needs the Calendars.Read application permission.

string

required

Enter the ID of the Exchange item to retrieve attachments from:

If Attachment source is Message, enter a valid email message ID.
If Attachment source is Event, enter a valid calendar event ID.

string

required

Enter the ID of the attachment to retrieve from the email message or calendar event identified in Source ID.

Troubleshooting Microsoft Exchange errors

403 Forbidden: With an OAuth 2.0 Client Credentials connection, this usually means either the mailbox is outside your app’s scope in Exchange role-based access control (RBAC) for applications, or your app doesn’t have admin-consented Mail.Read permission.
404 Not Found: This usually means the Mailbox, Source ID, or Attachment ID is wrong.

Network access for Microsoft Exchange

With a connection that uses your own Microsoft Entra app, the Maia runner calls login.microsoftonline.com (token requests) and graph.microsoft.com (data). For more information, read Network access and IP allowlist requirements.

Microsoft removed NTLM authentication support on January 31, 2026 and Basic authentication support on May 1, 2026 for SharePoint Online. If you’re connecting to SharePoint Online via SOAP, you must switch to REST with Azure AD OAuth immediately. SOAP authentication remains supported for SharePoint On-Premise. For more information, see the CData driver changelog.

drop-down

required

Select REST to connect to Microsoft SharePoint using the REST API.

drop-down

required

column editor

Parameter: A JDBC parameter supported by the database driver. The available parameters are explained in the data model. Manual setup isn’t usually required, since sensible defaults are assumed.
Value: A value for the given parameter.

To transfer files from subsites, in the Connection Options property, enter URL as a parameter, with your site URL as the value. This must be the full URL, e.g. https://companyname.sharepoint.com/sitename (for classic/root sites) or https://companyname.sharepoint.com/sites/sitename (for modern site collections).

drop-down

required

Select the type of file you want to transfer: Attachment or Document.

drop-down

required

Only visible if File type is Document.Select the SharePoint library that contains the file you want to transfer. The drop-down lists libraries in your SharePoint instance.For more information, read Introduction to libraries.

string

required

Enter the URL that points to the file you want to transfer. This must be the full DNS name and path for the file, e.g. https://companyname.sharepoint.com/sitename/path/to/file/file.ext or https://companyname.sharepoint.com/sites/sitename/path/to/file/file.ext.

file explorer

required

S3://<bucket>/<folder-path>/<file>

Example with variables:

S3://${jv_bucketName}/${jv_folderName}/report.txt

string

required

sftp://[username[:password]@]hostname[:port][path]

Example:

sftp://johndoe@example.com/home/johndoe/documents/report.txt

boolean

required

Yes: URLs are relative to the user’s home directory.
No: (Default) URLs are relative to the server root.

string

Enter your URL connection username for the source. This property is optional—your username will only be used if the data source requests it.

drop-down

Select the secret definition that represents your SFTP private key.Click Manage to navigate to the Secret definitions list to review and add secret definitions. Read Secrets and secret definitions to learn how to create a secret definition.This property is optional—your SFTP private key will only be used if the data source requests it.

Your private key must be in OpenSSH or PEM (PKCS#1 or PKCS#8) format, and one of the following types: RSA, Ed25519, ECDSA (nistp256–nistp521), or DSA.By default, the key must be encrypted using one of the following algorithms: ssh-ed25519, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521, rsa-sha2-512, or rsa-sha2-256. A DSA key or a legacy ssh-rsa (SHA-1) key isn’t accepted unless you add it to the algorithm list in the Source SFTP config property, for example:

PubkeyAcceptedAlgorithms=ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss

Passphrase-protected private keys aren’t supported. Remove the passphrase before use:

ssh-keygen -p -N "" -f YOUR_PRIVATE_KEY

Paste the complete key, including its header and footer lines, and keep its original line breaks. A key flattened onto a single line causes an “invalid private key” error.In a Hybrid SaaS configuration, you need to manually convert the private key into a format that allows it to be stored in your AWS Secrets Manager. You can do this with the following command:

ssh-keygen -p -f YOUR_PRIVATE_KEY -m pem

text editor

Specify custom SFTP configurations using key/value pairs. Enter each key/value pair on a new line. These configurations override standard SFTP behavior, enabling greater compatibility with different server security policies. These configurations only apply to the currently selected Data Transfer component. If you have multiple Data Transfer components in your pipeline, you must configure each one individually.For a list of all available keys, read the list of properties here.When using the kex key to define the key exchange algorithms to use, the only supported encryption algorithms are ssh-ed25519, ecdsa-sha2-nistp256, ecdsa-sha2-nistp384, ecdsa-sha2-nistp521, rsa-sha2-512, and rsa-sha2-256.To use variables in this field, type the name of the variable prefixed by the dollar symbol and surrounded by { } brackets, as follows: ${variable}. Once you type ${, a drop-down list of autocompleted suggested variables will appear. This list updates as you type; for example, if you type ${date, functions and variables containing date will be listed.

When Source type is Windows Fileshare:

The Data Transfer component uses the SMB2 protocol exclusively. SMB1 is not supported.
The Data Transfer component uses TCP port 445 by default if no port is explicitly specified.
Source credentials (domain, username, and password) are optional. The domain, username, and password must be provided as separate fields—do not use the domain\username format in the username field.

string

required

smb://[[authdomain;]user@]host[:port][/share[/dirpath][/name]][?context]

This source also accepts other methods such as HTTP and HTTPS.Examples:

smb://mydomain;jdoe@fileserver01/documents/projects/mydata

http://11.22.3.4/share5/mydata.txt

string

Enter the domain that the file to transfer is located on.

string

Enter your URL connection username for the source. This property is optional—your username will only be used if the data source requests it.

drop-down

Configure

boolean

required

Select Yes if the source data is a ZIP file that you want to unpack before being transferred. Default is No.

boolean

required

Select Yes to gzip the transferred data when it arrives at the target. Default is No.

string

required

Enter a name for the new file when it is transferred to your target location.

Destination

drop-down

required

Select the target location to transfer your files to. Use the tabs below to see the properties to configure for each target type.

Azure Blob Storage
Google Cloud Storage
S3
SFTP
Windows Fileshare

The service principal or identity that you use to connect to Azure must have the following permissions:

Storage Account Contributor at storage account level.
Storage Blob Data Contributor at storage account or container level.

No SAS tokens are required to connect to Azure Blob Storage. For more information, read Cloud provider credentials.

file explorer

required

Enter the URL that points to the location you want to transfer the file to.Either use the file explorer to navigate to the location for the file, or enter the URL into the path field at the top of the dialog. This is particularly useful if you want to use variables in the URL. Special characters used in this field must be URL-safe.The URL should be in the following format:

AZURE://<account>/<container>/<path>

Example with variables:

AZURE://${jv_blobStorageAccount}/${jv_containerName}

file explorer

required

Enter the URL that points to the location to transfer your file to.Either use the file explorer to navigate to the location to transfer the file to, or enter the URL into the path field at the top of the dialog. This is particularly useful if you want to use variables in the URL. Special characters used in this field must be URL-safe.The target URL should be in the following format:

GS://<bucket>/<folder-path>

Example with variables:

GS://${jv_gcsAccount}/${jv_folderName}

file explorer

required

S3://<bucket>/<folder-path>

Example with variables:

S3://${jv_bucketName}/${jv_folderName}

drop-down

If required, select ACL settings to apply to the transferred file. If you do not select any ACL settings, any ACL settings you currently have applied will be left unchanged. For more information about the available options, read the Amazon S3 canned ACLs list.

drop-down

required

Select how to encrypt the transferred file at the location specified in Target URL:

None: (Default) Do not encrypt the data.
SSE KMS: Encrypt the data according to a key stored on KMS. For more information, read AWS Key Management Service.
SSE S3: Encrypt the data according to a key stored on an S3 bucket. For more information, read Using server-side encryption with Amazon S3-managed encryption keys [SSE-S3].

drop-down

required

Only visible if Encryption is SSE KMS.Select the ID of the KMS encryption key you want to use to encrypt the data.

string

required

Enter the URL that points to the folder to transfer your file to. Special characters used in this field must be URL-safe.We recommend using the Target username and Target password properties to provide credentials. For more information, see Including credentials in URLs.If the folders in your target path don’t exist, the Data Transfer component creates them automatically.The target URL should be in the following format:

sftp://[username[:password]@]hostname[:port][path]

Example:

sftp://johndoe@example.com/home/johndoe/documents/

boolean

required

Yes: URLs are relative to the user’s home directory.
No: (Default) URLs are relative to the server root.

string

Enter your URL connection username for the target. This property is optional—your username will only be used if the target requests it.

drop-down

Select the secret definition that represents your credentials for your chosen target.Click Manage to navigate to the Secret definitions list to review and add secret definitions. Read Secrets and secret definitions to learn how to create a secret definition.This property is optional—your password will only be used if the target requests it.

drop-down

PubkeyAcceptedAlgorithms=ssh-ed25519,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ssh-dss

Passphrase-protected private keys aren’t supported. Remove the passphrase before use:

ssh-keygen -p -N "" -f YOUR_PRIVATE_KEY

ssh-keygen -p -f YOUR_PRIVATE_KEY -m pem

text editor

When Target type is Windows Fileshare:

The Data Transfer component uses the SMB2 protocol exclusively. SMB1 is not supported.
The Data Transfer component uses TCP port 445 by default if no port is explicitly specified.
A target username is required when authenticating; the target domain and target password remain optional. The domain, username, and password must be provided as separate fields—do not use the domain\username format in the username field.

string

required

smb://[[authdomain;]user@]host[:port][/share[/dirpath]][?context]

This target also accepts other methods, such as HTTP and HTTPS. Examples:

smb://mydomain;jdoe@fileserver01/documents/projects/

http://11.22.3.4/share5/

string

Enter the domain that you want the transferred file to be located on.

string

Enter your URL connection username for the target. This property is optional—your username will only be used if the target requests it.

drop-down