Data Transfer is an orchestration component that enables users to transfer files from a chosen source to a chosen target.
This component can use a number of common network protocols to transfer data to a variety of sources and supports transferring files of any type. This component copies, not moves, the target file. Setting up this component requires selecting a source type and a target type. The component’s other properties will change to reflect those choices.
Currently supported data sources include:
- Azure Blob Storage
- FTP
- SFTP
- HTTP
- HTTPS
- Amazon S3
- Windows Fileshare
- Microsoft SharePoint
- Google Cloud Storage
- Microsoft Exchange
Currently supported targets include:
- Azure Blob Storage
- Amazon S3
- Windows Fileshare
- SFTP
- Google Cloud Storage
Currently, only the following encryption algorithms are supported by the component:
- ssh-ed25519
- ecdsa-sha2-nistp256
- ecdsa-sha2-nistp384
- ecdsa-sha2-nistp521
- rsa-sha2-512
- rsa-sha2-256
- To ensure that instance credentials access is managed correctly at all times, we always advise that customers limit scopes (permissions) where applicable.
- When reading from or writing to Windows Fileshare, the SMB2 protocol will be preferred. SMB1 is still supported.
- If using Matillion Full SaaS: The component will use the cloud credentials associated with your environment to access resources.
- If using Hybrid SaaS: By default the component will inherit the agent’s execution role (service account role). However, if there are cloud credentials associated to your environment, these will overwrite the role.
Properties
A human-readable name for the component.
Select the type of data source. The source type will determine which source properties are required.Currently supported data sources include:
- Azure Blob Storage
- FTP
- Google Cloud Storage
- HTTP
- HTTPS
- S3
- Windows Fileshare
- Microsoft SharePoint
- SFTP
- Microsoft Exchange
Select Yes if the source data is a ZIP file that you wish to unpack before being transferred.
Select the target type for the new file. The target type will determine which target properties are required.Currently supported targets:
- Azure Blob Storage
- Google Cloud Storage
- S3
- Windows Fileshare
- SFTP
Select Yes if you wish to gzip the transferred data when it arrives at the target.
The filename of the new file.
Source properties
Azure Blob Storage
FTP
Google Cloud Storage
HTTP
HTTPS
S3
Windows Fileshare
Microsoft SharePoint
SFTP
Microsoft Exchange
The following permissions are required to connect to Azure Blob Storage:
- Storage Account Contributor at storage account level
- Storage Blob Data Contributor at storage account or container level
The URL that points to the source file that exists on Azure Blob Storage.Clicking this property will open the Blob Location dialog. This displays a list of all existing storage accounts. Select a storage account, then a container, and then a subfolder if required. This constructs a URL with the following format:AZURE://<account>/<container>/<path>
AZURE://${jv_blobStorageAccount}/${jv_containerName}
Set Home Directory as Root
- Yes: URLs are relative to the user’s home directory.
- No: (default) URLs are relative to the server root.
The URL, including full path and file name, that points to the source file. This source URL should follow the below logic where appropriate:ftp://[username[:password]@]hostname[:port][path]
ftp://johndoe@example.com/home/johndoe/documents/report.txt
This is your URL connection username. It is optional and will only be used if the data source requests it.
Choose the secret definition that represents your credentials for this connector.If you have not already saved your credentials for this connector as a secret definition, click Add secret to create a secret definition representing these credentials. Read Secrets and secret definitions for details about creating a secret definition.This is optional and will only be used if the data source requests it. The URL that points to the source file that exists on Google Cloud Storage.Clicking this property will open the Source URL dialog. This displays a list of all existing buckets. Select a bucket, then a folder, and then a subfolder if required. This constructs a URL with the following format:GS://<bucket>/<folder-path>
GS://${jv_gcsAccount}/${jv_folderName}
The URL, including full path and file name, that points to the source file. The source URL should follow the below logic where appropriate:http://[username[:password]@]hostname[:port][absolute-path]
http://user:password@www.example.com/products/list
This is your URL connection username. It is optional and will only be used if the data source requests it.
Choose the secret definition that represents your credentials for this connector.If you have not already saved your credentials for this connector as a secret definition, click Add secret to create a secret definition representing these credentials. Read Secrets and secret definitions for details about creating a secret definition.This is optional and will only be used if the data source requests it. Perform Certificate Validation
Check that the SSL certificate for the host is valid before taking data.
The URL, including full path and file name, that points to the source file. The source URL should follow the below logic where appropriate:https://[username[:password]@]hostname[:port][absolute-path]
https://user:password@www.example.com/products/list
Your URL connection username. It is optional and will only be used if the data source requests it.
Choose the secret definition that represents your credentials for this connector.If you have not already saved your credentials for this connector as a secret definition, click Add secret to create a secret definition representing these credentials. Read Secrets and secret definitions for details about creating a secret definition.This is optional and will only be used if the data source requests it. The URL, including full path and file name, that points to the source file.Clicking this property will open the Source URL dialog. This displays a list of all existing buckets. Select a bucket, then a folder, and then a subfolder if required. This constructs a URL with the following format:S3://<bucket>/<folder-path>
/ after a folder name, a validation of the file path is triggered. The URL, including full path and file name, that points to the source file. This preferably follows the format:smb://[[authdomain;]user@]host[:port][/share[/dirpath][/name]][?context]
smb://mydomain;jdoe@fileserver01/documents/projects
http://11.22.3.4/share5/mydata.txt
The domain that the source file is located on.
Your URL connection username. It is optional and will only be used if the data source requests it.
Choose the secret definition that represents your credentials for this connector.If you have not already saved your credentials for this connector as a secret definition, click Add secret to create a secret definition representing these credentials. Read Secrets and secret definitions for details about creating a secret definition.This is optional and will only be used if the data source requests it. To transfer files from subsites, in the Connection Options property, enter URL as a parameter, with your site URL as the value. This must be the full URL, e.g. https://companyname.sharepoint.com/sitename (for classic/root sites) or https://companyname.sharepoint.com/sites/sitename (for modern site collections).
Choose if you want to connect to the SOAP or REST API. SOAP asks for a username and password. REST asks for an OAuth entry.
(REST only)Choose your OAuth connection from the drop-down menu.Click Manage to navigate to the OAuth connections list to review OAuth connections and to add new connections. Read OAuth to learn how to create an OAuth connection.To set up a new OAuth:
- Click Manage under the drop-down menu.
- Click Add OAuth connection above the OAuth connections list.
- Provide a unique, descriptive OAuth name.
- Select your provider. In this case, SharePoint.
- Set the authentication type to OAuth 2.0 Authorization Code Grant.
- Provide your SharePoint URL. Refer to URL below for more information about the URL format you need to use.
- Click Authorize. A new browser tab will open to log in to
https://login.microsoftonline.com/.
- Provide your username and click Accept.
- The browser tab will close and you will be returned to .
- Return to the Authentication parameter in your Data Transfer component and locate your new OAuth profile in the drop-down menu. If the profile has not propagated, click out of the dialog and then try again.
(SOAP only)Enter the web address that you visit to log in to your SharePoint account. For example, https://companyname.sharepoint.com.
(SOAP only)Enter a valid SharePoint username to use for authentication.
(SOAP only)Choose the secret definition that represents your credentials for this connector.If you have not already saved your credentials for this connector as a secret definition, click Add secret to create a secret definition representing these credentials. Read Secrets and secret definitions for details about creating a secret definition. Select your edition of Microsoft SharePoint. Options are SharePoint Online and SharePoint On-Premise.
- Parameter: A JDBC parameter supported by the database driver. The available parameters are explained in the data model. Manual setup isn’t usually required, since sensible defaults are assumed.
- Value: A value for the given parameter.
To transfer files from subsites, in the Connection Options property, enter URL as a parameter, with your site URL as the value. This must be the full URL, e.g. https://companyname.sharepoint.com/sitename (for classic/root sites) or https://companyname.sharepoint.com/sites/sitename (for modern site collections).
Select whether the file type is a Document or Attachment.
The SharePoint library where the file is stored. The drop-down will be populated with libraries contained within your SharePoint instance.For more information, read Introduction to libraries. The URL that points to the file. The full DNS name and path for the file is required for a successful transfer, e.g. https://companyname.sharepoint.com/sitename/path/to/file/file.ext or https://companyname.sharepoint.com/sites/sitename/path/to/file/file.ext.
Set Home Directory as Root
- Yes: URLs are relative to the user’s home directory.
- No: URLs are relative to the server root.
The URL, including full path and file name, that points to the source file. This source URL should follow the below logic where appropriate:sftp://[username[:password]@]hostname[:port][path]
sftp://johndoe@example.com/home/johndoe/documents/report.txt
Your URL connection username. It is optional and will only be used if the data source requests it.
The corresponding password. It is optional and will only be used if the data source requests it.
Your SFTP private key. It is optional and will only be used if the data source requests it.This must be the complete private key, beginning with “-----BEGIN RSA PRIVATE KEY-----” and conforming to the same structure as an RSA private key.The following private key formats are currently supported:In a Hybrid SaaS configuration, you need to manually convert the private key into a format that allows it to be stored in your AWS Secrets Manager. You can do this with the following command:ssh-keygen -p -f YOUR_PRIVATE_KEY -m pem
Specify custom SFTP configurations using key/value pairs. Enter each key/value pair on a new line in the text editor. These configurations override standard SFTP behavior, enabling greater compatibility with different server security policies. These configurations only apply to this instance of the Data Transfer component only. If you have multiple instances of the component in your pipeline, you must configure each one accordingly.For example, use the kex key to define values that represent the key exchange algorithms to use. For a list of all available keys, read the list of properties here.To use variables in this field, type the name of the variable prefixed by the dollar symbol and surrounded by { } brackets, as follows: ${variable}. Once you type ${, a drop-down list of autocompleted suggested variables will appear. This list updates as you type; for example, if you type ${date, functions and variables containing date will be listed. Choose your OAuth connection from the drop-down menu.Click Manage to navigate to the OAuth connections list to review OAuth connections and to add new connections. Read OAuth to learn how to create an OAuth connection. Select the type of Exchange item to retrieve attachments from.
- Message: Retrieves attachments from an email message.
- Event: Retrieves attachments from a calendar event.
Enter the ID of the selected Exchange item. This must be a valid message ID or event ID, depending on the selected Attachment source.
Enter the ID of the attachment to retrieve from the specified message or event.
Target properties
Azure Blob Storage
Google Cloud Storage
S3
Windows Fileshare
SFTP
The following permissions are required to connect to Azure Blob Storage:
- Storage Account Contributor at storage account level
- Storage Blob Data Contributor at storage account or container level
The filename of the new file.
The URL that points to the target file location on Azure Blob Storage.Clicking this property will open the Blob Location dialog. This displays a list of all existing storage accounts. Select a storage account, then a container, and then a subfolder if required. This constructs a URL with the following format:AZURE://<account>/<container>/<path>
AZURE://${jv_blobStorageAccount}/${jv_containerName}
The filename of the new file.
The URL that points to the target file location on Google Cloud Storage.Clicking this property will open the Target URL dialog. This displays a list of all existing buckets. Select a bucket, then a folder, and then a subfolder if required. This constructs a URL with the following format:GS://<bucket>/<folder-path>
GS://${jv_gcsAccount}/${jv_folderName}
The filename of the new file.
The URL (without file name) that points to where the new file will be created.Clicking this property will open the Target URL dialog. This displays a list of all existing buckets. Select a bucket, then a folder, and then a subfolder if required. This constructs a URL with the following format:S3://<bucket>/<folder-path>
Access Control List Options
Choose from ACL settings that Amazon provide. Leaving it empty doesn’t change the current settings. A full list can be found here. Decide how the files are encrypted inside the S3 bucket. This property is available when using an existing Amazon S3 location for staging. The ID of the KMS encryption key you have chosen to use in the Encryption property.
The filename of the new file.
The URL (without file name) that points to where the new file will be created. This should follow the below logic where appropriate:smb://[[authdomain;]user@]host[:port][/share[/dirpath]][?context]
smb://mydomain;jdoe@fileserver01/documents/projectfiles/
The domain that the newly created file is to be located on.
Your URL connection username. It is optional and will only be used if the target system requests it.
The corresponding password. It is optional and will only be used if the target system requests it.
Set Home Directory as Root
- Yes: URLs are relative to the user’s home directory.
- No: URLs are relative to the server root.
The URL (without file name) that points to where the new file will be created. This should follow the below logic where appropriate:sftp://[username[:password]@]hostname[:port][path]
sftp://johndoe@example.com/home/johndoe/documents/
Your SFTP connection username. This is optional and will only be used if the data source requests it.
Your SFTP password. This is optional and will only be used if the target system requests it.Select the secret definition storing the password for your SFTP username. The password should be saved as a secret definition before using this component. Your SFTP private key. This is optional and will only be used if the target system requests it.Select the secret definition storing the private key for your SFTP account. The password should be saved as a secret definition before using this component.This must be the complete private key, beginning with “-----BEGIN RSA PRIVATE KEY-----” and conforming to the same structure as an RSA private key.The following private key formats are currently supported:In a Hybrid SaaS configuration, you need to manually convert the private key into a format that allows it to be stored in your AWS Secrets Manager. You can do this with the following command:ssh-keygen -p -f YOUR_PRIVATE_KEY -m pem
Specify custom SFTP configurations using key/value pairs. Enter each key/value pair on a new line in the text editor. These configurations override standard SFTP behavior, enabling greater compatibility with different server security policies. These configurations only apply to this instance of the Data Transfer component only. If you have multiple instances of the component in your pipeline, you must configure each one accordingly.For example, use the kex key to define values that represent the key exchange algorithms to use. For a list of all available keys, read the list of properties here.To use variables in this field, type the name of the variable prefixed by the dollar symbol and surrounded by { } brackets, as follows: ${variable}. Once you type ${, a drop-down list of autocompleted suggested variables will appear. This list updates as you type; for example, if you type ${date, functions and variables containing date will be listed. 
