2023

​

- New features

• Added Google Cloud Storage as a “direct to storage” option when using Flex connectors.
• Added REST API service support to SharePoint Query. This service requires OAuth authentication.
• Added the following Flex connector for developing data pipelines:
  • SendGrid

​

Streaming - Improvements

• Updated CDC agent version from 2.88.3 to 2.89.4.
• Added option to disable the table name prefix when using Snowflake as a target.
• Fix support for Snowflake when using the QUOTED_IDENTIFIERS_IGNORE_CASE property.
• Additional logging when unable to connect to the Matillion Platform.

​

- New features

• Added support for “connect your own GitHub repo” to . This feature lets you connect your GitHub repository when you create a project in . Follow the documentation to get started:
  • Prerequisites for connecting a GitHub repository to
  • Installing the Matillion app in GitHub
  • Troubleshooting guide for connecting a GitHub repository to .

​

- New features and improvements

• New Update Scalar component. This component allows you to change the value of any pipeline variable or project variable during pipeline execution, to a new value decided at the time the component is configured.
• New Print Variable component. This component prints the value that a pipeline or project variable has at that specific point in the pipeline execution.
• Added the following Flex connector for developing data pipelines:
  • Shopify
• Improved the Git Merge workflow to let users merge changes that have been pushed to a remote branch into your local branch. This improvement empowers users to validate and test changes before pushing merged changes to the remote repository.

​

- New features

• Added Azure as a staging location to all connectors.
• Added the following Flex connector for developing data pipelines:
  • Zendesk Talk

​

- New features

• Added a Recent pipelines menu to UI. will dynamically display your most recent data pipelines, with identifying metadata such as the project platform, Git branch, and pipeline folder name.

​

- New features

• Added the following component for developing data pipelines:
  • Query Result To Scalar

​

- New features

• Added the following Flex connector for developing data pipelines:
  • Square

​

Batch - New features and improvements

• Improved the Set frequency page of the batch pipeline wizard, including adding an advanced mode with support for Quartz cron expressions.

​

Streaming - Improvements

• Fixed issue where data would not load into tables for pipelines with Snowflake as a destination.

​

- New features

• Added the following connectors for developing data pipelines:
• Workday
• Pipedrive

​

- New features

• Added the Workday Custom Reports connector to . This component uses the Workday RaaS (Reporting as a Service) API to retrieve and store data—such as employee, financial, and business-related data—from Workday, to be loaded into a table.

​

Streaming - New features and improvements

• Added Copy Table with Soft Deletes as a transformation type for pipelines with Snowflake as a destination.
• Updated CDC agent version from 2.87.8 to 2.88.2.
• Improved upload speed when writing to a Snowflake staging table.

​

- New features

• Added the following connectors for developing data pipelines:
  • Braze
  • Chargebee
  • Eventbrite
  • Gong
  • PayPal
  • Slack
• Added Azure Storage as a storage destination for the following connectors:
  • ActiveCampaign
  • Amplitude
  • Anaplan
  • Braze
  • Brevo
  • Chargebee
  • CircleCI
  • Concord
  • Confluence
  • Datadog
  • Delighted
  • Eventbrite
  • Freshdesk
  • Gong
  • Klaviyo
  • LaunchDarkly
  • PagerDuty
  • PayPal
  • Pendo
  • Productboard
  • Recurly
  • Slack
  • Smartsheet
  • Snyk
  • TikTok
  • Toggl
  • Twilio

​

- New features

• Added the Anaplan connector.

​

- Improvements

• Updated the Add Components panel with clickable filter buttons under the search bar. Click a filter button to only display components tagged for that filter. You can activate multiple filter buttons.
• Updated the Add component call-to-action (CTA) dialog to include an output connection setting. When adding a component to the canvas, you can now confirm whether that component should be run:
  • On success (green connection line).
  • Unconditionally (gray connection line).
  • On failure (red connection line).
• You can now sample Query components in orchestration pipelines. To do this:
  • Click on a validated Query component in your orchestration pipeline, e.g. Salesforce Query.
  • Click the Sample tab.
  • Click Sample data.
• Added row count when sampling components in a transformation pipeline. To view the total rows:
  • Click on a validated transformation component, e.g, Table Input.
  • Click the Sample tab.
  • Click the refresh button.

​

Custom Connector - New features

• The ability to import pre-existing connectors using the Custom Connector import feature. This feature supports two types of connectors:
  • Custom connectors that have been previously exported.
  • Matillion ETL connectors.

​

- New features

• Added support for Flex connectors to . In , Flex connectors let you connect to a curated set of endpoints to load data. 20 Flex connectors are initially available for :

1. ActiveCampaign
2. Amplitude
3. Brevo
4. CircleCI
5. Concord
6. Confluence
7. Datadog
8. Delighted
9. Freshdesk
10. Klaviyo
11. LaunchDarkly
12. PagerDuty
13. Pendo
14. Productboard
15. Recurly
16. Smartsheet
17. Snyk
18. TikTok
19. Toggl
20. Twilio

​

- New features

• Added the Assert View component, which lets you verify that certain conditions are true of a view, or otherwise stop the query.

​

- New features and improvements

• Updated the Add Components panel to un-nest components. We believe this will make finding the component you’re looking for much easier. Components are listed alphabetically and include a descriptor keyword, e.g. “Connectors” or “Flow”. You can use these descriptors to filter out components that don’t match that keyword. For example, typing “flow” in the search bar will return only the Flow components (And, End Failure, End Success, If, Or, Retry, Restart). The search also includes a number of synonym keywords to help filter components. For example, typing “logic” into the search bar will also return the Flow components.
• Updated the pipeline canvas to include a + call-to-action button. When you select a component on the canvas, the + button will appear to the right. Clicking this button will open the Add component dialog. This dialog works similarly to the Add Component panel.
  • Components are listed alphabetically.
  • Descriptor and synonym keywords help you refine your search.
  • Clicking a component in this dialog will add that component to the canvas in place of the + button and close the dialog. The new component will be connected to the previously selected component.
  • This newly added component will become the selected component, meaning the + button will shift to the right.
• Added the Table Update component, which lets you update a target table with a set of input rows. The rows to update are based on matching keys.

​

- Improvements

• The Git Commit changes and push option has been split into two separate operations: Commit and Push local changes.

​

- Improvements

• The Data Transfer component now supports SFTP as a source type.

​

- New features and improvements

• Oracle is now provided as a driver and a supported database type in the Database Query and RDS Query components.
• Added support for multi-line secret values for Full SaaS, Matillion hosted projects when creating secrets and secret definitions.
• Added an Add schedule call-to-action button to the UI to easily navigate from to the Create a new schedule menu.
• Added the ability to associate newly created cloud credentials to an environment in a single workflow. You can associate credentials from multiple providers, but only one set of credentials for each cloud provider. For example, you can associate both AWS and Azure credentials, but not two different AWS credentials.
• Added a “hard reset” functionality to Git in . Hard reset lets users reset their branch to the last local commit.

​

- New features

• Added the following component:
  • Google Ads Query

​

- New features and improvements

• Registered customers with Account and User Administrator privileges can edit the Account name and Subdomain name for their account.
• Data Loader Batch pipeline runs are now observable from the Pipeline Observability dashboard.
• Pipeline error messages are now displayed at the top of the Pipeline run details page.

​

Streaming - New features and improvements

• Updated CDC agent version from 2.83.5 to 2.87.8.
• Added “Copy Table With Soft Deletes” as a transformation type for pipelines with Snowflake as a destination.
• Fixed an issue where Snowflake role names could not contain non-standard characters.
• Improved connection reuse for Snowflake as a destination.

​

- New features

• Added the following components:
  • Azure Blob Storage Load
  • Azure Blob Storage Unload
  • Dynamics 365 Query
  • Facebook Ads Query
  • Facebook Query
  • Salesforce Marketing Cloud Query

​

- New features

• Added the ability to store cloud provider credentials for authenticating with AWS and Azure.
• The following components are now supported in a Matillion Full SaaS environment (previously were Hybrid SaaS only):
  • Data Transfer
  • Excel Query
  • File Iterator
  • RDS Bulk Output
  • RDS Query
  • S3 Load
  • S3 Unload
  • SNS Message
  • SQS Message
• Added the ability to use an S3 location as the stage platform on all query (load) components in a Matillion Full SaaS environment (previously was a Hybrid SaaS function only).

​

Streaming - New features and improvements

• Updated CDC agent version from 2.83.5 to 2.87.1.
• Added on-demand snapshots for pipelines with Db2 for IBM i as a source.
• Added Change Log as a transformation type for pipelines with Snowflake as a destination.

​

Streaming - Improvements

• Added the in-client Help bar with contextual documentation to the “Create Pipeline” pages.

​

- New features and improvements

• Added the following components:
  • Marketo Query
• Replaced the + button with an Add button that opens a context menu for creating pipelines and folders.

​

Streaming - Improvements

• Updated CDC Agent version from 2.81.12 to 2.83.5.
• Fixed issue where on-demand snapshots could fail to trigger.
• Additional validation of the stage format for the Snowflake destination.
• Added rs_id and ssn metadata fields to the Oracle change records.

​

- New features

• Added the following components:
  • Google BigQuery Query
  • NetSuite Query
• Added a Getting started wizard for users who load into an instance of with no existing pipelines.

​

- New features

• Added the following component:
  • Bing Ads Query

​

- New features

• Added new item Invite your Teammates to the help widget’s Task checklist, to facilitate collaboration.

​

- New features and improvements

• Added the following components:
  • File Iterator
  • Stream Input
• Added the ability to organize pipelines into folders.
• Tooltips added to the UI for type of pipeline and type of component.
• The product tour has been replaced with a checklist to help users with creating a pipeline, adding components, and running a pipeline.

​

Streaming - New features and improvements

• Updated CDC Agent version from 2.81.6 to 2.81.12.
• Added Direct to Snowflake as a target. Read Connect to Snowflake to learn more.

​

- Improvements

• Improved the code editor to provide features such as:
  • Syntax highlighting.
  • IntelliSense.
  • Validation support for SQL, Python, and Bash.
  • Command palette menu via F1.
  • Visual Studio Code shortcut keys, e.g. CMD/CTRL for multi-cursor editing.

​

- New features

• Added the following components:
  • Email Query
  • HubSpot Query
  • LDAP Query
  • Stripe Query

​

Streaming - Improvements

• Updated CDC Agent version from 2.77.1 to 2.81.6.
• Changed “Unavailable Value” placeholder—read PostgreSQL connector to learn more.

​

Streaming - New features

• Enabled on-demand snapshotting functionality for Oracle, SQL Server, and MySQL databases.

​

- New features

• Added the SharePoint Query component. This component’s driver currently supports:
  • Windows SharePoint Services 3.0
  • Microsoft Office SharePoint Server 2007
  • SharePoint Server 2010
  • SharePoint Server 2013
  • SharePoint Server 2016
  • SharePoint Online

​

Streaming - New features and improvements

• Updated CDC Agent version from 2.68.1 to 2.76.1.
• Updated CDC Agent version from 2.76.1 to 2.77.1.
• Added on-demand snapshotting functionality for Oracle, SQL Server, and MySQL databases.

​

- New features

• Added the Data Transfer component. Users can transfer files from a chosen source to a chosen target using common network protocols.
  • Currently supported data sources include: FTP, HTTP, HTTPS, Amazon S3, and Windows Fileshare.
  • Currently supported targets: Amazon S3 and Windows Fileshare.

​

Streaming - New features

• Added on-demand snapshot functionality for PostgreSQL. This feature requires your agent to be version 2.65.0 or higher.

​

Streaming - Improvements

• CDC agent version updated to 2.67.4.
• Updates to the DB2 for IBM i connector.
  1. Pipelines using this connector will now fail on startup if any of the journals for the tracked tables do not have IMAGES(*BOTH) configured.
  2. Pipelines using this connector will now reconnect to the DB2 instance between snapshotting and streaming. This should rectify issues where the connection times out after a long-running snapshot.

​

Batch - New features

• DB2 for IBM i is now available as a Batch loading pipeline.

​

Batch - New features

• Added support for POST API methods to Matillion Custom Connector.
• Added BODY parameter support for Matillion Custom Connector.

​

Streaming - Improvements

• Improved performance related to journal lookup for the Db2 for i connector.

​

Streaming - Improvements

• Improved error handling during the pipeline startup process.
• Db2 for IBM i connector: Fix for the SQL catalog and system name differences.

​

Streaming - Improvements

• Users can now choose a schema before choosing a table when building a CDC pipeline. Read Create a CDC Pipeline to learn more

​

Streaming - Improvements

• Introduced a starting status to pipelines that are starting up.
• Made the request to stop a pipeline non-blocking.
• Db2 for i connector: Fix handling of tables where the table name and system name differ.

​

Streaming - Improvements

• Users can now edit pipelines in-place, without the need to delete and recreate. Read Create and manage a CDC Pipeline to learn more.

​

Batch - New features

• Users can now stop a pipeline. Do this from the Dashboard page or the Pipeline Details page.
• The Stop pipeline button will display the current status of a running pipeline.

​

Batch - New features

• Users can now observe the URL being queried, so they know the endpoint they’re working with. This feature is specific to the Custom Connector pipeline.

​

Streaming - Improvements

• Added an improvement so that users will be alerted via email if their CDC pipeline fails.
• deployment approach has improved. Users can now quick-create an agent within AWS, using the link from the Data Loader UI (it will pre-fill values into the template for you). Read CloudFormation ECS Fargate Advanced Template for more information.
• Fixed an issue that prevented data from being properly erased while creating destinations. Now, when editing one destination after another or adding new destinations, users should be able to establish pipelines with cleared data.

​

Streaming - Improvements

• Users can now manually reset their own key pair without needing to contact Matillion support. Read Platform Keys for more information.

​

Batch - Improvements

• Users can now download error logs for their pipelines as a .txt file.
• Improved validation when creating and editing pipelines by adding per-field validation.
• Users can now set table names when creating a pipeline (as opposed to the table name being assigned on the source name.)

​

Streaming - Improvements

• Column names are now sanitized to ensure conformity with the Avro file format rules.
• Fixed an issue where special characters in table names would cause issues in the pipeline configuration.
• Improvement added to now log an error if your Organization ID or Agent ID values are not in the correct format.