> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maia.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# 2024

export const m_runner = "Maia runner";

export const cdc_runner = "CDC agent";

export const s_runner = "Streaming runner";

export const designer = "Designer";

export const maia = "Maia";

Here you'll find the 2024 changelog for {maia}. Just want to read about new features? Read our [New Features](https://roadmap.matillion.com/changelog) blog.

For an explanation of minimum agent version, read [Agent version tracks](/docs/guides/runner-overview#agent-version-tracks).

<Update label="December 12, 2024" description={designer + ", " + m_runner + "s"}>
  ## Connectors - New features

  * Added the `Retrieve Run` endpoint to the [dbt Cloud](/docs/components/dbt-cloud) Flex connector.

  ## Maia runners - Improvements

  * The [{m_runner} template for Azure](/docs/guides/azure-arm-runner-install) now deploys with an environment type of "Workload profile - Dedicated" (was previously "Consumption Only"). This allows the {m_runner} to better facilitate complex networking, while keeping the ability for the {m_runner} to auto-update.
</Update>

<Update label="December 9, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated the [{cdc_runner} version](/docs/streaming/agent-versions) from `2.98.7` to `2.101.2`.
  * For customers who may have experienced an out-of-memory issue with the {cdc_runner}, we have added a new advanced property `matillion.compact-history` that can be set to `true` in any affected pipeline definition. This will reduce the memory footprint of the {cdc_runner} by compacting the schema history of the pipeline.
  * For pipelines that stream data directly to Snowflake, the {cdc_runner} will create a new column `MTLN_CDC_LOAD_TIMESTAMP` in each target table to store the load timestamp of each row.
  * Library updates and security fixes.
</Update>

<Update label="December 6, 2024" description="API">
  ## API - New endpoints

  The following endpoints have been added to the [{maia} REST API](/docs/api-reference/maia-api-overview):

  **Credit consumption**

  | Method  | Endpoint                    | Description                                                        |
  | ------- | --------------------------- | ------------------------------------------------------------------ |
  | **GET** | `/v1/consumption`           | Get a breakdown of your credit consumption for flat-rated products |
  | **GET** | `/v1/consumption/etl-users` | Get a breakdown of your credit consumption for ETL users           |
</Update>

<Update label="December 5, 2024" description={designer}>
  ## Designer - New features

  * Added the new **Skip on run** action to the component action bar. This allows you to temporarily disable a component in a pipeline, without removing it. For more information, read [Skip components](/docs/guides/components-overview#skip-components).

  New transformation component:

  * [Cortex Parse Document](/docs/components/cortex-parse-document) is a transformation component that uses [Snowflake Cortex](https://www.snowflake.com/en/data-cloud/cortex/) to extract content from a document on an internal or external stage in the form of an object that contains JSON-encoded objects as strings.
</Update>

<Update label="December 4, 2024" description={designer}>
  ## Designer - New features

  * Added the option to copy and paste pipelines and pipeline folders in {designer}. For more information, read [Managing pipelines](/docs/guides/pipelines#managing-pipelines) and [Copying pipeline folders](/docs/guides/pipelines#copying-pipeline-folders).
</Update>

<Update label="December 3, 2024" description={designer + ", Streaming"}>
  ## Designer - New features

  * Added the option to rename pipeline folders in {designer}. For more information, read [Pipeline folders](/docs/guides/pipelines#pipeline-folders).
  * Added a new **Logging** tab to the detailed view that appears when you click a task in the **Task history** tab. This shows a detailed log of each step in a task, as described in [Logging](/docs/guides/designer-ui-basics#logging).

  ## Streaming - Improvements

  * Added a graph of number of rows changed over time to the [Manage streaming pipelines dashboard](/docs/streaming/streaming-pipelines/#manage-streaming-pipelines).
</Update>

<Update label="November 19, 2024" description={designer}>
  ## Designer - New features

  New connectors:

  * Added a [Kafka](/docs/components/kafka) connector. The Kafka connector is an orchestration component that lets you load data from Kafka topics into either schemaless JSON or JSON schema objects. New to Kafka? Read Apache Kafka's [Introduction](https://kafka.apache.org/intro).
  * Added the following Flex connectors for developing data pipelines:
    * [Box](/docs/components/box)
    * [Fanatics](/docs/components/fanatics)
    * [Snapchat](/docs/components/snapchat)

  Git actions in {designer}:

  * Added a new [Compare changes](/docs/guides/git-compare) feature that allows you to compare your local changes to the previous commit.
  * Added commit indicators that show how many commits your local branch is ahead of or behind the remote branch when committing, pushing, or pulling changes.

  API:

  * Added a [GET pipeline executions](/api-reference/pipeline-execution/pipeline-executions) endpoint, which returns a list of summarized pipeline executions.
</Update>

<Update label="November 15, 2024" description="API">
  ## API - New endpoints

  The following endpoint has been added to the [{maia} REST API](/docs/api-reference/maia-api-overview):

  **Pipeline execution**

  | Method  | Endpoint                  | Description                                      |
  | ------- | ------------------------- | ------------------------------------------------ |
  | **GET** | `/v1/pipeline-executions` | Returns a list of summarized pipeline executions |
</Update>

<Update label="November 13, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated the [{cdc_runner} version](/docs/streaming/agent-versions) from `2.98.4` to `2.98.7`.
  * Enhanced the resilience of connecting the {cdc_runner} to {maia}.
  * Library updates and security fixes.
</Update>

<Update label="November 12, 2024" description={designer}>
  ## Designer - New features and improvements

  New connectors:

  * Added the following Flex connectors for developing data pipelines:
    * [Attio](/docs/components/attio)
    * [Buildkite](/docs/components/buildkite)

  Component improvements:

  * The [Python Pushdown](/docs/components/python-pushdown) component now includes a `Warehouse` parameter, allowing you to specify which Snowflake warehouse to run your Python scripts on.
  * The [Detect Changes](/docs/components/detect-changes) component is now available on Databricks projects.
</Update>

<Update label="November 7, 2024" description={designer}>
  ## Designer - New features and improvements

  * [Maia (formerly Copilot)](/docs/guides/maia-ai-agents-overview) is now available for Databricks and Amazon Redshift projects (previously only available for Snowflake projects). In {maia}, Maia introduces a new era of AI-driven data productivity tools, letting you prompt generative AI to create data pipelines, improving your efficiency in managing and processing data.
  * Updated the [S3 Unload](/docs/components/s3-unload) component for Amazon Redshift projects. Previously, only CSV file types were supported. Now, the component also supports delimited, fixed with, and Parquet file types.
</Update>

<Update label="November 6, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated the [{cdc_runner} version](/docs/streaming/agent-versions) from `2.98.2` to `2.98.4`.
  * Library updates and security fixes.
</Update>

<Update label="November 5, 2024" description={maia}>
  ## Maia - New features

  * {maia} is now available in Snowflake Partner Connect, allowing a quick and simple way to create an automatically configured {maia} account directly from a Snowflake instance.
</Update>

<Update label="October 31, 2024" description={designer}>
  ## Designer - New features

  Connect your own Git repo:

  * You can now connect your own [Bitbucket](/docs/guides/matillion-bitbucket-app-overview) repository to {maia}. This feature lets you connect your Bitbucket repository when you create a [project](/docs/guides/projects) in {maia}. Follow the documentation to get started:
    * [Prerequisites for connecting a Bitbucket repository to {maia}](/docs/guides/connect-bitbucket-repository-prerequisites)
    * [Troubleshooting guide for connecting a Bitbucket repository to {maia}](/docs/guides/troubleshooting-bitbucket-repository-connection)

  In case you missed our announcements in previous changelogs, we also support GitHub, GitLab, and Azure DevOps. Read [Connect your own Git repo](/docs/guides/installing-git-provider-overview) for an overview.

  {m_runner}s:

  * You can now [refresh a {m_runner}'s credentials](/docs/guides/refresh-credentials), which generates a new **client\_secret** to authenticate with the {m_runner} application in your infrastructure (Hybrid SaaS deployments only).
</Update>

<Update label="October 29, 2024" description={designer}>
  ## Designer - New features

  Components:

  * [JDBC Table Metadata to Grid](/docs/components/jdbc-table-metadata-to-grid) is now available for Amazon Redshift projects.
  * [Truncate Table](/docs/components/truncate-table) is now available for Databricks projects.
</Update>

<Update label="October 22, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated the [{cdc_runner} version](/docs/streaming/agent-versions) from `2.97.14` to `2.98.2`.
  * Change behavior of the Snowflake destination - if the Replication Type is `Copy Table` or `Copy Table with Soft Deletes` and the source table does not have a primary key then the Replication Type will be set to `Change Log` for that table.
  * Library updates and security fixes.
</Update>

<Update label="October 15, 2024" description={designer}>
  ## Designer - Improvements

  Component updates:

  * Updated [Python Pushdown](/docs/components/python-pushdown). Added a `Snowflake Secret` parameter, which is activated when an external access integration is selected. You can select Snowflake secrets and assign them to Snowflake variables, which can then be used in the Python script.
</Update>

<Update label="October 9, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated the [{cdc_runner} version](/docs/streaming/agent-versions) from `2.97.12` to `2.97.14`.
  * Fixed Azure Blob Storage authentication flow.
</Update>

<Update label="October 8, 2024" description={designer}>
  ## Designer - New features

  New orchestration components:

  * [Azure Document Intelligence](/docs/components/azure-document-intelligence). This component uses the Azure AI Document Intelligence API to automate the extraction of text, handwriting, layout elements, and other key data from forms and documents. The output format can be either Markdown or text.
  * [Azure Speech Transcribe](/docs/components/azure-speech-transcribe). This component uses the Azure Speech service to transcribe speech to text from an audio file in your specified Azure Blob location and Speech resource endpoint. Speaker diarization clarifies which speaker spoke and when, increasing the accuracy of your transcriptions.

  New transformation components:

  * [Databricks Vector Search](/docs/components/databricks-vector-search). This component lets you perform a search on an input table to find content that best answers specific questions, using vector embeddings to identify suitable answers within data located in your Databricks account. Only available for Databricks projects.
</Update>

<Update label="October 3, 2024" description={designer}>
  ## Designer - New features

  New orchestration components:

  * [Add Partition](/docs/components/add-partition). This orchestration component lets users define the S3 directory structure for partitioned external table data. Amazon Redshift projects only.
  * [Create External Table](/docs/components/create-external-table). This orchestration component creates a table to reference data stored in an Amazon S3 bucket. The table references external data, meaning the table itself does not hold the data. Amazon Redshift projects only.
</Update>

<Update label="October 2, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated the [{cdc_runner} version](/docs/streaming/agent-versions) from `2.95.33` to `2.97.12`.
  * Improved the process of {cdc_runner} shutdowns so that on clean shutdown the {cdc_runner} status shows as **Stopped** rather than **Unknown**.
  * Library updates and security fixes.
</Update>

<Update label="September 26, 2024" description={designer}>
  ## Designer - Improvements

  Projects:

  * Snowflake projects now allow the choice of key-pair authentication when configuring [environments](/docs/guides/environments#specify-credentials), so a Snowflake private key can be used.
  * Databricks projects now require a Databricks personal access token for authentication when configuring [environments](/docs/guides/environments#specify-credentials-1). Username/password has been deprecated by Databricks and can no longer be used.

  Git version control:

  * Additional options are available for [Git reset](/docs/guides/git-hard-reset), allowing a reset to your selected working state.

  New orchestration components:

  * [Bash Pushdown](/docs/components/bash-pushdown). This component enables users running Full SaaS configurations (Matillion-hosted {m_runner}s) to use SSH connections to run their Bash scripts in their own instances.
</Update>

<Update label="September 24, 2024" description={designer}>
  ## Designer - New features

  New orchestration components:

  * [Snowflake Vector Upsert](/docs/components/snowflake-vector-upsert). This component converts data into vector embeddings in a Snowflake table.

  Component updates:

  * You can now cross-query two databases with the [Table Input](/docs/components/table-input) and [Multi Table Input](/docs/components/multi-table-input) transformation components for Amazon Redshift projects.

  {designer} UI:

  * You can now reorder your pipeline tabs in {designer} with click-and-drag.
</Update>

<Update label="September 12, 2024" description={designer}>
  ## Designer - New features

  New transformation components:

  * [Vector Search](/docs/components/vector-search). This component uses Snowflake vectors to perform a retrieval augmented generation (RAG) search on an input table. This component is available in public preview for Snowflake projects.

  Component updates:

  * Added SFTP as a data input type in the [File Iterator](/docs/components/file-iterator) component.
  * [Table Iterator](/docs/components/table-iterator) is now available on Amazon Redshift projects.

  {designer}:

  * The [Git Commit](/docs/guides/git-commit) dialog now lists all uncommitted changes.
  * Added a command palette to {designer}. In its first phase, the following command is supported:
    * Use `cmd + k` or `ctrl + k` to activate a file explorer dialog. Alternatively, click the search icon in the upper-right corner of the UI.
</Update>

<Update label="September 10, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated the [{cdc_runner} version](/docs/streaming/agent-versions) from `2.95.27` to `2.95.33`.
  * Library updates and security fixes.
</Update>

<Update label="September 5, 2024" description={designer}>
  ## Designer - New features

  New orchestration components:

  * [Amazon Textract](/docs/components/amazon-textract-input). This component uses the Textract API to extract text, handwriting, layout elements, and data from scanned documents. You can optionally choose to include footers and page numbers in the extraction process.
  * [Amazon Transcribe](/docs/components/amazon-transcribe-input). This component uses the Transcribe API to extract data from media and audio files and then converts that data into transcriptions.

  Component updates:

  * [Chunk Text](/docs/components/chunk-text) is now out of public preview and generally available for all Snowflake projects.
</Update>

<Update label="September 3, 2024" description={designer}>
  ## Designer - New features

  New components:

  * [Cortex Embed](/docs/components/cortex-embed). This transformation component lets you convert English-language text into a Cortex vector embedding.
  * [Remove From Grid](/docs/components/remove-from-grid). This orchestration component takes in grid variable data and remove data based on the user's definition.

  New connector:

  * [NetSuite SuiteAnalytics](/docs/components/netsuite-suiteanalytics). This new connector is an orchestration component that uses NetSuite SuiteAnalytics Connect to retrieve data and load it into a table.
</Update>

<Update label="August 29, 2024" description={designer}>
  ## Designer - New features

  * You can now connect your own [GitLab](/docs/guides/matillion-gitlab-app-overview) repository to {maia}. This feature lets you connect your GitLab repository when you create a [project](/docs/guides/projects) in {maia}. Follow the documentation to get started:
    * [Prerequisites for connecting a GitLab repository to {maia}](/docs/guides/connect-gitlab-repository-prerequisites)
    * [Troubleshooting guide for connecting a GitLab repository to {maia}](/docs/guides/troubleshooting-gitlab-repository-connections)
  * Updated [Schedules](/docs/guides/schedules) so that you can now specify a time and date to enable pipeline schedules from.
  * Updated the "Connect your own GitHub" workflow so that you can now view the list of authorized accounts or organizations after a successful authorization with GitHub.
</Update>

<Update label="August 27, 2024" description="Streaming">
  ## Streaming - New features

  * Added the following destinations to streaming pipelines:
    * [Amazon S3](/docs/streaming/amazon-s3-streaming-destination/)
    * [Azure Blob Storage](/docs/streaming/azure-blob-storage-streaming-destination/)
</Update>

<Update label="August 20, 2024" description={designer}>
  ## Designer - New features

  * Shared pipelines are now available in {maia}. Any pipeline can be saved as a [shared pipeline](/docs/guides/shared-pipelines), helping you to maintain consistency of core functionality over multiple different projects across the organization.
  * Added the [Run Shared Pipeline](/docs/components/run-shared-pipeline) component, which allows you to call a shared pipeline from within another pipeline.
  * Added improvements to environments. When you create an environment, you can now set a [default access level](/docs/administration/environment-roles) for that environment. This access level can also be set on a per-user basis.
    * As part of this change, all users in existing environments have been set to have Admin access by default, to avoid any disruption. Project admins can change a user's environment access.
</Update>

<Update label="August 16, 2024" description={designer}>
  ## Designer - New features

  * Added a [Document AI Predict](/docs/components/document-ai-predict) component, which invokes the [Snowflake PREDICT](https://docs.snowflake.com/en/sql-reference/classes/document-intelligence/methods/predict) function and allows users to call their [Document AI](https://docs.snowflake.com/en/user-guide/snowflake-cortex/document-ai/overview) models from the context of a {maia} pipeline. This component is available in public preview.
</Update>

<Update label="August 14, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated the [{cdc_runner} version](/docs/streaming/agent-versions) from `2.95.22` to `2.95.27`.
  * CDC will now retain snapshot metrics once a pipeline snapshot completes and streaming begins. Note: these will be lost after a {cdc_runner} restart.
  * Improved resilience of connection between {s_runner}s and the platform.
  * Library updates and security fixes.
</Update>

<Update label="August 8, 2024" description={"Streaming, " + designer}>
  ## Streaming - New features

  * [Streaming pipelines](/docs/streaming/streaming-pipelines/) are now available in {maia}. Streaming pipelines are a complete end-to-end solution for near-real-time data ingestion, allowing you to capture and synchronize data from source databases and write to a target data warehouse.
    * Supported data sources are:
      * [Db2 for IBM i](/docs/streaming/db2-for-ibm-i-streaming-source/)
      * [Microsoft SQL Server](/docs/streaming/microsoft-sql-server-streaming-source/)
      * [MySQL](/docs/streaming/mysql-streaming-source/)
      * [Oracle](/docs/streaming/oracle-streaming-source/)
      * [PostgreSQL](/docs/streaming/postgresql-streaming-source/)
    * Supported data targets are:
      * [Snowflake](/docs/streaming/snowflake-streaming-destination/)
    * Streaming pipelines are available on our Hybrid SaaS offering, meaning you'll need to create a {s_runner} in your AWS or Azure infrastructure. Read [Create a {s_runner} in your infrastructure](/docs/streaming/create-streaming-agent) to set up your {s_runner}.

  ## Designer - New features

  * Options to run a single component or run a pipeline starting from a specific component have been added to the [pipeline canvas](/docs/guides/designer-ui-basics#component-context-menu).
  * Options to control the [behavior](/docs/guides/variables#variable-behavior-copied-or-shared) of variables in a branching pipeline have been added.
  * The [Push local changes](/docs/guides/git-push) dialog now shows a list of commits to be pushed.
</Update>

<Update label="August 6, 2024" description={designer}>
  ## Designer - New features

  * Users can now terminate running pipelines from the [Pipeline run history](/docs/guides/pipeline-run-history).
</Update>

<Update label="July 30, 2024" description={"API, Streaming"}>
  ## API - New features

  * The [Pipeline execution API](/api-reference/pipeline-execution/execute-published-pipeline) now supports overriding scalar or grid variables when making a call. This enhancement allows for greater flexibility and customization during pipeline execution.

  ## Streaming - Improvements

  * Updated [{cdc_runner} version](/docs/streaming/agent-versions) from `2.95.21` to `2.95.22`.
</Update>

<Update label="July 25, 2024" description={designer}>
  ## Designer - New features

  * Added the [Cloud Pub/Sub](/docs/components/pubsub) component, which allows messages to be published to a topic on Google Cloud. Other applications can subscribe to these topics to receive such messages.
</Update>

<Update label="July 23, 2024" description={designer + ", Streaming"}>
  ## Designer - New features

  * Google Cloud Storage is now available for Snowflake projects in the following areas:
    * Added the [Google Cloud Storage Load](/docs/components/google-cloud-storage-load) component, which lets you load data stored on the Google Cloud Storage service into an existing Snowflake table.
    * Added the [Google Cloud Storage Unload](/docs/components/google-cloud-storage-unload) component, which writes data from a Snowflake table or view into a specified Google Cloud Storage (GCS) bucket in your specified file format (CSV, JSON, or Parquet).
    * Added Google Cloud Storage as a staging platform to all "Query" components (Facebook Query, Salesforce Query, Shopify Query, and so on).
  * Added Google Cloud Storage as a source and destination to the [Data Transfer](/docs/components/data-transfer) component.
  * Added Google Cloud Storage as a source and destination to the [Excel Query](/docs/components/excel-query) component.
  * Added Google Cloud Storage as a source and destination to the [File Iterator](/docs/components/file-iterator) component.

  <Note>
    To interact with Google Cloud services, you'll need to set up [Google Cloud credentials](/docs/guides/cloud-credentials) and [associate your credentials with your environment](/docs/guides/environments#associate-cloud-provider-credentials-with-an-environment).
  </Note>

  ## Streaming - Improvements

  * Updated [{cdc_runner} version](/docs/streaming/agent-versions) from `2.95.1` to `2.95.21`.
  * Fixed an issue where the thread cap for table writers could be reached when capturing changes on a large number of tables at once and cause the {cdc_runner} to stop streaming.
  * The Oracle connector will now gracefully handle skipping transactions when the retention.ms property is configured.
  * Library updates and security fixes.
</Update>

<Update label="July 18, 2024" description={designer}>
  ## Designer - New features and improvements

  * Updated the font across {maia} to a more modern, readable font.
  * Added HTML as a supported data format to the [Chunk Text](/docs/components/chunk-text) component.
  * Added the ability to use multiple Azure Key Vault vaults to store secrets. When you [add a new secret definition](/docs/guides/secrets-and-secret-definitions), you can choose which of your key vaults the secret is stored in.
</Update>

<Update label="July 16, 2024" description={designer}>
  ## Designer - New features

  * Added a [JDBC](/docs/components/jdbc) orchestration component for the Hybrid SaaS offering. The JDBC component allows you to run queries on a JDBC-compatible data source that you have a driver for. You can upload your drivers to {maia}, and so connect to any sources that {maia} doesn't provide drivers for as standard.
  * Added a [Chunk Text](/docs/components/chunk-text) orchestration component, which performs pushdown text chunking using a Python user-defined function (UDF) in Snowflake via the computational power of your Snowflake warehouse. You can choose text or Markdown as your data format.
  * Added the following Flex connectors for developing data pipelines:
    * [{maia}](/docs/components/maia-flex-connector)
    * [SeatGeek](/docs/components/seatgeek)
  * Added a [Refresh Table](/docs/components/refresh-table) orchestration component, which invalidates cached entries of a specified table or view, including both data and metadata. Databricks projects only.
</Update>

<Update label="July 4, 2024" description={designer}>
  ## Designer - New features

  * Added a new [Postgres Vector Upsert](/docs/components/postgres-vector-upsert) component, which lets you convert text data stored in your cloud data warehouse into embeddings and then store these embeddings as vectors in your Postgres vector database.
  * Added Postgres as a vector database option when retrieval augmented generation (RAG) is enabled on the following components:
    * [OpenAI Prompt](/docs/components/openai-prompt)
    * [Azure OpenAI Prompt](/docs/components/azure-openai-prompt)
    * [Amazon Bedrock Prompt](/docs/components/amazon-bedrock-prompt)
    * [Snowpark Container prompt](/docs/components/snowpark-container-prompt)
  * You can now use [Databricks volumes](https://docs.databricks.com/en/sql/language-manual/sql-ref-volumes.html) as your staging platform when using our Query components (Facebook Query, Salesforce Query, and so on).
</Update>

<Update label="July 2, 2024" description={designer}>
  ## Designer - New features and improvements

  * Added support for [AWS Guardrails](https://aws.amazon.com/bedrock/guardrails/) to the [Amazon Bedrock Prompt](/docs/components/amazon-bedrock-prompt) component.
  * Removed `Pinecone Environment` and `Pinecone Project ID` parameters from AI components where users can interact with Pinecone as these parameters are no longer required.
  * The following new components that support Databricks LLM operations are available for Databricks projects in **public preview**:
    * [Analyze sentiment](/docs/components/databricks-ai-analyze-sentiment)
    * [Classify](/docs/components/databricks-ai-classify)
    * [Extract](/docs/components/databricks-ai-extract)
    * [Fix grammar](/docs/components/databricks-ai-fix-grammar)
    * [Mask](/docs/components/databricks-ai-mask)
    * [Query](/docs/components/databricks-ai-query)
    * [Similarity](/docs/components/databricks-ai-similarity)
    * [Summarize](/docs/components/databricks-ai-summarize)
    * [Translate](/docs/components/databricks-ai-translate)
</Update>

<Update label="June 25, 2024" description={designer}>
  ## Designer - New features

  * Connecting your own Azure DevOps repo to {maia} projects is now generally available. This feature lets you connect your Azure DevOps repository when you create a [project](/docs/guides/projects) in {maia}. Follow the documentation to get started:
    * [Overview of the Matillion Azure app](/docs/guides/matillion-azure-devops-app-overview)
    * [Prerequisites for connecting an Azure DevOps repository to {maia}](/docs/guides/connect-azure-devops-repository-prerequisites)
    * [Installing the Matillion Azure app](/docs/guides/installing-matillion-app-azure-devops)
    * [Troubleshooting an Azure DevOps repository connection](/docs/guides/troubleshooting-azure-devops-repository-connection)
  * Added the following Flex connectors for developing data pipelines:
    * [ClickUp](/docs/components/clickup)
    * [Rootly](/docs/components/rootly)
  * Added an [SAP ODP](/docs/components/sap-odp) orchestration component, for connecting directly through SAP ODP to access available data sources in SAPI and ABAP CDS views.
  * Added the ability to [upload external drivers](/docs/guides/uploading-external-drivers) to allow you to enable third-party connectors that don't have a driver supplied by default in a [Hybrid SaaS](/docs/guides/runner-overview#hybrid-saas) configuration, such as SAP ODP.
  * Added support for the [OAuth 2.0 client credentials authentication type in {designer} when using custom connectors](/docs/guides/oauth-client-credentials).
  * Updated data sampling for orchestration components to include a drop-down menu where you can set the number of rows sampled, for example 1, 10, 250, 1000. The default is 25 rows.
</Update>

<Update label="June 20, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated the [{cdc_runner} version](/docs/streaming/agent-versions) from `2.93.8` to `2.95.1`.
  * Added an option to map date and time source columns to the corresponding Snowflake data types when using Snowflake as a target.
  * Changed the `MTLN_CDC_LAST_COMMIT_TIMESTAMP` column when using Snowflake as a target with the Change Log transformation type.
  * Library updates and security fixes.
</Update>

<Update label="June 18, 2024" description={designer}>
  ## Designer - New features

  * Added a component action bar that will be displayed when you click on a component in the canvas. The action bar currently supports copying the component and deleting the component.
  * Updated data sampling for transformation components to include a drop-down menu where you can set the number of rows sampled, for example 1, 10, 250, 1000. The default is 25 rows.
  * Added an [Append To Grid](/docs/components/append-to-grid) component, which lets you append or prepend data to a pre-existing [grid variable](/docs/guides/grid-variables) in {maia}. Running this component can add rows to existing columns, but doesn't add columns to the grid.
  * Added the following Flex connectors for developing data pipelines:
    * [Float](/docs/components/flex-float)
    * [Ortto](/docs/components/ortto)
</Update>

<Update label="June 10, 2024" description={designer}>
  ## Designer - New features

  * Project administrators can now [delete projects](/docs/guides/projects#delete-a-project).
</Update>

<Update label="June 6, 2024" description={designer}>
  ## Designer - New features

  * Added support for [GPT-4o](https://platform.openai.com/docs/models/gpt-4o) to the [OpenAI Prompt](/docs/components/openai-prompt) component. When this model is selected, the Image input type is available. Images should be in either Base64 format or a direct, public URL of the image. If you're feeding images in Base64 format, you will need to add `data:image/{format_of_your_image};base64,` before the encoded image.
  * Added a [Volume to Delta Table](/docs/components/volume-to-delta-table) orchestration component for Databricks, which lets users transfer data from a pre-existing [volume](https://docs.databricks.com/en/connect/unity-catalog/volumes.html) in Databricks in to a Delta Lake table without replacing or deleting any existing data.
  * Added a [Convert String To Struct](/docs/components/convert-string-to-struct) transformation component, which takes a string input and converts it to a struct data type.
  * Renamed the Extract Nested Data component to [Extract Structured Data](/docs/components/extract-structured-data) for Databricks. This transformation component unpacks arrays of structured data into columns and rows of data in a table. Any existing pipelines will still display an Extract Nested Data component.
  * The [Query Result To Grid](/docs/components/query-result-to-grid) component is now available for Databricks projects.
  * Added the following Flex connectors for developing data pipelines:
    * [dbt Cloud](/docs/components/dbt-cloud)
    * [Pingdom](/docs/components/pingdom)
  * A user's account ID is now visible in the user menu (top right of {maia} UI), to help users easily identify their account.
</Update>

<Update label="May 31, 2024" description="Custom Connector">
  ## Custom Connector - New features

  * Added a [Script paging](/docs/guides/scripted-paging) option for connector pagination. Paging scripts give you complete control over response pagination in Custom Connector.
</Update>

<Update label="May 30, 2024" description={designer}>
  ## Designer - New features

  * Maia (formerly Copilot) for {maia} is now available in **public preview** for transformation pipelines in Snowflake projects. Maia lets you prompt generative AI to create data pipelines, improving your efficiency in managing and processing data. Read our docs to get started:
    * [Maia overview](/docs/guides/maia-ai-agents-overview)
  * The [Query Result to Scalar](/docs/components/query-result-to-scalar) component is now available for Databricks projects. This orchestration component allows you to query a table and return a scalar value. This value can then be mapped to a project or pipeline variable for use in other pipeline components.
</Update>

<Update label="May 23, 2024" description={designer}>
  ## Designer - New features

  * Added Amazon Bedrock as an embedding provider and added [Titan Embeddings G1 - Text](https://docs.aws.amazon.com/bedrock/latest/userguide/model-parameters-titan-embed-text.html) as an embedding model for the following AI components:
    * [OpenAI Prompt](/docs/components/openai-prompt)
    * [AzureOpenAI Prompt](/docs/components/azure-openai-prompt)
    * [Amazon Bedrock Prompt](/docs/components/amazon-bedrock-prompt)
    * [Snowpark Container Prompt](/docs/components/snowpark-container-prompt) (public preview)
    * [Pinecone Vector Query](/docs/components/pinecone-vector-query)
    * [Pinecone Vector Upsert](/docs/components/pinecone-vector-upsert)
  * Added the ability to [configure {m_runner}s to work with a proxy server](/docs/guides/customizing-runner-networking-and-connectivity) in AWS and Azure environments.
</Update>

<Update label="May 21, 2024" description={designer}>
  ## Designer - New features

  * The [Snowpark Container Prompt](/docs/components/snowpark-container-prompt) is now available in **public preview**. This orchestration component lets you host your large language models within your [Snowpark Container Services](https://docs.snowflake.com/en/developer-guide/snowpark-container-services/overview#).
  * The [Pinecone Vector Upsert](/docs/components/pinecone-vector-upsert) component is now available for Databricks and Amazon Redshift projects.
  * Added the following Flex connectors for developing data pipelines:
    * [Orbit](/docs/components/orbit)
    * [Persona](/docs/components/persona)
  * The [{m_runner} for Azure](/docs/guides/create-a-runner) is now available in **public preview**. This lets you run a [Hybrid SaaS](/docs/guides/runner-overview#hybrid-saas) configuration with the {m_runner} hosted in your own Azure infrastructure.
</Update>

<Update label="May 16, 2024" description={designer}>
  ## Designer - New features

  * The AI prompt components are now available for Databricks projects:
    * [OpenAI Prompt](/docs/components/openai-prompt)
    * [AzureOpenAI Prompt](/docs/components/azure-openai-prompt)
    * [Amazon Bedrock Prompt](/docs/components/amazon-bedrock-prompt)
  * The following new components that support Snowflake Cortex LLM operations are available for Snowflake projects:
    * [Cortex Completions](/docs/components/cortex-completions)
    * [Cortex Extract Answer](/docs/components/cortex-extract-answer)
    * [Cortex Sentiment](/docs/components/cortex-sentiment)
    * [Cortex Summarize](/docs/components/cortex-summarize)
    * [Cortex Translate](/docs/components/cortex-translate)
</Update>

<Update label="May 14, 2024" description={designer}>
  ## Designer - New features

  * The AI prompt components are now available for Amazon Redshift projects:
    * [OpenAI Prompt](/docs/components/openai-prompt)
    * [AzureOpenAI Prompt](/docs/components/azure-openai-prompt)
    * [Amazon Bedrock Prompt](/docs/components/amazon-bedrock-prompt)
  * Added two new connectors:
    * [Gmail](/docs/components/gmail), which lets you query the Gmail API to retrieve data and load it into a table. You can then use transformation components to enrich and manage the data in permanent tables.
    * [SurveyMonkey](/docs/components/surveymonkey), which lets you query the SurveyMonkey API to retrieve data and load it into a table. You can then use transformation components to enrich and manage the data in permanent tables.
  * Added the following components for Snowflake projects:
    * [Azure Queue Storage Message](/docs/components/azure-queue-storage-message) lets you post a message to [Azure Queue Storage](https://azure.microsoft.com/en-gb/services/storage/queues/). Other applications can read those messages and perform further processing.
    * [JDBC Table Metadata to Grid](/docs/components/jdbc-table-metadata-to-grid) takes the metadata from a JDBC table and uses this data to populate a [grid variable](/docs/guides/grid-variables).
    * [Table Metadata to Grid](/docs/components/table-metadata-to-grid) takes the metadata from a table and uses this data to populate a [grid variable](/docs/guides/grid-variables) with data.
  * The Git menu can now be accessed by clicking the active branch.
  * The [Data Transfer](/docs/components/data-transfer) component now supports SFTP as a target.
</Update>

<Update label="May 10, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated [{cdc_runner} version](/docs/streaming/agent-versions) from `2.93.1` to `2.93.8`.
  * Improved messaging and logging for some scenarios.
  * Library updates and security fixes.
</Update>

<Update label="May 2, 2024" description={designer}>
  ## Designer - New features

  * The [Python Pushdown](/docs/components/python-pushdown) component is now generally available for Snowflake users. This component lets you execute a Python script using the [Snowpark](https://www.snowflake.com/en/data-cloud/snowpark/) service in your Snowflake account.
</Update>

<Update label="May 1, 2024" description={designer}>
  ## Designer - New features

  * Amazon Redshift is now generally available as a cloud data warehouse in {maia}.
</Update>

<Update label="April 30, 2024" description={designer}>
  ## Designer - New features and improvements

  * Added a [Microsoft Exchange](/docs/components/microsoft-exchange) component, which lets you query the Microsoft Exchange API to retrieve data and load it into a table. You can then use transformation components to enrich and manage the data in permanent tables.
  * Added a [ServiceNow Query](/docs/components/servicenow-query) component, which lets you query the ServiceNow API to retrieve data and load it into a table. You can then use transformation components to enrich and manage the data in permanent tables.
  * Added logging to Custom Connector and to Flex and custom connectors used in {designer}. You can choose one of five logging options: Error, Warn, Info, Trace, or Debug.
  * Improved component sampling to enable column resizing.
</Update>

<Update label="April 24, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated {s_runner} version from `2.92.3` to `2.93.1`.
  * Removed a number of unnecessary dependency overrides and updated some minor dependencies.
</Update>

<Update label="April 18, 2024" description={designer}>
  ## Designer - New features

  * Added the following Flex connectors for developing data pipelines:
    * [Firehydrant](/docs/components/firehydrant)
    * [Infobip](/docs/components/infobip)
    * [Jotform](/docs/components/jotform)
    * [Lob](/docs/components/lob)
    * [Mailgun](/docs/components/mailgun)
    * [Notion](/docs/components/notion)
</Update>

<Update label="April 17, 2024" description={designer}>
  ## Designer - New features

  * Maia (formerly Copilot) is now available as an individual assignable [service](/docs/administration/account-roles#services) in the **Edit User** menu.
    * Super Admins can enable and disable Maia access for all users in an account.
    * User Admins can enable and disable Maia access for all User Admins and Users in an account (not the Super Admin).
</Update>

<Update label="April 11, 2024" description={designer}>
  ## Designer - Improvements

  * Added an improvement to connecting a GitHub repository to a project. Now, when you connect an existing repository, any active branches will be cloned into {designer} and selectable from the **Branches** tab. Previously only the `main` branch was cloned. If any of the active branches in your repository only include files that are not Data Pipeline Language (DPL) files, those branches will appear as empty while in {designer} until you create a pipeline.
</Update>

<Update label="April 4, 2024" description={designer}>
  ## Designer - New features

  * Added the following Flex connector for developing data pipelines:
    * [Azure Cosmos DB for NoSQL](/docs/components/azure-cosmos-db-for-nosql)
</Update>

<Update label="April 2, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated [{cdc_runner} version](/docs/streaming/agent-versions) from `2.92.0` to `2.92.3`
  * Library updates and security fixes
  * Improved {cdc_runner} logging around data handling edge cases to help with debugging
</Update>

<Update label="March 28, 2024" description={designer}>
  ## Designer - New features

  * Added the ability to export [pipelines](/docs/guides/pipelines#import-and-export-pipelines) and import them into a new project.
  * Added the following connectors for developing data pipelines:
    * [Databricks](/docs/components/databricks) connector
    * [Mandrill](/docs/components/mandrill) flex connector
</Update>

<Update label="March 20, 2024" description="Streaming">
  ## Streaming - Improvements

  * Updated [{cdc_runner} version](/docs/streaming/agent-versions) from `2.91.5` to `2.92.0`.
  * Added optional `STREAMING_AGENT_DEBUG_DISABLE_COMPRESSION` Environment Variable to disable compression when uploading to Snowflake Staging area.
  * Library updates and security fixes.
</Update>

<Update label="March 19, 2024" description={designer}>
  ## Designer - New features

  * The following AI components are now generally available to all {maia} users in {designer}:
    * Prompt components:
      * [OpenAI Prompt](/docs/components/openai-prompt)
      * [AzureOpenAI Prompt](/docs/components/azure-openai-prompt)
      * [Amazon Bedrock Prompt](/docs/components/amazon-bedrock-prompt)
    * Vector database operations:
      * [Pinecone Vector Upsert](/docs/components/pinecone-vector-upsert)
      * [Pinecone Vector Query](/docs/components/pinecone-vector-query)
  * [AI Note](/docs/guides/ai-note) is also now generally available, which lets you invoke generative AI to annotate your data pipelines using notes.
  * The project details page in {maia} now displays project metadata including the {m_runner} type, the cloud data platform, and the type of Git repository (Matillion hosted repository or customer's GitHub repository).
</Update>

<Update label="March 14, 2024" description={designer + ", API"}>
  ## Designer - New features

  * The ability to connect your own GitHub repo to {maia} is now generally available. This feature lets you connect your GitHub repository when you create a [project](/docs/guides/projects) in {maia}. Follow the documentation to get started:
    * [Prerequisites for connecting a GitHub repository to {maia}](/docs/guides/connect-github-repository-prerequisites)
    * [Installing the {maia} Integration app in GitHub](/docs/guides/installing-matillion-app-github-marketplace)
    * [Troubleshooting guide for connecting a GitHub repository to {maia}](/docs/guides/troubleshooting-github-repository-connection)

  ## API - New endpoints

  * The [{maia} API](/docs/api-reference/maia-api-overview) is now generally available and includes the following endpoints:

  **Projects**

  | Method  | Endpoint       | Description       |
  | ------- | -------------- | ----------------- |
  | **GET** | `/v1/projects` | List all projects |

  **Environments**

  | Method  | Endpoint                                | Description                         |
  | ------- | --------------------------------------- | ----------------------------------- |
  | **GET** | `/v1/projects/{projectId}/environments` | Lists all environments in a project |

  **Pipelines**

  | Method  | Endpoint                                       | Description                                |
  | ------- | ---------------------------------------------- | ------------------------------------------ |
  | **GET** | `/v1/projects/{projectId}/published-pipelines` | Lists all published pipelines in a project |

  **Pipeline execution**

  | Method    | Endpoint                                                                   | Description                                                                           |
  | --------- | -------------------------------------------------------------------------- | ------------------------------------------------------------------------------------- |
  | **POST**  | `/v1/projects/{projectId}/pipeline-executions`                             | Execute a published pipeline                                                          |
  | **GET**   | `/v1/projects/{projectId}/pipeline-executions/{pipelineExecutionId}`       | Returns the status of a pipeline                                                      |
  | **PATCH** | `/v1/projects/{projectId}/pipeline-executions/{pipelineExecutionId}`       | Terminate a pipeline                                                                  |
  | **GET**   | `/v1/projects/{projectId}/pipeline-executions/{pipelineExecutionId}/steps` | Represents a single execution of a pipeline triggered from a step of another pipeline |
</Update>

<Update label="March 13, 2024" description={designer + ", Streaming"}>
  ## Designer - New features

  * Added a **Text mode** option to [components](/docs/guides/components-overview#text-mode) and [grid variables](/docs/guides/grid-variables#text-mode), to facilitate rapid input and editing of multiple properties.

  ## Streaming - Improvements

  * Updated [{cdc_runner} version](/docs/streaming/agent-versions) from `2.90.16` to `2.91.5`.
  * Increased the {s_runner} JVM memory allocation to 60%.
  * Fixed a bug with the Direct to Snowflake flow, in which a change to the scale of a numeric column of a table in the source database would not trigger a schema change in the target Snowflake table.
  * Library updates and security fixes.
</Update>

<Update label="March 12, 2024" description={designer}>
  ## Designer - New features

  * Added support for [lineage for transformation pipelines](/docs/guides/data-lineage). With lineage, you get a visual representation of each transformation, which can make understanding the evolution of your data faster and easier. Lineage is currently in public preview and includes:
    * Transformation lineage at runtime.
    * Table-level lineage.
    * Column-level lineage.
    * Table lists.
    * Table metadata—column information and data types.
  * Added the following Flex connectors for developing data pipelines:
    * [Bugcrowd](/docs/components/bugcrowd)
    * [Salesloft](/docs/components/salesloft)
    * [Zendesk Ticketing](/docs/components/zendesk-ticketing)
</Update>

<Update label="March 5, 2024" description={designer}>
  ## Designer - Improvements

  * Updated the [AI Note](/docs/guides/ai-note) feature with the ability to refine notes. When you click **Refine** when creating a note, you can elaborate on the note (adding detail), shorten the note's length and detail, or regenerate the note.
  * Added a `Load Strategy` parameter to connectors that support storage-only outputs. Choose between appending files in folder and overwriting files in folder, based on exact structure matching.
</Update>

<Update label="February 29, 2024" description={designer + ", Streaming"}>
  ## Designer - New features

  * Added the ability to access the Snowflake `Session` object from the [Python Pushdown](/docs/components/python-pushdown) component, to allow reading from and writing to the Snowflake database from within a Python script.
  * Added the following Flex connectors for developing data pipelines:
    * [Asana](/docs/components/asana)
    * [New Relic](/docs/components/new-relic)
    * [SAP SuccessFactors](/docs/components/sap-successfactors)
    * [Ticketmaster](/docs/components/ticketmaster)
  * Added Azure Blob Storage as a data input type for the [Excel Query](/docs/components/excel-query) component.
  * Added Azure Blob Storage as a source and target type for the [Data Transfer](/docs/components/data-transfer) component.

  ## Streaming - Improvements

  * Updated [{cdc_runner} version](/docs/streaming/agent-versions) from `2.90.16` to `2.90.35`.
  * Logging improvements.
  * Library updates and security fixes.
</Update>

<Update label="February 28, 2024" description={designer}>
  ## Designer - New features

  * Added the [Query Result to Grid](/docs/components/query-result-to-grid) component, which queries a table and returns rows of data that are loaded into a predefined grid variable.
</Update>

<Update label="February 27, 2024" description={designer}>
  ## Designer - New features

  * Improved user interface for easier pipeline creation. Read [{designer} UI basics](/docs/guides/designer-ui-basics) for details.
</Update>

<Update label="February 21, 2024" description="Streaming">
  ## Streaming - New features

  * Added Snapshot Overview to display improved metrics when a pipeline is snapshotting.
</Update>

<Update label="February 20, 2024" description={designer}>
  ## Designer - New features

  * Added Azure Blob Storage as a data input type for the [File Iterator](/docs/components/file-iterator) component.
</Update>

<Update label="February 9, 2024" description={designer}>
  ## Designer - New features

  * Added support for [grid variables](/docs/guides/grid-variables) in pipelines.
  * Added a [Grid Iterator](/docs/components/grid-iterator) component for working with grid variables.
</Update>

<Update label="February 8, 2024" description={designer}>
  ## Designer - New features

  * Added the [Mailchimp](/docs/components/mailchimp) component, which lets you query the Mailchimp API to retrieve data and load it into a table. You can then use transformation components to enrich and manage the data in permanent tables.
  * Added the following Flex connector for developing data pipelines:
    * [Mixpanel](/docs/components/mixpanel)
  * Added a toggle to the [Schedules](/docs/guides/schedules) tab where users can easily enable or disable their schedules.
</Update>

<Update label="February 6, 2024" description={designer}>
  ## Designer - New features

  * Added the [Azure SQL](/docs/components/azure-sql) component, which lets you run an SQL query on an Azure SQL database and copy the results to a table. You can then use transformation components to enrich and manage the data in permanent tables.
</Update>

<Update label="February 1, 2024" description={designer + ", Streaming"}>
  ## Designer - New features

  * Databricks is now a generally available cloud data platform for {maia}.
  * The following components have been added to {maia} for Databricks:
    * [Except](/docs/components/except)
    * [Extract Nested Data](/docs/components/extract-nested-data)
    * [Map Values](/docs/components/map-values)
    * [Pivot](/docs/components/pivot)
    * [Unpivot](/docs/components/unpivot)

  ## Streaming - Improvements

  * Updated [{cdc_runner} version](/docs/streaming/agent-versions) from `2.89.4` to `2.90.16`.
  * Library updates and security fixes.
</Update>

<Update label="January 25, 2024" description={designer + ", " + maia}>
  ## Designer - New features

  * Added a [Commands for dbt Core](/docs/components/dbt-core) component. This component empowers you to run [dbt](https://www.getdbt.com/product/what-is-dbt) commands as part of your orchestration pipelines.

  ## Maia - Improvements

  * Improved the navigation menu, adding links to manage your custom connectors and {m_runner}s.
</Update>

<Update label="January 23, 2024" description={designer}>
  ## Designer - New features and improvements

  * Added a [Python Pushdown](/docs/components/python-pushdown) component to {designer}. This component lets you execute a Python script using the [Snowpark](https://www.snowflake.com/en/data-cloud/snowpark/) service in your Snowflake account. This component is currently available as a public preview.
  * Improved the [schedule](/docs/guides/schedules) feature to include a Standard mode and an Advanced mode.
    * Standard mode lets you define intervals with the units Days (default), Weeks, Hours, and Minutes.
    * Advanced mode retains the Cron expression scheduling feature.
</Update>

<Update label="January 17, 2024" description="Streaming">
  ## Streaming - New features

  * Added option to configure or disable the table name prefix when using Snowflake as a target.
</Update>

<Update label="January 15, 2024" description={designer}>
  ## Designer - New features

  * Added the following Flex connector for developing data pipelines:
    * [Intercom](/docs/components/intercom)
</Update>

<Update label="January 11, 2024" description={designer}>
  ## Designer - New features and improvements

  * Added the following Flex connector for developing data pipelines:
    * [Ongoing WMS](/docs/components/ongoing-wms)
  * Fixed an issue where using the Rename component would reorder columns.
  * Fixed an issue where using the Convert Type component would reorder columns.
</Update>

<Update label="January 9, 2024" description={designer}>
  ## Designer - New features

  * Added the following Flex connector for developing data pipelines:
    * [Dixa](/docs/components/dixa)
</Update>

<Update label="January 5, 2024" description={designer}>
  ## Designer - New features

  * Added the following Flex connectors for developing data pipelines:
    * [GitHub](/docs/components/github)
    * [Yelp](/docs/components/yelp)
</Update>