> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maia.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Data Pipeline Language (DPL)

export const maia_team = "Maia Team";

export const designer = "Designer";

export const maia = "Maia";

Every pipeline you build in {designer} is stored as a file written in Matillion's Data Pipeline Language (DPL). DPL is a YAML-based format: each pipeline component, its parameters, and the connections between components are represented as structured YAML. The only place you see raw DPL in {designer} is when you review a pipeline's changes using the [Compare changes](/docs/guides/git-compare) Git action.

<Note>
  You never need to write DPL directly. You build pipelines by configuring components on the {designer} canvas or by prompting {maia_team}, and {maia} keeps the underlying DPL file in sync with those changes. This page explains what DPL is and how it's structured, for background only.
</Note>

{designer} stores each pipeline type in a file with a corresponding extension:

* Orchestration pipelines: `.orch.yaml`
* Transformation pipelines: `.tran.yaml`
* Test pipelines: `.test.yaml`

To learn about building these pipeline types, read [Building pipelines](/docs/guides/maia-pipelines) and [Test pipelines](/docs/components/test-pipelines). For a full list of pipeline types available in {maia}, read [Maia product overview](/docs/guides/maia-overview).

***

## Why pipelines use DPL

Pipelines are written in a YAML-based format so they are easier to read, version, and manage as code. The DPL format is:

* **More human-readable:** The YAML structure makes a pipeline's components and configuration easier to read and reason about.
* **Versioned:** Every DPL file declares a schema version, so Matillion can evolve the format while maintaining compatibility with existing pipelines.
* **Extensible:** New capabilities can be added to DPL without breaking pipelines that were created on an earlier version of the format.
* **Suited to programmatic creation:** Pipelines can be generated and modified programmatically, not only through manual configuration. This is part of what allows {maia_team} to build and update pipelines from natural language prompts.
* **Git-friendly:** YAML produces cleaner, more meaningful diffs when pipelines are committed to a Git repository, making changes easier to review and collaborate on.
* **Aligned with data management and DevOps best practices:** Storing pipelines as structured, versioned text supports managing pipelines the same way you manage other code artifacts.
* **Backed by runtime integrity guarantees:** The platform validates a pipeline's DPL definition when it executes, so pipelines run consistently with how they were configured.

***

## Structure of a DPL file

A DPL file is organized into a small number of top-level sections:

* **`type` and `version`:** identify the pipeline type the file represents, and which version of the DPL schema it conforms to.
* **`pipeline.components`:** each component on the canvas, keyed by its component name, including the component's type, its configured parameters, and either its transitions to other components (orchestration and test pipelines) or its upstream sources (transformation pipelines).
* **`variables`:** any scalar or grid variables used in the pipeline, along with metadata such as type, description, scope, and visibility.
* **`design`:** the canvas layout information, such as each component's position, that {designer} uses to render the pipeline visually.

Test pipelines are created on the canvas in the same way as orchestration pipelines, so their DPL follows the same `pipeline.components` and `transitions` structure. A test's DPL typically also includes a [Run Pipeline To Test](/docs/components/run-pipeline-to-test) component and one or more dedicated test components, such as assert or compare components, used to check the results of the pipeline under test. Read [Test pipelines](/docs/components/test-pipelines) for more information.

The following is a simplified, illustrative extract of an orchestration pipeline's DPL, shown only to illustrate the structure described above:

```yaml theme={null}
type: "orchestration"
version: "1.0"
pipeline:
  components:
    Start:
      type: "start"
      transitions:
        unconditional:
          - "Create staging table"
      parameters:
        componentName: "Start"
    Create staging table:
      type: "create-table"
      transitions:
        success:
          - "Load data"
      parameters:
        componentName: "Create staging table"
        newTableName: "STAGING_TABLE"
    Load data:
      type: "s3-load"
      parameters:
        componentName: "Load data"
  variables:
    batch_id:
      metadata:
        type: "TEXT"
        description: "batch_id"
        scope: "COPIED"
        visibility: "PRIVATE"
      defaultValue: "batch-001"
design:
  components:
    Start:
      position:
        x: 0
        "y": 0
    Create staging table:
      position:
        x: 195
        "y": 0
```

<Note>
  {designer} generates and updates this file automatically as you configure components on the canvas. Editing an `.orch.yaml`, `.tran.yaml`, or `.test.yaml` file outside {designer} is not a supported workflow.
</Note>

***

## DPL and Git

{maia} is built on top of Git, and {designer} integrates with Git natively so you don't need to install or maintain a local copy of the pipeline files yourself. Because DPL is YAML-based, changes to a pipeline produce Git diffs that map to the change you made in {designer}, which makes pipeline changes easier to review, track, and collaborate on.

To learn more about how Git fits into the wider platform, read [Maia product architecture](/docs/guides/architecture-overview).

***

## Related reading

* [Building pipelines](/docs/guides/maia-pipelines)
* [Test pipelines](/docs/components/test-pipelines)
* [Using Designer](/docs/guides/using-designer)
* [Shared pipelines](/docs/guides/shared-pipelines)
* [Maia product architecture](/docs/guides/architecture-overview)