Skip to main content
Every pipeline you build in is stored as a file written in Matillion’s Data Pipeline Language (DPL). DPL is a YAML-based format: each pipeline component, its parameters, and the connections between components are represented as structured YAML. The only place you see raw DPL in is when you review a pipeline’s changes using the Compare changes Git action.
You never need to write DPL directly. You build pipelines by configuring components on the canvas or by prompting , and keeps the underlying DPL file in sync with those changes. This page explains what DPL is and how it’s structured, for background only.
stores each pipeline type in a file with a corresponding extension:
  • Orchestration pipelines: .orch.yaml
  • Transformation pipelines: .tran.yaml
  • Test pipelines: .test.yaml
To learn about building these pipeline types, read Building pipelines and Test pipelines. For a full list of pipeline types available in , read Maia product overview.

​
Why pipelines use DPL

Pipelines are written in a YAML-based format so they are easier to read, version, and manage as code. The DPL format is:
  • More human-readable: The YAML structure makes a pipeline’s components and configuration easier to read and reason about.
  • Versioned: Every DPL file declares a schema version, so Matillion can evolve the format while maintaining compatibility with existing pipelines.
  • Extensible: New capabilities can be added to DPL without breaking pipelines that were created on an earlier version of the format.
  • Suited to programmatic creation: Pipelines can be generated and modified programmatically, not only through manual configuration. This is part of what allows to build and update pipelines from natural language prompts.
  • Git-friendly: YAML produces cleaner, more meaningful diffs when pipelines are committed to a Git repository, making changes easier to review and collaborate on.
  • Aligned with data management and DevOps best practices: Storing pipelines as structured, versioned text supports managing pipelines the same way you manage other code artifacts.
  • Backed by runtime integrity guarantees: The platform validates a pipeline’s DPL definition when it executes, so pipelines run consistently with how they were configured.

​
Structure of a DPL file

A DPL file is organized into a small number of top-level sections:
  • type and version: identify the pipeline type the file represents, and which version of the DPL schema it conforms to.
  • pipeline.components: each component on the canvas, keyed by its component name, including the component’s type, its configured parameters, and either its transitions to other components (orchestration and test pipelines) or its upstream sources (transformation pipelines).
  • variables: any scalar or grid variables used in the pipeline, along with metadata such as type, description, scope, and visibility.
  • design: the canvas layout information, such as each component’s position, that uses to render the pipeline visually.
Test pipelines are created on the canvas in the same way as orchestration pipelines, so their DPL follows the same pipeline.components and transitions structure. A test’s DPL typically also includes a Run Pipeline To Test component and one or more dedicated test components, such as assert or compare components, used to check the results of the pipeline under test. Read Test pipelines for more information. The following is a simplified, illustrative extract of an orchestration pipeline’s DPL, shown only to illustrate the structure described above:
generates and updates this file automatically as you configure components on the canvas. Editing an .orch.yaml, .tran.yaml, or .test.yaml file outside is not a supported workflow.

​
DPL and Git

is built on top of Git, and integrates with Git natively so you don’t need to install or maintain a local copy of the pipeline files yourself. Because DPL is YAML-based, changes to a pipeline produce Git diffs that map to the change you made in , which makes pipeline changes easier to review, track, and collaborate on. To learn more about how Git fits into the wider platform, read Maia product architecture.