Skip to main content
A shared pipeline is a way of saving an entire pipeline workflow and then re-using that same workflow in any project in . This feature helps maintain consistency over multiple different projects. For example, a designated “library” project could be used for functionality that must be used in a consistent way across the organization. This functionality would be created as version-controlled shared pipelines, and all other users instructed to use these pipelines in their own projects, using the Run Shared Pipeline component. Shared pipelines have similarities to the Run Orchestration and Run Transformation components, but allow pipelines to be shared between projects, and give you more options for pipeline versioning. If you use multiple cloud data warehouses across different projects, your shared pipelines can be used in projects that connect to any warehouse. However, you need to take care when using components that aren’t available, or aren’t identical in operation, across all warehouses. Any pipeline can become a shared pipeline, however there are some best-practice considerations, described below, that you should be aware of when creating a pipeline that you intend to be shared. For full details of creating pipelines, read Pipelines.
We call a pipeline that makes use of a shared pipeline the consumer or consuming pipeline.

​
Video example

Expand this box to watch our video about shared pipelines.

​
Sharing a pipeline

When you share a pipeline, you define how it will appear to other users, including which variables are required and how they are labeled.
  1. Open the orchestration or transformation pipeline you want to share.
  2. In the Files panel, click the three dots … next to the pipeline you want to share and then click Share.
  3. In the Share pipeline dialog, configure the following:
    • Pipeline ID: A unique identifier for the pipeline.
    • Description: A detailed description of the pipeline’s purpose.
  4. In the Scalar variables and Grid variables tabs, all existing variables created for the pipeline are automatically listed. Each variable displays its name, description, and default value as read-only. You can configure the following:
    • Display name (optional): Enter a user-friendly name to appear in the component properties.
    • Required: Select this checkbox if the variable must have a value provided by the consumer.
    • Reorder variables: Use the drag handles to change the order in which variables are displayed to consumers.
To change a variable’s name, description, or default value, click the Manage variables icon in Designer, then click the pencil icon next to the specific variable you want to edit. See Variables for details.
  1. Click Share to create the shared pipeline configuration.
  2. Commit the pipeline’s branch, as described in Git commit.
  3. Push and Publish the branch, as described in Git push local changes. Publishing is required to make it available to other pipelines. Ensure the Include shared pipelines toggle is enabled during publishing (this is toggled on by default) to make the pipeline available to other projects.
When a pipeline is shared, either creates or updates the configuration in a file named shared-pipelines.yaml. You must not rename or move this file, and we recommend that you do not attempt to manually edit its contents. See Editing the configuration file, below, for further information.

​
Shared pipeline artifacts

When a shared pipeline is published using the Push local changes action, it is available at the account level. This means that it will be available to all projects and environments. In contrast, the artifacts that are published for non-shared pipelines are only available at the project level. If you delete a project, shared pipeline artifacts are not deleted, even if they were published from within the deleted project. gives each shared pipeline artifact a project prefix, so version names only need to be unique at the project level. For example, let’s say you have a project called Project1 containing an artifact with the version name version1.0, and a project called Project2 containing an artifact with the same version name. When you publish shared pipelines using these version names, will store them as Project1/version1.0 and Project2/version1.0.

​
Using a shared pipeline

Use the Run Shared Pipeline component to add the shared pipeline to any orchestration pipeline. The default values for any pipeline variables set in the original pipeline will be used in the consuming pipeline, unless new values are set in the Run Shared Pipeline component.
  1. Add the Run Shared Pipeline component to an orchestration pipeline.
  2. Select the Shared Pipeline you want to run.
  3. Select the version to execute:
    • [Latest] runs the most recently published version.
    • Selecting a specific version ensures a fixed, predictable execution.
  • Variables marked as Required must have a value at runtime. If a required variable has a default value, that value is used automatically and no input is needed. If it has no default, you must provide a value before the pipeline can run.
  • Optional variables are labeled (Optional) and do not block execution, even if no default or user-provided value is set.

​
Set scalar variables

  • All public scalar variables defined in the shared pipeline are listed.
  • Variable names are populated automatically from the shared pipeline.
  • Required scalar variables must have a value provided before the pipeline can run and optional variables use their default values unless overridden.
  • For each scalar variable, you can:
    • Enter a static value directly.
    • Enable Use Grid Variable to drive the value dynamically from a grid variable.

​
Set grid variables

  • All public grid variables defined in the shared pipeline are listed.
  • Required grid variables must contain valid rows before execution, and optional grid variables use their default values unless overridden.
  • Grid variables accept multiple rows of structured input data.

​
Editing a shared pipeline

To edit a shared pipeline, click the three dots … next to the pipeline you want to edit and then click Edit details. The Scalar variables and Grid variables tabs display the same fields as when the pipeline was first shared. As with sharing, only the Display name, Required checkbox, and variable order can be updated here. The variable name, description, and default value remain read-only.
To change a variable’s name, description, or default value, click the Manage variables icon in Designer, then click the pencil icon next to the specific variable you want to edit. See Variables for details.
  1. You can update:
    • Display name (optional): For scalar and grid variables.
    • Required: To mark variables as mandatory or optional.
    • Reorder variables: Use the drag handles to change the order in which variables are displayed to consumers.
  2. Click Update to save the changes.
  3. Commit, push, and publish to apply your changes.
Changes take effect based on the version selected in consuming pipelines.

​
Stop sharing a pipeline

To stop sharing a pipeline, click the three dots … next to the pipeline you want to stop sharing and then click Unshare. This will make the pipeline disappear from the drop-down in the Run Shared Pipeline component, so it can’t be used by any new consumers. However, pipelines already consuming it will continue to run the latest available version after sharing is stopped. A Run Shared Pipeline component will show as “invalid” to warn that it’s referencing an unshared pipeline, but it will continue to run correctly using the selected version.

​
Best practices for shared pipelines

When creating a shared pipeline, be aware of the following best practices.

​
General principles

  • Avoid using the same name for both a transformation and an orchestration pipeline in the same folder, as this will cause confusion over which is referenced by the Run Shared Pipeline component.
  • Avoid changing the name of a project containing shared pipelines. If you change the project name, any Run Shared Pipeline components currently configured to use a pipeline from that project will continue to run, but won’t be able to access subsequent changes you may make to the shared pipeline, unless the consumer re-points the Run Shared Pipeline component to the new project name.
  • If you use multiple cloud data warehouses, your shared pipelines can be used in projects that connect to any warehouse, but you need to take care to avoid component incompatibilities between warehouses. One suggestion is to include the warehouse type as part of the pipeline name, to alert the consumer that it might not be compatible with the warehouse they are using.

​
Consuming pipelines

  • Avoid self-referencing a shared pipeline in the same project it came from. If both are shared, there will be no way to distinguish them in the Run Shared Pipeline component.
  • Take care if referencing a shared pipeline in a pipeline that will itself be shared. This can create issues with circular dependencies and make troubleshooting difficult.

​
Variables

  • Avoid hard coding [Environment Default] into the component parameters of a shared pipeline. It’s better to use pipeline variables to define parameters wherever possible, and the value [Environment Default] can then be set in the consuming pipeline, as this makes all declarations explicit for the consumer.
  • Avoid using project variables within a shared pipeline, as the pipeline won’t work if consumed in a project without those variables defined. It’s better to use pipeline variables only, as these can then be given suitable new values by the consumer.
    For example, if you created a shared pipeline in “Project 1” using a project variable called ${proj_var}, you would also need to define ${proj_var} in the consuming project, “Project 2”. This is because a shared pipeline will try to resolve variables in its current project, not the one it was shared from. We do not recommend using shared pipelines in this way.
  • Consider deleting variable defaults for any variables that are required inputs by the consumer. This will avoid any uncertainty over whether the consumer should use that default value or set their own value in Run Shared Pipeline. This only applies to variables with Public visibility.
  • If you want to set a variable value that can’t be overridden by the consumer, set the variable visibility to Private.
  • Consider using error trapping in the shared pipeline, to verify that the consumer isn’t giving invalid values to a variable used by the shared pipeline. You could do this by checking the variable values in an If component and passing the pipeline flow to an End Failure component if an invalid value is found.

​
Versioning and release workflow

Shared pipelines are versioned as part of the project, to ensure that any dependencies are versioned together. When using shared pipelines, you should consider a version management strategy that works best for you. Some possible approaches are:
  • Consumers choose a particular version of the project to take the shared pipeline from. This avoids unexpected “breaking” changes to the shared pipeline being pushed to the consumer, as the owner of the consuming pipeline can choose when and how often to update to a newer project version. The disadvantage is that the consumers won’t automatically inherit necessary updates made to the shared pipeline. If you parameterize the Version property of your Run Shared Pipeline components, using Variables, you can easily upgrade multiple consuming pipelines in a single operation by updating the project variable.
  • Create a new shared pipeline for each new “breaking” change to the pipeline. The existing pipeline is only updated for essential, non-breaking changes. The consumer could then select “Latest” as the shared pipeline version, so they automatically receive all essential changes from the latest version but have confidence that no breaking changes will be forced on them. Some disadvantages of this approach are:
    • “Legacy” shared pipeline need to be maintained in the project for as long as they are being consumed, and a deprecation strategy may be needed to deal with an ever-expanding number of versioned pipelines.
    • Consumers need to trust the shared pipeline producers not to apply breaking changes to a shared pipeline they are consuming.
    If using this strategy, you can parameterize the name of the pipeline being called by your Run Shared Pipeline components, using a project variable, to allow easy upgrading to a named new version when an old version is deprecated. For example, project-name#pipeline-name-${pipeline_version}.
In the context of shared pipelines, “breaking” changes are any changes that could cause the consuming pipeline to fail. For example, adding or removing pipeline variables that the consumer is trying to map values to.

​
Editing the configuration file

Shared pipeline configuration is stored in a file called shared-pipelines.yaml, found in the .matillion folder of your Git repository.
You must never move or rename this file.
This is only applicable if you are using your own Git provider. On Matillion-hosted Git, this file is not accessible.
The file is automatically updated when changes are made to shared pipelines, so you should never normally need to manually edit the file, and we recommend that you don’t attempt to. However, there may be occasions when editing is required, for example to resolve a conflict arising during merging. In these cases, it’s important to understand the structure of the file. In shared-pipelines.yaml, each pipeline is listed, along with the properties needed to define the pipeline. The structure is as follows: 
version: "1.0"
type: shared-pipelines-config
pipelines:
  # <pipeline name> is the full path of the pipeline, for example folder1/sub-folder1/Shared Pipeline.orch.yaml
  - pipeline: <pipeline name>
    # <display name> is not actively used
    displayName: <display name>
    # <pipeline-id> is the value set the first time the pipeline is shared and what is selected by the consuming user. It can only contain letters, numbers and the following characters: - _ . ! ~ * ' ( )
    id: <pipeline-id>
    description: ""
    # enabled reflects the current sharing status of the pipeline. If pipelines are Unshared, this value is set to 'false', which allows the ID to persist in the event it is shared in a future version of the project
    enabled: true
    # parameters list the public scalar and grid variables in the root pipeline. The below show two variables, var-1 and var-2
    parameters:
      var-1:
        type: TEXT
        exportAllowed: true
        description: ""
        displayName: ""
        required: true
        default: ""
      var-2:
        type: GRID
        exportAllowed: true
        description: ""
        displayName: ""
        required: true
        default: ""
        columns:
          test-1:
            columnType: TEXT
If you need to edit this file for any reason, take care to avoid the following errors, which would cause the pipeline to fail:
  • Duplicate, missing or mis-spelled object properties.
  • Mandatory properties having a blank value.
  • Two or more pipelines with the same <pipeline-id>.
  • Invalid pipeline reference.
  • <pipeline-id> containing invalid characters.