Pipelines

Pipelines are ‘s way of designing, organizing, and executing workflows. You use to build strings of configured components inside a pipeline and then run that pipeline to carry out a desired task such as loading or transforming data. Pipelines are created, configured, and managed through the main user interface. There are two types of pipeline in : orchestration and transformation.

• Orchestration pipelines deal with the loading of data from source system to target data warehouse. Typical orchestration components are connectors, flow logic components, and scripting components.
• Transformation pipelines deal with transforming table data that exists in your target data warehouse, typically after loading that data with an orchestration pipeline. Transformation components are often analogs of SQL operations such as creating and deleting tables, joining data, or performing calculations.

Components are the basic building blocks of pipelines. Each component is specifically applicable to one type of pipeline, orchestration or transformation, and can’t be added to the other type of pipeline. To learn more about components, read Components overview. Any pipeline can be saved as a shared pipeline, which can then be referenced by any other pipeline in any project in your . This feature helps maintain consistency of core functionality over multiple different projects across the organization. Our Pipelines tutorial is a step-by-step walkthrough that explains how to create an orchestration pipeline and a transformation pipeline using sample data provided by Matillion. This guide may be useful if you are new to Designer and want to practice creating pipelines and configuring components.

---

​

Adding a pipeline

To create a new pipeline:

1. In the user interface, click Files at the top of the canvas.
2. Click the + icon in the top right of the Files panel, and select either Orchestration pipeline or Transformation pipeline.
3. In Name, enter a name for the pipeline. The name may include alphanumerics, underscores, single spaces, parentheses, and hyphens. The name must be unique within each pipeline folder.
4. In the Description field, you can optionally enter a description for the pipeline. This is useful if you want to share this pipeline with other users. Max 1000 characters.
5. Click Add. The new pipeline immediately opens in a new tab on the canvas.

This will place the pipeline at the top level of your pipeline folder tree. To create a pipeline within a folder, read Pipeline folders, below. The newly created pipeline is blank at this stage. You now need to add components to the pipeline canvas to construct a workflow that will perform your data extraction and transformation tasks. When you start using , you can also use the Orchestration pipeline or Transformation pipeline tiles to create your first pipelines, or click the Watch video links to see a short video tutorial on pipeline creation.

---

​

Managing pipelines

In the user interface, click Files to view your pipelines. The Files panel lists every pipeline you have created on your current branch. Icons identify the type of pipeline: a blue O for orchestration, or green T for transformation. The pipelines can be organized into a hierarchy of folders, if required. To view a pipeline on the canvas, do one of the following:

• Click the pipeline in the list.
• Hover over the pipeline, click the three dots … that appear next to it, and click Open.

To make a copy of a pipeline, do one of the following:

• Click the pipeline to select it, then press CTRL + c (Windows) or CMD + c (Mac) to copy it.
• Hover over the pipeline, click the three dots … that appear next to it, and click Copy.

Then do one of the following:

• Click the folder in which you want to paste a copy of the pipeline, then press CTRL + v (Windows) or CMD + v (Mac) to paste it.
• Hover over the folder in which you want to paste a copy of the pipeline, click the three dots … that appear next to it, and click Paste.

To rename a pipeline or edit a pipeline’s description, hover over the pipeline, click the three dots … that appear next to it, and click Edit details. Then edit the name and/or description, and click Update. To delete a pipeline, hover over the pipeline, click the three dots … that appear next to it, and click Delete. Then click Delete to confirm the deletion. Deleted pipelines can’t be recovered.

​

Cancelling a running pipeline

You can cancel a pipeline while it’s running in one of two ways. A soft cancellation will allow the current task to complete and then no other pipeline tasks will run. A forced cancellation will interrupt and terminate the current task, set the pipeline status to “Stopped”, and will prevent the pipeline from receiving further information from the agent. Click the Task history tab, and then click the “X” icon on the far-right of the task you wish to cancel. A single click will perform a soft cancellation. Clicking twice will perform a forced cancellation. A pop-up dialog will appear asking you to confirm the force cancel. Click, Yes, force cancel to terminate the pipeline, without waiting for any in-progress steps to finish. You can also cancel pipeline runs from the Observability Dashboard.

---

​

Pipeline folders

A branch may contain a large number of pipelines, which by default are listed alphabetically in the Files panel. To organize pipelines in a branch, you can arrange them into named folders. Folders can be nested inside folders, up to 10 levels deep, to create a structure that makes sense to you and your team. Pipeline names must be unique within a folder but can be duplicated in different folders. To create a top-level folder, click Files at the top of the canvas, then click the + icon in the Files panel and click New folder. Enter a name for the folder and click Create. Folder names can contain alphanumeric characters, dashes, and underscores. You can collapse or expand a folder by clicking the > icon. Each folder has its own context menu where you can add a pipeline to the folder, add another folder as a subfolder, rename the folder, or delete the folder. Click the three dots … next to the folder name to open the context menu. When you delete a folder, all the pipelines and subfolders it contains will also be deleted. However, active schedules for pipelines in a deleted folder will continue to use the latest published version of the pipelines. Deleting a folder can’t be undone. You can move a pipeline between folders, or between root and folder, by drag-and-drop. You can’t move a pipeline to a folder where another pipeline with the same name already exists, and attempting to do so will result in a warning notice.

​

Copying pipeline folders

To make a copy of a pipeline folder and all of its contents, including pipelines and subfolders, do one of the following:

• Click the pipeline folder to select it, then press CTRL + c (Windows) or CMD + c (Mac) to copy it.
• Hover over the pipeline folder, click the three dots … that appear next to it, and click Copy.

Then do one of the following:

• Click the folder in which you want to paste a copy of the pipeline folder, then press CTRL + v (Windows) or CMD + v (Mac) to paste it.
• Hover over the folder in which you want to paste a copy of the pipeline folder, click the three dots … that appear next to it, and click Paste.

---

​

Importing and exporting pipelines

Pipelines can be shared between projects using the export and import functions. Export copies the pipeline definition to a zipped data file in your local filesystem. You can then use the Import function to import that data into another project. You can also export/import an entire pipeline folder, including all the pipelines and subfolders it contains, or export/import the entire project including all folders and pipelines within it. There is a limit on the size of the pipeline you want to import:

• Maximum compressed size: Fails if import exceeds 8 MB.
• Maximum decompressed size: Fails if import exceeds 100 MB.
• Maximum file size: Fails if any file exceeds 5 MB.
• Maximum files: Fails if import exceeds 2500 files.

Each export operation creates a single zipped .yaml format data file containing all the export pipeline definitions. If the export is at a folder or project level, the zip file preserves the pipeline folder structure within it. The export includes everything required to recreate the exported pipelines in a new project, including all component property values, pipeline variables, component connections, and canvas notes. It does not include project-level configurations such as schedules, secrets, OAuths, and project variables, so you must ensure the target project has these configured as required before running the imported pipeline.

​

Exporting a pipeline

In the Files panel, hover over a pipeline, a folder, or the project name at the top of the folder tree, click the three dots … that appear next to it, and click Export. The file will be exported to the default download location of your local filesystem, with a name of the form <project name>_<branch>_<file/folder name>_<yyyymmdd>T<hhmmss>.zip, creating a unique name for each export.

​

Importing a pipeline

1. In the Files panel, hover over the folder that you want to import into, click the three dots … that appear next to it, and click Import.
2. Browse your filesystem to locate the exported zip file that you want to import, select the file, and click Open.

The pipelines contained in the zip file will be imported to the selected pipeline folder. If you are importing a pipeline folder or an entire project, it will form a subfolder of the folder you are importing into.

​

Importing a pipeline from the Matillion Exchange

Pre-built pipelines are available from Matillion Exchange. To import a pipeline or set of pipelines from Matillion Exchange into , follow these steps:

1. In the Files panel, click Add, then click Browse Exchange.
2. In the Browse Exchange panel that opens on the right of the canvas, search for and click the pipeline or set of pipelines that you want to import.

Your chosen pipelines will be added to a new folder called Imported from Exchange. Each pipeline or set of pipelines that you import from Matillion Exchange will be added to a subfolder inside this one.