Skip to main content
Components are the basic building blocks of pipelines. Each component represents a single task within the pipeline. For example, the Create Table component executes a task that creates a table when the pipeline runs, while the Salesforce Query component executes a task that returns data from Salesforce. Multiple components can be linked together to perform complex pipelines involving multiple tasks. For example, a pipeline may pull data from Salesforce, filter the data according to some criteria, and write the data to files in an S3 bucket, using the Salesforce Query, Filter, and S3 Unload components executed sequentially. Each component is specifically applicable to one type of pipeline, orchestration or transformation, and you can’t add it to the other type of pipeline. However, see the Run Transformation component for a way to include a transformation pipeline (and its components) in an orchestration pipeline.

​
Component availability

Some components are only available for specific data warehouses. In addition, due to security and data flow considerations, some components are only available for Hybrid SaaS environments. To see which data warehouses and environments a component is available for, read the corresponding component guide in the Components section of the Maia documentation. For more information about Hybrid SaaS environments, read Matillion Full SaaS vs Hybrid SaaS.

​
View components

To view the components that make up a pipeline, click Files at the top of the canvas, then click the pipeline you want to open. This displays the full pipeline on the canvas. Each component is represented by a separate icon on the canvas, and the connections between the component icons show the pipeline flow—the order in which components are executed. When you have a pipeline open, you can view the components available in that pipeline by clicking the Add component icon on the canvas.
If a component appears grayed out on the canvas, this indicates that it has been superseded by a newer component. This behavior may depend on the cloud data warehouse your project uses. For example, in Snowflake projects the HubSpot Query connector appears grayed out because it has been superseded by the HubSpot Load connector.

​
Add components to a pipeline

To add a component to a pipeline, open the pipeline in the canvas and use any of the following methods to open the Component dialog, where you can search for and click the component you want to add:
  • Click the Add component icon on the left of the canvas or the + icon on an empty canvas. The new component will appear on the canvas and will not be connected to any existing components.
  • Click the + icon to the right of a component already on the canvas. When using this method, you can also select the Output Connection that is used to connect the previous component to the new component (On success, Unconditional, or On failure).
  • Click on an existing connection line linking two components. The new component will be placed directly between the two existing components.
All new components are given a default name when placed on the canvas, which can be changed in the component’s properties.

​
Search and filter components

The Add component dialog only shows the components that can be used in the type of pipeline currently open. For example, Create Table is an orchestration component that can only be added to orchestration pipelines. As a result, it is only shown in the Add component dialog when you have an orchestration pipeline open. Use the Search box in the Add component dialog to search for a component by name or function. For example, the Send Email component will appear if you search for messaging. The search will also attempt to match based on synonyms—for example, searching for “logic” will find the If component. Click one or more of the category filters under the Search field to show only the selected types of component.

​
Add warehouse data to a pipeline

To bring existing warehouse data into a transformation pipeline, you can also click the Add component icon and select the Warehouse data tab. By default, this tab shows all tables and views in your environment’s default database—use the drop-down in the top left of the tab to select a different database, and search for schemas within the selected database. Select a table or view to see its metadata and sample the data it contains. Once you’ve selected the table or view you want to use in your transformation pipeline, click Use to create a Table Input component in your pipeline containing the selected data.

​
Component context menu

Right-click a component to display the component context menu, where you can perform the actions listed below. For more information about Windows and Mac shortcuts, read Keyboard shortcuts in Windows and Mac keyboard shortcuts, respectively.

​
Component action bar

Click a component to display the component action bar, where you can perform the actions in the table below. Some actions are only available in orchestration pipelines. If you select multiple components, the Copy, Skip on run, and Delete actions are available in the action bar.

​
Copy components

You can copy a component to create a duplicate elsewhere in the pipeline. Use the component’s context menu or the component action bar to copy the component, and then use the context menu to paste a copy of the component, including all configured properties, elsewhere on the canvas. Connect the copy into the pipeline as required. You can also copy a component between pipelines. Use the component’s context menu to copy the component, switch to a different pipeline, and use the context menu to paste the component onto the new pipeline. The copied component is identical to the original, including all configured properties. The two pipelines must be of the same type; you can’t paste from an orchestration pipeline to a transformation pipeline, or vice versa. Select multiple components by holding down the SHIFT key while you click all the components you want to select, or hold down SHIFT while you drag the mouse pointer over the components you want to select. You can then copy all selected components as a single operation. To deselect a single component, hold the SHIFT key and click the component. To deselect all components, click on any empty part of the canvas.

​
Connect components

Each component on the canvas has two “connection rings,” one on the left and one on the right, which are used to connect the component to other components in the pipeline. The rings become visible when you hover over the component. The order in which components are joined is crucial to the order in which tasks are executed in the pipeline. The flow of pipeline execution proceeds from left to right, so the connection ring at the left of a component represents the input to the component, passed from the previous component, while the ring on the right represents the output of the component, which is passed to the next component.
You can only connect an output (right-hand ring) to an input (left-hand ring). You can’t connect an input to an input or an output to an output.
You can create complex, branching pipelines by linking a component’s output connection to two or more other components. Pipeline branches can be recombined into a single branch at any component that accepts two or more inputs. The component documentation states how many inputs and outputs a component allows. Because each branch could complete at a different time, you may need flow-control logic, such as an And or Or component, at the point they recombine. Most components allow a different connection to be followed depending on whether the component execution succeeds or fails. Components that perform a conditional check (for example, the If component) have different conditional connections that are followed depending on whether the condition evaluates to true or false. The possible connection types are:
  • On success: Displayed as a green line with a checkmark symbol. This connection is followed if the component execution succeeds.
  • On failure: Displayed as a red line with an X symbol. This connection is followed if the component execution fails.
  • True: Displayed as a blue line with a thumbs-up symbol. This connection is followed if the component’s condition evaluates to true.
  • False: Displayed as an orange line with a thumbs-down symbol. This connection is followed if the component’s condition evaluates to false.
  • Unconditional: Displayed as a gray line with no symbol. This connection is followed in all cases, no matter what the result of the component’s execution is.
To connect two components already on the canvas (output to input):
  1. Hover over a component to make its connection rings visible.
  2. Click and drag the right-hand connection ring to the other component. This draws a line to connect the two components.
  3. From the action bar, select the connection type you want. Other options on the action bar allow you to delete the connection, or to add a new component and connect to that instead of connecting to the one you dragged to.
To connect a new component to an existing component’s output or input:
  1. Hover over a component to make its connection rings visible.
  2. Click one of the rings, left or right, to open the Component dialog.
  3. Select the type of connection.
  4. Select the new component from the list. Use the Search field and filter buttons to locate a component.
To disconnect components, click the connection line, then either click Delete in the action bar, or press Backspace or Delete on your keyboard. To change the type of a connection (for example, On success to On failure), click the connection line, then click the icon for the connection type you want.

​
Example

A pipeline has a Salesforce Query component that takes data from Salesforce and writes it to an S3 bucket with an S3 Unload component.
  1. The pipeline starts with the Salesforce Query, so you place that on the left side of the canvas.
  2. Place an S3 Unload component somewhere to the right of Salesforce Query on the canvas—this left-to-right positioning isn’t strictly required, but will make your pipeline layouts neater and easier for you to follow.
  3. The output ring on the right-hand side of the Salesforce Query component should then be connected to the input ring at the left-hand side of the S3 Unload component, using click-and-drag.
This connection tells the pipeline which order these components are to be executed in, and that the data generated by the first component is to be made available for the second component to work with.

​
Skip components

Use the Skip on run option to temporarily disable a component in your pipeline. This allows you to run the pipeline as if it did not contain the skipped component, without actually needing to remove the component from the pipeline. To skip a component, click the component and then click Skip on run in the component action bar. The component will appear darker, and the Skip on run icon in the action bar will change to show that this component will be skipped. Additionally, when you run a pipeline containing a skipped component and then open the detailed task view, the skipped component’s success icon is grayed out and its Message notes that this component is skipped. To un-skip a component, click the component and click Unskip on run. The appearance of the component and icon will return to normal.
You can only skip components in orchestration pipelines, and you can’t skip Flow components such as If.

​
Iterate on components

Use iterators to loop a component—repeating it without adding multiple copies to your pipeline. There are two ways to attach an iterator to a component:
  • Click the component and then click Iterate in the component action bar. Next, select the iterator you want to use. The iterator will automatically be attached to the selected component on the canvas.
  • Search for and click an iterator component using the Add component option. Next, use the connection ring beneath the iterator to connect it to the input of the component you want to iterate on.
The following iterators are available in Designer:
  • Use the File Iterator to loop over matching files in a remote file system.
  • Use the Fixed Iterator to loop over values of a simple sequence.
  • Use the Grid Iterator to loop over the rows of a grid variable.
  • Use the Loop Iterator to loop over values of a simple sequence using a for loop.
  • Use the Table Iterator to loop over rows of data within a table.
For information about attaching, stacking, and detaching multiple iterators, read Attaching, stacking, and detaching iterators.

​
Delete components

To delete a component from the canvas, you can:
  • Select the component, then click Delete in the component action bar.
  • Right-click the component and click Delete on the context menu.
  • Select the component and press Delete on your keyboard.

​
Component properties

Every component has properties that can be configured to tailor the component’s behavior. When a component is selected on the canvas, its properties are displayed in the Properties panel on the right of the canvas. Each component’s properties are described in its reference page in the Components documentation. Click the Help icon in the top right of a component’s Properties panel to view its documentation. Configuring a component’s properties doesn’t affect any other components, even components of the same type. Each component’s name must be unique within a pipeline and cannot be left blank. When configuring a component with properties in sections, such as a connector, click the header of each section to expand or collapse it. You can also toggle on Expand all to display all available properties. This toggle is off by default.

​
Post-processing

If the selected component supports post-processing, click the Post-processing tab to update user-defined pipeline variables and project variables via component execution. For more information, read Post-processing.

​
Preview SQL

Most transformation components produce SQL statements that act on your data. Once a transformation component has been validated, you can toggle Preview SQL on at the top of the Properties panel to see the raw SQL of the selected component in a read-only editor.

​
Variables in properties

Many properties allow you to use project and pipeline variables or grid variables to set them dynamically. For example, any connectors that output to file storage can have the storage folder path set using variables, as described in Organizing file storage.

​
Reorder items

Many property dialogs allow you to reorder rows of information. For example, when you configure the Columns property of the Create Table component, you can reorder the columns. To do this in any applicable editor, use the ⋮⋮ icon to drag and drop a single row to change its position, or use the checkboxes to select multiple rows and then drag all selected rows into the required order. Reordering rows can have a significant effect on your data. For example, it may change the order in which your data is displayed and processed, or the order in which filter conditions are applied.

​
Text mode

Text mode is a feature that lets you rapidly add information to a property dialog. Instead of typing or selecting values in the individual fields of the dialog, you can click the Text mode toggle to open a multi-line editor that lets you add all the items in a single block, one item to a line, with the “fields” separated by tabs. For example, when you complete the Columns property of the Create Table component, you are required to specify Column Name, Data Type, Size, and so on, for every column in the table. In text mode you can type the following:
Once you have entered details in text mode, you must switch out of text mode by clicking the Text mode toggle once more, to verify the details are valid before you continue. For example, if you mistyped a data type as Numbor, the Data Type field in the dialog would reject the invalid type and remain blank. You must correct the type, either directly in the field or in text mode again, before you click Save in the dialog. You can go into text mode to edit the fields of an existing dialog, regardless of whether it was originally completed in text mode or not. Text mode also lets you paste values from other text sources, as long as the values match the columns in the dialog and are separated by tabs and new lines. You can also rapidly copy the fields from a completed dialog to another identical dialog, allowing rapid duplication of properties between or within components:
  1. Open a completed dialog in text mode.
  2. Copy the text containing the column definitions.
  3. Open another dialog, either of the same type or having identical field layout.
  4. Paste the copied text into the text mode field of the new dialog.