Skip to main content
A variable lets you pass dynamic values into a component’s properties, without you needing to manually edit a property each time the pipeline runs. There are two classes of variable: pipeline variables and project variables. Each type has a different scope, as defined in Variable scope. Grid variables are a special type of pipeline variable. A grid variable is a two-dimensional array that holds multiple values in named columns. This variable type is fully explained in Grid variables. Example uses of project or pipline variables:
  • Create a variable called “table_name”, and set the Create Table component to use this variable for its New Table Name property. Each time you run the pipeline containing “table_name”, you can set a new value for the variable, allowing you to re-use the same pipeline multiple times to create tables with different names, without editing the component’s properties.
  • If you have multiple components in a pipeline that all reference the same table, you can use a variable to represent the table name. To reference a different table throughout the pipeline, you only need to change the value of the variable in one place, without needing to edit the properties of each separate component.
Variables can also be accessed and set or re-set through Python scripts, giving you great flexibility in designing pipelines that are configured dynamically at runtime.

​
Variable scope

The scope of a project variable is the entire project in which it is created. That is, any pipeline in any branch of the current project can use the same variable. A project variable can be given a different default value in different project environments. The scope of a pipeline variable is the single pipeline in which it is created. No other pipeline in the project will be able to see the variable, though an identically named variable could be created in other pipelines. A pipeline variable can be given the same name as a project variable. Where this happens, only the pipeline variable will be used within that specific pipeline, not the project variable. For avoidance of confusion, we recommend that you don’t use the same names for different variables.

​
Using variables

Variables can be used to provide property values in many components. The documentation for each component will tell you whether it supports the use of variables in properties. To use a variable in a string property, type the name of the variable surrounded by brackets and prefixed by the dollar symbol, as follows: ${variable}. Once you type ${, a drop-down of autocompleted suggested variables will appear. This list updates as you type; for example, if you type ${name, only variables containing name will be listed. This list also shows whether a variable is a project variable or pipeline variable, and its type. For example, if you have defined a variable called “table_name”, which will specify the name of a new table to be created by the Create Table component in your pipeline, you will enter ${table_name} for the component’s New Table Name property. To use a variable in a property with a drop-down of options, the drop-down will include all currently defined variables for you to choose from. When a pipeline run begins, all of its variables are initialized with the default value that was specified when the variable was created. Variable values can be assigned or reassigned dynamically while the pipeline runs, typically through the use of Python scripts or the Update Scalar component. Values set in this way are valid only for the current run of the pipeline, and for the next run of the pipeline the variable will revert to its default value, or a new value must be assigned. You can query the current value of a variable at any point in a pipeline by using a Print Variables component. Variables can be used in file path parameters when addressing a storage bucket or data lake from a pipeline component. For details of this feature, read Organizing file storage

​
Double variable resolution

You can reference a variable as the value of another variable, up to one level. This is useful when creating dynamic values, such as when calling parameterized pipelines. For example, if a pipeline includes a variable called ${schema_name}, you can set its value to myschema_${sysvar.environment.name} so that it changes dynamically based on the environment. When the variable ${schema_name} is resolved, its value might be myschema_PRODUCTION.
Double variable resolution supports up to one level of nesting.

​
Creating project and pipeline variables

These instructions tell you how to create project and pipeline variables. For instructions on creating grid variables, read Creating grid variables.
Your project role determines how you can work with project and pipeline variables within your project:
  • Owners can create, edit, and delete project and pipeline variables.
  • Contributors can create project variables, but cannot edit or delete them once they have been created. Contributors can create, edit, and delete pipeline variables with no limitations.
  • Viewers can see project and pipeline variables, but cannot create, edit, or delete them.
  1. Open the Variables panel at the bottom of the pipeline canvas.
  2. Select Project or Pipeline in the top left of the panel.
  3. Click Add in the top right of the panel, then select the variable type: Text, Number, or Grid. For more information, read Variable types.
    • If you’re creating a Text or Number variable, continue following these steps.
    • If you’re creating a Grid variable, follow the steps in Creating grid variables.
  4. In the new row containing your variable, complete the following fields:
    • Name: A name for the variable.
      Every variable must have a unique name within its scope. This means that all project variables must have unique names within the project, and all pipeline variables must have unique names within a pipeline, although a pipeline variable can have the same name in a different pipeline. The name must follow these rules:
      • Names must only include letters, digits, and underscores.
      • Names must not begin with a digit. For example, my_table_02 and _02_my_table are valid, but 02_my_table is not.
      • Names must not be a JavaScript reserved word (e.g. var or const).
      • Names of variables used in Python or Bash scripts must not be a reserved word in those scripts.
    • Behavior: Select Shared or Copied. For more information, read Variable behavior. The default behavior is Shared.
    • Visibility: Only for pipeline variables. Select Public or Private. For more information, read Variable visibility.
    • Default: The value initially assigned to a variable when a pipeline runs. Can be left blank during initial creation. The variable’s value can be reset by components as the pipeline runs, but it will always begin each new pipeline run with its default value. There’s no maximum length for scalar variable values. For project variables, this can be superseded by a different default value in a specific environment using environment overrides.
    • Environment overrides: Only for project variables. Click in this field to open the Environment overrides table for the selected project variable. This displays a table of all environments and the project variable’s default value in each environment. When an environment override is set, the value shown in the Default column here behaves as the project variable’s default value in that environment.
      • If no environment override is set, the Default column for that environment shows [Default value].
      • To remove an environment override, click the Delete icon in the corresponding row. This restores the project variable’s default value for that environment.
    • Description: A description to inform other users of the purpose of the variable.

​
Managing project and pipeline variables

Only users with the Owner project role can edit and delete project variables. Users with the Owner or Contributor role can edit and delete pipeline variables.
Changes to variables take effect immediately. Deleted variables cannot be restored.
  • When editing a variable, make sure that all pipelines using this variable will continue to run as intended with the new variable configuration.
  • When deleting a variable, make sure that it is not in use in any pipelines. Deleting a variable that is referenced in a pipeline will cause the pipeline to fail.
To edit or delete a variable:
  1. Open the Variables panel at the bottom of the pipeline canvas.
  2. Find the variable you want to edit or delete. Click the search icon in the top right of the panel to open the Search variables field, where you can search for variables by name.
  3. Manage your variables:
    • To edit a variable, edit the values shown in any of the fields as described above.
    • To delete a variable, hover over the variable and click the Delete icon that appears on the right in its row. Then click Delete in the confirmation dialog. This action cannot be undone.

​
Export and import variables

To reuse configurations across different workflows, you can export and import both pipeline and project variables across different pipelines and projects. This portability allows you to maintain consistent variable naming conventions and default schemas when migrating or duplicating pipelines without manually recreating each variable entry.

​
Exporting variables

  1. Open the Variables panel at the bottom of the .
  2. Click the ⋮ icon on the top-right of the panel.
  3. Select the Export option from the list.
  4. Select the Pipeline variables that you want to export and then click Continue.
  5. Now, choose the Project variables and click Export.
  6. Your selected variables will be downloaded in a JSON file.

​
Importing variables

  1. Open the target pipeline or project where you want to apply the variables.
  2. Open the Variables panel and click the ⋮ icon on the top-right of the panel.
  3. Select the Import option from the list.
  4. Choose your exported variables configuration file.
  5. Review the variables that will be imported and click Import in the bottom-right corner.
Importing variables with names identical to those already defined in your current pipeline or project will overwrite the existing variable values. Review your variable names before proceeding with an import.

​
Variable type

The Type assigned to a variable determines what kind of information it contains. Variables can have one of the following types:
TypeDescription
TextAny text string.
NumberAny number, with or without decimals.
GridA two-dimensional array that holds multiple values in named columns, and can only be a pipeline variable. Grid variables differ from project and pipeline variables, and are documented separately in Grid variables.
It is important that the variable type matches the data type of the property you want to use the variable in. For example, the New Table Name property is type string, so it requires a Text variable type. Timeout is type integer, so it requires a Numeric variable type. Note that while the value of a Text variable might contain digits, for example 1234, its type is still Text, and so it can’t be used where a Numeric type is required.

​
Variable visibility

Setting the correct visibility for a pipeline variable is important when you want to call a pipeline from another pipeline, using the Run Orchestration or Run Transformation components. A Private pipeline variable is only visible to the pipeline it is defined in. If the pipeline is called from another pipeline, the calling pipeline can’t “see” the Private variable and so can’t use its value, reset its value, or otherwise interact with it in any way. A Public pipeline variable is visible outside the pipeline it is defined in, so it can be “seen” and used by any pipeline that calls the pipeline where it’s set. This only applies to pipeline variables. Project variables are automatically visible to every pipeline within the project.

​
Variable behavior (copied or shared)

This refers to the “branching behavior” of a variable inside a pipeline. A “branch” in this context means a divergence of connectors within the pipeline, for example following the result from an If component, giving the pipeline a branched structure with both (or all) branches of the pipeline running in parallel. This behavior applies to both project variables and pipeline variables. Because a variable may be used in all branches of the pipeline, you need to specify how it will behave when updated within any one branch. There are two behavior options, Copied and Shared.
  • Copied variables can be updated within one branch of the pipeline without updating the same variable in other branches. Effectively, each branch has its own copy of the variable, which is unaffected by changes to the copies used by any other branch. If the branches later rejoin, for example through the use of an And or Or component, the variable will revert to its default value, regardless of any updates made in either branch.
  • Shared variables are updated in a pipeline-wide fashion. If a shared variable value is updated in one branch, all other branches will use that updated value from that point in time onwards.

​
Using variables in scripts

​
Python scripts

Variables are visible to Python scripts created in the Python Script component, and in those scripts they will act as any other Python variable. For example, to show the value of a pipeline variable myVar you can use the following Python command: 
print (myVar)
From the Python script, you can use Python’s context object to assign a new value to the variable. For example: 
context.updateVariable ('myVar', 'a_new_value')
This new value for the variable will now be used in all subsequent components as the pipeline continues to run. This gives you a powerful tool for creating complex yet flexible pipelines, as some very simple Python scripts can dynamically change the behavior of the pipeline at runtime by manipulating variable values. Note that you can manipulate the variable in any desired way within the Python script, but the value of the variable within the pipeline itself won’t change unless you pass it back with the context object.
Python will allow you to change the type of a variable, for example by assigning a string to a variable that previously held an integer. When you intend to pass the variable back to the pipeline using the context object, you must take care that you do not change the variable type in your Python script. Doing so will have consequences in your pipeline and will likely cause it to fail.
A full discussion of Python scripting is beyond the scope of this article. Refer to any good Python reference for further information.

​
Bash scripts

Variables are visible to Bash scripts created in the Bash Pushdown component. Variables are read-only in Bash scripts; their values can’t be updated and passed back to the pipeline. To use a variable in a Bash script, prefix the variable name with $, as follows: 
echo $myVar

​
Passing variables between pipelines

Variables can be used to pass values between pipelines when you call one pipeline from another using the Run Orchestration and Run Transformation components. You can reset the value of the variable when it’s passed to the child by using the Set Scalar Variables property of the Run Orchestration/Transformation component. You can also dynamically reset the value through a component running in the child pipeline. When assigning a new value to a variable passed from parent to child, ensure that the new value is of the correct type.