Skip to main content
Script Pushdown is an orchestration component that lets you run Bash scripts and Python scripts in your pipelines. The component requires a to be installed in your cloud infrastructure to run the scripts.
Other script components are available. Choose the component that best suits your use case.
  • Use the Python Script component for small Python tasks that don’t consume a high level of resources. The executes these scripts, so large scripts can cause the to run out of memory, affecting all other pipelines that the is executing. Script Pushdown uses a dedicated and is a better option for resource-intensive scripts.
  • The Python Pushdown component lets you execute a Python script using the Snowpark service in your Snowflake account. This is a good option for resource-intensive scripts, provided your Snowflake account is scaled accordingly. If you don’t use Snowflake, this component isn’t available.
  • The Bash Pushdown component is now deprecated. Use Script Pushdown for all Bash scripts.

​
Shared script runner

Python and Bash scripts are executed by the , which is a lightweight script interpreter and SSH host supplied by Matillion and installed in your cloud infrastructure. You can choose to install the as part of a deployment. Read Shared script runner for details. Your scripts are stored within your pipeline, and at runtime the script is pushed to the via SSH so the script executes in your cloud infrastructure. The includes both Python 3 and Bash interpreters. Therefore, a single can execute either type of script. You direct the to use the correct interpreter by setting the Interpreter property on the component.
You must ensure that your cloud infrastructure is correctly sized to run the expected load imposed by your scripts. performs no checks on script performance or resource requirements.

​
Other SSH hosts

The advantage of the is that it is a pre-built SSH target that resides on the existing Hybrid SaaS infrastructure used for your , so you don’t need to manage any other infrastructure to run your scripts. However, Script Pushdown can be used with any SSH-accessible host, without the need to install a , by configuring the component’s SSH Host, User name, and Port properties appropriately.

​
Script behavior

  • Each instance of the component contains a single script, which may be either a Python script or a Bash script. Setting the Interpreter property ensures that the correct interpreter is used for the script.
  • You can pass pipeline and project variables to the script.
  • If you cancel a pipeline task while a script is running, then the script terminates.
  • Script Pushdown can only be used in a Hybrid SaaS deployment.

​
Bash scripts

  • Any output from a Bash script appears in the task completion message, and so output should be brief.
  • Any command line tools or credentials that you wish to use must be installed and set up on the instance you’re using.
  • Pipeline variables and project variables are passed to the script as Bash environment variables. Variable values can’t be returned to the pipeline from the script.
If your Bash script calls the Azure CLI and the is deployed on Azure Kubernetes Service (AKS), you must refresh the Azure CLI credentials at the start of your script. AKS deployments use Workload Identity rather than Managed Identity, so the Azure CLI session isn’t authenticated automatically. Add the following command to the beginning of your script:

​
Python scripts

  • Only Python 3 is supported.
  • Any output from a print statement in a Python script appears in the task completion message, and so output should be brief.
  • You may import any modules from the Python Standard Library. Optionally, you may also import your own Python libraries from cloud storage (Amazon S3 bucket, Azure Blob Storage container, or GCS bucket). To do this, you need to specify the location of your libraries in the environment variable EXTENSION_LIBRARY_LOCATION. For more information, read Installing Python libraries. To use these additional Python libraries, you need to include appropriate imports in your Python script, following standard Python practice.
  • Validation of the Python syntax is handled by the at runtime. If validation fails, the script doesn’t execute and the component terminates with a failure condition.
  • Pipeline and project variables can be used in the script. Note the following:
    • You can set variables within the script using regular expressions, date and numeric calculations, and interpolated values.
    • Date and time manipulation in variables (for example, datetime, timedelta, or similar) is supported, but note that datetime isn’t supported as a data type in variables.
    • Updated variables can be passed back from the script to the pipeline.
    • To learn how to manipulate variables through a Python script, read Using variables in scripts.

​
Properties

Reference material is provided below for the Connection, Authentication, and Execution details properties.
​
string
required
A human-readable name for the component.

​
Connection

​
string
required
Your SSH host.
​
string
required
Your SSH username.Run echo "$USER" to prompt your shell for your username.
​
integer
required
The length of time in milliseconds before the connection times out. The default is 3000.
​
integer
required
The port number used to connect to the SSH host. The default is set to port 22.

​
Authentication

​
drop-down
required
Select whether to use a basic password or a key pair to authenticate on the host. The default is Key Pair.
​
drop-down
required
Required when Authentication type is set to Basic Password.Use the drop-down to select the secret definition that contains the connection’s password.Read Secrets and secret definitions to learn how to create a new secret definition.
​
drop-down
required
Required when Authentication type is set to Key Pair.Use the drop-down to select the secret definition that contains the connection’s private key.Read Secrets and secret definitions to learn how to create a new secret definition.
​
boolean
required
Required when Authentication type is set to Key Pair. Select whether to require a passphrase as part of using key pair authentication. Default is No.
​
drop-down
required
Required when Authentication type is set to Key Pair and Require passphrase is set to Yes. Use the drop-down to select the secret definition that contains the connection’s passphrase.Read Secrets and secret definitions to learn how to create a new secret definition.

​
Execution details

​
drop-down
required
Select the type of script that the component runs: Bash or Python.
​
boolean
Select whether to enable variable resolution in the script. When set to Yes, variables can be referenced in the script. Default is Yes.
​
text editor
If Interpreter is Bash, enter the Bash script that the component runs.
​
text editor
If Interpreter is Python, enter the Python script that the component runs.Read Python scripts for details to consider when writing your scripts.
​
boolean
required
Select whether to terminate the script if any command fails. Only applicable if Interpreter is Bash. Default is Yes.
​
integer
The number of seconds to wait for script termination. After the set number of seconds has elapsed, the script is forcibly terminated. The default is 360 seconds (6 minutes).