Skip to main content
Script Pushdown is an orchestration component that lets you run Bash scripts and Python scripts in your pipelines. The component requires a to be installed in your cloud infrastructure to run the scripts.
Other script components are available. Choose the component that best suits your use case.
  • Use the Python Script component for small Python tasks that don’t consume a high level of resources. The executes these scripts, so large scripts can cause the to run out of memory, affecting all other pipelines that the is executing. Script Pushdown uses a dedicated and is a better option for resource-intensive scripts.
  • The Python Pushdown component lets you execute a Python script using the Snowpark service in your Snowflake account. This is a good option for resource-intensive scripts, provided your Snowflake account is scaled accordingly. If you don’t use Snowflake, this component isn’t available.
  • The Bash Pushdown component is now deprecated. Use Script Pushdown for all Bash scripts.

​
Shared script runner

Python and Bash scripts are executed by the , which is a lightweight script interpreter supplied by Matillion and installed in your cloud infrastructure. You can choose to install the as part of a deployment. Read Shared script runner for details. Your scripts are stored within your pipeline, and at runtime the script is pushed to the via SSH so the script executes in your cloud infrastructure. The includes both Python3 and Bash interpreters. Therefore, a single can execute either type of script. You direct the to use the correct interpreter by setting the Interpreter property on the component.
You must ensure that your cloud infrastructure is correctly sized to run the expected load imposed by your scripts. performs no checks on script performance or resource requirements.

​
Script behavior

  • Each instance of the component contains a single script, which may be either a Python script or a Bash script. Setting the Interpreter property ensures that the correct interpreter is used for the script.
  • You can pass pipeline and project variables to the script.
  • If you cancel a pipeline task while a script is running, then the script terminates.
  • Script Pushdown can only be used in a Hybrid SaaS deployment.

​
Bash scripts

  • Any output from a Bash script appears in the task completion message, and so output should be brief.
  • Any command line tools or credentials that you wish to use must be installed and set up on the instance you’re using.
  • Pipeline variables and project variables are passed to the script as Bash environment variables. Variable values can’t be returned to the pipeline from the script.

​
Python scripts

  • Only Python3 is supported.
  • Any output from a print statement in a Python script appears in the task completion message, and so output should be brief.
  • You may import any modules from the Python Standard Library. Optionally, you may also import your own Python libraries from cloud storage (Amazon S3 bucket, Azure Blob Storage container, or GCS bucket). To do this, you need to specify the location of your libraries in the environment variable EXTENSION_LIBRARY_LOCATION. For more information on this, read Installing Python libraries. To use these additional Python libraries, you will need to include appropriate imports in your Python script, following standard Python practice.
  • Validation of the Python syntax is handled by the at runtime. If validation fails, the script doesn’t execute and the component terminates with a failure condition.
  • Pipeline and project variables can be used in the script. Note the following:
    • You can set variables within the script using regular expressions, date and numeric calculations, and interpolated values.
    • Date and time manipulation in variables (for example, datetime, timedelta, or similar) is supported, but note that datetime isn’t supported as a data type in variables.
    • Updated variables can be passed back from the script to the pipeline.
    • To learn how to manipulate variables through a Python script, read Using variables in scripts.

​
Properties

Reference material is provided below for the Connection, Authentication, and Execution details properties.
​
Name
string
required
A human-readable name for the component.

​
Connection

​
Host
string
required
Your SSH host.
​
User name
string
required
Your SSH username.Run echo "$USER" to prompt your shell for your username.
​
Connection timeout (ms)
integer
required
The length of time in milliseconds before the connection times out. The default is 3000.
​
Port
integer
required
The port number used to connect to the SSH host. The default is set to port 22.

​
Authentication

​
Authentication type
drop-down
required
Select whether to use a basic password or a key pair to authenticate on the host. The default is Key Pair.
​
Password
drop-down
required
Required when Authentication type is set to Basic Password.Use the drop-down to select the secret definition that contains the connection’s password.Read Secrets and secret definitions to learn how to create a new secret definition.
​
Private key
drop-down
required
Required when Authentication type is set to Key Pair.Use the drop-down to select the secret definition that contains the connection’s private key.Read Secrets and secret definitions to learn how to create a new secret definition.
​
Require passphrase
boolean
required
Required when Authentication type is set to Key Pair. Select whether to require a passphrase as part of using key pair authentication. Default is No.
​
Passphrase
drop-down
required
Required when Authentication type is set to Key Pair and Require passphrase is set to Yes. Use the drop-down to select the secret definition that contains the connection’s passphrase.Read Secrets and secret definitions to learn how to create a new secret definition.

​
Execution details

​
Interpreter
drop-down
required
Select the type of script that the component runs: Bash or Python.
​
Allow inline variable replacement
boolean
Select whether to enable variable resolution in the script. When set to Yes, variables can be referenced in the script. Default is Yes.
​
Bash script
text editor
If Interpreter is Bash, enter the Bash script that the component runs.
​
Python script
text editor
If Interpreter is Python, enter the Python script that the component runs.
​
Exit script on failure
boolean
required
Select whether to terminate the script if any command fails. Only applicable if Interpreter is Bash. Default is Yes.
​
Script timeout (s)
integer
The number of seconds to wait for script termination. After the set number of seconds has elapsed, the script is forcibly terminated. The default is 360 seconds (6 minutes).