Skip to main content
The Upgrade Assistant is only intended for customers who wish to convert their workloads to pipelines.
The Upgrade Assistant is a tool for automatically converting legacy workloads to run in . The Upgrade Assistant connects from your instance to the APIs of your instance to securely retrieve the information used to configure your jobs, and uses this information to create new pipelines.

​
Prerequisites

To run the Upgrade Assistant, you must ensure the following requirements are met:
  • You must have a username and password.
  • You must know the URL for your instance.
  • Your user account must have the following permissions enabled:
    • Server Admin role
    • API role
    • Global Project Admin role
    • All Global Access permission group
Upgrades are performed at the project level. You must have a project to hold the recreated pipelines before beginning the upgrade, including a suitable environment and branch for the pipelines to use. We recommend that you create a new project which you can use for testing the pipelines instead of importing directly into an existing production project. Follow the normal process for creating a project.

​
Security considerations

The Upgrade Assistant reads and copies information from your instance. This information is strictly limited to the following:
  • Project structure
  • Jobs and components used
  • Shared job details
  • Schedules
  • Task History
  • List of JDBC drivers within the instance
  • Secret references
Under no circumstances are your data or the values of secrets ever read or copied. If you have passwords in which point to AWS Secrets Manager, a new secret reference will be created that points to the same secret. For other passwords, placeholder secret references will be created.
If you use Google Chrome as your browser, there’s an issue with how the browser tags “Not secure” connections.When the Upgrade Assistant contacts a instance that has a self-signed certificate, your browser will identify it as not secure, and you will have to accept the risk and continue, using the browser’s standard mechanism for this. The connection is not insecure, as the target is your own trusted instance, but the browser is only aware of the fact that your security certificate is self-signed and therefore (as far as the browser knows) is a potential risk.The issue is that the “Not secure” tag then persists in Chrome, making it appear that your entire session is insecure from that point on. It’s important to be aware that your security has not been compromised. You may continue working securely despite the browser warning. Restarting Chrome will remove the warning.This issue only surfaces with the Google Chrome browser.

​
Run the Upgrade Assistant

The Upgrade Assistant migrates a single project. To migrate multiple projects, run the Upgrade Assistant multiple times. This allows you to choose different migration options for different projects. You can migrate each project into a different project or migrate several projects into the same project. As the upgrade is granular to the job level, you could migrate certain jobs from a project into one project, then switch to a different project and run the Upgrade Assistant there to migrate different jobs. The Upgrade Assistant has a number of configuration steps that must be completed in order.

​
Matillion ETL connection

  1. In your project, click the Upgrade Assistant tab.
  2. Configure the following properties:
    • Base URL: The URL of your instance. You must include https://.
    • Username: The username of a user account with sufficient permissions to read the information that you want to migrate (see above).
    • Password: The password for the user account.
  3. Click Test Connection. The Upgrade Assistant verifies that the instance is reachable and the credentials are valid. If there is a connection issue, troubleshoot it and re-test.
  4. Click Next.
When running the Upgrade Assistant a second or subsequent time, at any point in the configuration you can click Resume previous scan to jump straight to the Scan using your previous configuration options. Click Previous on any screen to return to the previous configuration screen.

​
Options

  1. Under Target Project Version, select the project you want to migrate. You can only select a single project.
  2. Select the project version you want to migrate. If a project has multiple versions, we recommend that you select the version that reflects the most up-to-date snapshot of the jobs you intend to migrate.
  3. Select Scan password references to migrate secret manager references. This option is selected by default. This does not read passwords or secret values.
  4. Select Scan API Extract profiles to migrate API extract profiles. These are recreated as Custom Connectors in . This option is selected by default.
  5. Use legacy scanning method enables an alternative method for retrieving jobs, by retrieving every job body during the scan instead of fetching them on demand during migration. This is disabled by default, and should not be selected unless your initial scan produces unexpected results.
  6. Set the Shared Job Download Concurrency. This is the number of concurrent requests to the server while downloading shared jobs. The default setting is 5, which is recommended as it should not impact performance, but you can adjust it if needed.
  7. Set the Job Download Concurrency. This is the number of concurrent requests to the server while downloading jobs. The default setting is 5, which is recommended as it should not impact performance, but you can adjust it if needed.
  8. Click Next.

​
Scan

  1. Click Start Scanning to initiate a scan of the configuration with the options you selected in the previous step.
  2. When the scan is complete, click Next.

​
Migration options

  1. If you use a Hybrid SaaS deployment for , select the runner you want migrated pipelines to use.
  2. Select the Environment that the migrated pipelines will use to connect to your cloud data warehouse. The Environment drop-down lists all environments in your project.
  3. Select the Artifact that will contain the migrated pipelines. The Artifact drop-down contains all artifacts in the current project. For non-production projects, we recommend selecting Always use latest to allow schedules to update automatically as new versions are published. If no artifact versions have been published, this choice is omitted and schedules will be created using Always use latest.
  4. Select the Branch that will contain the imported pipelines. The Branch drop-down lists all branches in your project.
  5. Select any of the following Job import options and Shared pipeline options that you require.
  6. Click Next to begin the migration.

​
Job import options

The following options automate specific conversions of components to use equivalent functionality. None of these options are selected by default. Select as many of them as your project requires.

​
Convert Python 2.x to 3

supports Python 3 only. If you have a component that uses the Python 2 or Jython interpreters, you may need to rewrite your scripts to be compatible with Python 3. The Upgrade Assistant can convert scripts automatically. Select Convert Python 2.x to 3 to convert all scripts in the migrated project. The Upgrade Assistant uses the 2to3 utility that’s part of the standard Python library. This tool can handle many of the more common changes between Python 2 and Python 3, but you may need to make additional manual changes to the script to ensure it works as expected. Read the Python documentation for more details. You should always validate the scripts independently. After conversion, both the original and the converted versions of the scripts are stored. In the branch’s Files panel, navigate to the .matillion → migration → <migration date> → <pipeline name> folder. There you’ll find two versions of each converted script, labeled _before and _after. This allows you to compare the two versions and see what changes were made during the conversion. These _before and _after scripts are not used by the migrated pipeline component, so you can delete them both when you no longer need them for verification purposes.
Matillion can’t guarantee the converted script will work as expected. Always review the converted script and test it thoroughly.

​
Convert to Python Pushdown

Python Pushdown is an orchestration component that lets you execute a Python script using the Snowpark service in your Snowflake account, benefiting from scalable Snowflake compute resources, additional library support, and database connectivity. This is a good option for resource-intensive scripts, provided your Snowflake account is scaled accordingly. This component is only available to Snowflake projects. The Upgrade Assistant can convert Python Script components to Python Pushdown automatically. Select Convert to Python Pushdown to convert all components in the migrated project.

​
Convert Python Script to Script Pushdown

Script Pushdown is an orchestration component that lets you run Python scripts in a Python interpreter running on a remote host, via SSH. The Upgrade Assistant can convert Python Script components to Script Pushdown automatically. Select Convert Python Script to Script Pushdown to convert all components in the migrated project. If Convert Python 2.x to 3 is also selected, that conversion runs first so the resulting Script Pushdown component uses Python 3 code.

​
Convert Bash Script to Script Pushdown

Script Pushdown is an orchestration component that lets you run Bash scripts in a Bash interpreter running on a remote host, via SSH. The Upgrade Assistant can convert Bash Script components to Script Pushdown automatically. Select Convert Bash Script to Script Pushdown to convert all components in the migrated project.

​
Predict parameterized job dependencies

When a Run Orchestration or Run Transformation component targets a name built from variables (for example, ${env}_load), the Upgrade Assistant substitutes known variable values to predict the target jobs and include them in the dependency list for migration. Select Predict parameterized job dependencies to make this substitution for all components in the project.

​
Shared pipeline options

These options may be required if your migrated project includes shared jobs. shared jobs are converted to shared pipelines.

​
Shared pipeline root folder

The default root folder for shared pipelines in is Shared Pipelines. Enter a different root folder name if required. The folder will be created automatically.

​
Nest shared pipelines by package

By default, each shared pipeline is migrated into its own subfolder directly under the root folder. To place each migrated shared pipeline under its package path within the root folder (for example, <root>/<package>/<shared job>), select Nest shared pipelines by package.

​
Re-migrate shared jobs that already exist in other projects

By default, a shared pipeline that already exists in this or another accessible project (matched by name) is reused and counted as already migrated. Select Re-migrate shared jobs that already exist in other projects to migrate matching shared jobs again, instead of reusing the existing pipeline.

​
Migrate

Click Next to move to the Migrate screen and begin the migration scan. The scan discovers what items in are available to migrate. After the scan completes, the Migrate screen lists all discovered items in four tabs: Schedules, Individual Jobs, Shared Jobs, and Others (which includes API Query Profiles and project variables). The migration of a single item will begin as soon as you click one of the Migrate options on these tabs; it will not wait for every item to be selected. Each item will display Done when the migration of that item is complete.

​
Schedules

On the Schedules tab, click the + symbol next to any schedules you want to migrate. You can select multiple schedules. Dependencies for all selected schedules will be identified and included. Dependencies include the job that the schedule is running, and can include subjobs, API Query Profiles, and shared pipelines required to run that job. Your options are then:
  • Click Migrate to migrate only the selected schedules, without including dependencies.
  • Click With Dependencies to migrate the selected schedules and their dependencies.
  • Click Migrate All Dependencies to migrate the dependencies without the schedules that depend on them.
  • Click Migrate against specific dependencies to migrate only those dependencies.

​
Individual jobs

On the Individual Jobs tab, the jobs are shown in the tree view of the project. Click the + symbol next to any jobs you want to migrate. You can select multiple jobs. Click + at a higher level in the tree view to select all the jobs under that level. Dependencies for all selected jobs will be identified and included. Dependencies can include subjobs, API Query Profiles, and shared pipelines required to run the selected jobs. Your options are then:
  • Click Migrate to migrate only the selected jobs, without including dependencies.
  • Click With Dependencies to migrate the selected jobs and their dependencies.
  • Click Migrate All Dependencies to migrate the dependencies without the jobs that depend on them.
  • Click Migrate against specific dependencies to migrate only those dependencies.

​
Shared jobs

On the Shared Jobs tab, click the + symbol next to any jobs you want to migrate. You can select multiple jobs. Dependencies for all selected jobs will be identified and included. The dependencies are the shared job’s included jobs. Your options are then:
  • Click Migrate to migrate only the selected jobs, without including dependencies.
  • Click With Dependencies to migrate the selected jobs and their dependencies.
  • Click Migrate All Dependencies to migrate the dependencies without the jobs that depend on them.
  • Click Migrate against specific dependencies to migrate only those dependencies.

​
Others

The Others tab includes API Query Profiles, Custom Connectors, and project variables.
  • Click Migrate next to an individual item to migrate that item.
  • Click Migrate All to migrate every item in the list.