> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maia.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Upgrade Assistant

export const metl = "Matillion ETL";

export const maia = "Maia";

<Note>
  The Upgrade Assistant is only intended for customers who wish to convert their {metl} workloads to {maia} pipelines.
</Note>

The Upgrade Assistant is a tool for automatically converting legacy {metl} workloads to run in {maia}. The Upgrade Assistant connects from your {maia} instance to the APIs of your {metl} instance to securely retrieve the information used to configure your {metl} jobs, and uses this information to create new {maia} pipelines.

***

## Prerequisites

To run the Upgrade Assistant, you must ensure the following requirements are met:

* You must have a {metl} username and password.
* You must know the URL for your {metl} instance.
* Your {metl} user account must have the following permissions enabled:
  * Server Admin role
  * API role
  * Global Project Admin role
  * All Global Access permission group

Upgrades are performed at the project level. You must have a {maia} project to hold the recreated pipelines before beginning the upgrade, including a suitable environment and branch for the pipelines to use. We recommend that you create a new project which you can use for testing the pipelines instead of importing directly into an existing production project. Follow the normal process for [creating a project](/docs/guides/projects#add-a-new-project).

***

## Security considerations

The Upgrade Assistant reads and copies information from your {metl} instance. This information is strictly limited to the following:

* Project structure
* Jobs and components used
* Shared job details
* Schedules
* Task History
* List of JDBC drivers within the instance
* Secret references

Under no circumstances are your data or the values of secrets ever read or copied.

If you have passwords in {metl} which point to AWS Secrets Manager, a new secret reference will be created that points to the same secret. For other passwords, placeholder secret references will be created.

<Note>
  If you use Google Chrome as your browser, there's an issue with how the browser tags "Not secure" connections.

  When the Upgrade Assistant contacts a {metl} instance that has a self-signed certificate, your browser will identify it as not secure, and you will have to accept the risk and continue, using the browser's standard mechanism for this. The connection is *not* insecure, as the target is your own trusted {metl} instance, but the browser is only aware of the fact that your security certificate is self-signed and therefore (as far as the browser knows) is a potential risk.

  The issue is that the "Not secure" tag then persists in Chrome, making it appear that your entire {maia} session is insecure from that point on. It's important to be aware that your {maia} security has *not* been compromised. You may continue working securely despite the browser warning. Restarting Chrome will remove the warning.

  This issue only surfaces with the Google Chrome browser.
</Note>

***

## Run the Upgrade Assistant

The Upgrade Assistant migrates a single {metl} project. To migrate multiple projects, run the Upgrade Assistant multiple times. This allows you to choose different migration options for different projects. You can migrate each {metl} project into a different {maia} project or migrate several {metl} projects into the same {maia} project. As the upgrade is granular to the job level, you could migrate certain jobs from a project into one {maia} project, then switch to a different {maia} project and run the Upgrade Assistant there to migrate different jobs.

The Upgrade Assistant has a number of configuration steps that must be completed in order.

### Matillion ETL connection

1. In your project, click the **Upgrade Assistant** tab.

2. Configure the following properties:

   * **Base URL:** The URL of your {metl} instance. You must include `https://`.
   * **Username:** The username of a {metl} user account with sufficient permissions to read the information that you want to migrate (see [above](#prerequisites)).
   * **Password:** The password for the user account.

3. Click **Test Connection**. The Upgrade Assistant verifies that the {metl} instance is reachable and the credentials are valid. If there is a connection issue, troubleshoot it and re-test.

4. Click **Next**.

When running the Upgrade Assistant a second or subsequent time, at any point in the configuration you can click **Resume previous scan** to jump straight to the [Scan](#scan) using your previous configuration options.

Click **Previous** on any screen to return to the previous configuration screen.

### Options

1. Under **Target Project Version**, select the project you want to migrate. You can only select a single project.
2. Select the project version you want to migrate. If a project has multiple versions, we recommend that you select the version that reflects the most up-to-date snapshot of the jobs you intend to migrate.
3. Select **Scan password references** to migrate secret manager references. This option is selected by default. This does *not* read passwords or secret values.
4. Select **Scan API Extract profiles** to migrate API extract profiles. These are recreated as [Custom Connectors](/docs/guides/custom-connector-overview) in {maia}. This option is selected by default.
5. **Use legacy scanning method** enables an alternative method for retrieving jobs, by retrieving every job body during the scan instead of fetching them on demand during migration. This is disabled by default, and should not be selected unless your initial scan produces unexpected results.
6. Set the **Shared Job Download Concurrency**. This is the number of concurrent requests to the {metl} server while downloading shared jobs. The default setting is `5`, which is recommended as it should not impact {metl} performance, but you can adjust it if needed.
7. Set the **Job Download Concurrency**. This is the number of concurrent requests to the {metl} server while downloading jobs. The default setting is `5`, which is recommended as it should not impact {metl} performance, but you can adjust it if needed.
8. Click **Next**.

### Scan

1. Click **Start Scanning** to initiate a scan of the {metl} configuration with the options you selected in the previous step.
2. When the scan is complete, click **Next**.

### Migration options

1. If you use a [Hybrid SaaS](/docs/guides/runner-overview#hybrid-saas) deployment for {maia}, select the runner you want migrated pipelines to use.
2. Select the {maia} [Environment](/docs/guides/environments) that the migrated pipelines will use to connect to your cloud data warehouse. The **Environment** drop-down lists all environments in your project.
3. Select the [Artifact](/docs/guides/artifacts) that will contain the migrated pipelines. The **Artifact** drop-down contains all artifacts in the current project. For non-production projects, we recommend selecting **Always use latest** to allow schedules to update automatically as new versions are published. If no artifact versions have been published, this choice is omitted and schedules will be created using **Always use latest**.
4. Select the [Branch](/docs/guides/branches) that will contain the imported pipelines. The **Branch** drop-down lists all branches in your project.
5. Select any of the following [Job import options](#job-import-options) and [Shared pipeline options](#shared-pipeline-options) that you require.
6. Click **Next** to begin the migration.

### Job import options

The following options automate specific conversions of {metl} components to use equivalent {maia} functionality. None of these options are selected by default. Select as many of them as your project requires.

#### Convert Python 2.x to 3

{maia} supports Python 3 only. If you have a {metl} component that uses the Python 2 or Jython interpreters, you may need to rewrite your scripts to be compatible with Python 3.

The Upgrade Assistant can convert scripts automatically. Select **Convert Python 2.x to 3** to convert all scripts in the migrated project.

The Upgrade Assistant uses the `2to3` utility that's part of the standard Python library. This tool can handle many of the more common changes between Python 2 and Python 3, but you may need to make additional manual changes to the script to ensure it works as expected. Read the [Python documentation](https://docs.python.org/3.10/library/2to3.html) for more details. You should always validate the scripts independently.

After conversion, both the original and the converted versions of the scripts are stored. In the branch's [Files panel](/docs/guides/using-designer#files-panel), navigate to the `.matillion` → `migration` → `<migration date>` → `<pipeline name>` folder. There you'll find two versions of each converted script, labeled `_before` and `_after`. This allows you to compare the two versions and see what changes were made during the conversion. These `_before` and `_after` scripts are not used by the migrated pipeline component, so you can delete them both when you no longer need them for verification purposes.

<Warning>
  Matillion can't guarantee the converted script will work as expected. Always review the converted script and test it thoroughly.
</Warning>

#### Convert to Python Pushdown

[Python Pushdown](/docs/components/python-pushdown) is an orchestration component that lets you execute a Python script using the Snowpark service in your Snowflake account, benefiting from scalable Snowflake compute resources, additional library support, and database connectivity. This is a good option for resource-intensive scripts, provided your Snowflake account is scaled accordingly. This component is only available to Snowflake projects.

The Upgrade Assistant can convert {metl} Python Script components to Python Pushdown automatically. Select **Convert to Python Pushdown** to convert all components in the migrated project.

#### Convert Python Script to Script Pushdown

[Script Pushdown](/docs/components/script-pushdown) is an orchestration component that lets you run Python scripts in a Python interpreter running on a remote host, via SSH.

The Upgrade Assistant can convert {metl} Python Script components to Script Pushdown automatically. Select **Convert Python Script to Script Pushdown** to convert all components in the migrated project.

If **Convert Python 2.x to 3** is also selected, that conversion runs first so the resulting Script Pushdown component uses Python 3 code.

#### Convert Bash Script to Script Pushdown

[Script Pushdown](/docs/components/script-pushdown) is an orchestration component that lets you run Bash scripts in a Bash interpreter running on a remote host, via SSH.

The Upgrade Assistant can convert {metl} Bash Script components to Script Pushdown automatically. Select **Convert Bash Script to Script Pushdown** to convert all components in the migrated project.

#### Predict parameterized job dependencies

When a [Run Orchestration](/docs/components/run-orchestration) or [Run Transformation](/docs/components/run-transformation) component targets a name built from variables (for example, `${env}_load`), the Upgrade Assistant substitutes known variable values to predict the target jobs and include them in the dependency list for migration. Select **Predict parameterized job dependencies** to make this substitution for all components in the project.

### Shared pipeline options

These options may be required if your migrated project includes shared jobs. {metl} shared jobs are converted to {maia} shared pipelines.

#### Shared pipeline root folder

The default root folder for [shared pipelines](/docs/guides/shared-pipelines) in {maia} is `Shared Pipelines`. Enter a different root folder name if required. The folder will be created automatically.

#### Nest shared pipelines by package

By default, each shared pipeline is migrated into its own subfolder directly under the root folder.

To place each migrated shared pipeline under its {metl} package path within the root folder (for example, `<root>/<package>/<shared job>`), select **Nest shared pipelines by package**.

#### Re-migrate shared jobs that already exist in other projects

By default, a shared pipeline that already exists in this or another accessible {maia} project (matched by name) is reused and counted as already migrated.

Select **Re-migrate shared jobs that already exist in other projects** to migrate matching shared jobs again, instead of reusing the existing pipeline.

### Migrate

Click **Next** to move to the **Migrate** screen and begin the migration scan. The scan discovers what items in {metl} are available to migrate.

After the scan completes, the **Migrate** screen lists all discovered {metl} items in four tabs: **Schedules**, **Individual Jobs**, **Shared Jobs**, and **Others** (which includes API Query Profiles and project variables).

The migration of a single item will begin as soon as you click one of the **Migrate** options on these tabs; it will not wait for every item to be selected. Each item will display **Done** when the migration of that item is complete.

#### Schedules

On the **Schedules** tab, click the **+** symbol next to any schedules you want to migrate. You can select multiple schedules. Dependencies for all selected schedules will be identified and included. Dependencies include the job that the schedule is running, and can include subjobs, API Query Profiles, and shared pipelines required to run that job.

Your options are then:

* Click **Migrate** to migrate only the selected schedules, without including dependencies.
* Click **With Dependencies** to migrate the selected schedules *and* their dependencies.
* Click **Migrate All Dependencies** to migrate the dependencies *without* the schedules that depend on them.
* Click **Migrate** against specific dependencies to migrate only those dependencies.

#### Individual jobs

On the **Individual Jobs** tab, the jobs are shown in the tree view of the {metl} project. Click the **+** symbol next to any jobs you want to migrate. You can select multiple jobs. Click **+** at a higher level in the tree view to select all the jobs under that level. Dependencies for all selected jobs will be identified and included. Dependencies can include subjobs, API Query Profiles, and shared pipelines required to run the selected jobs.

Your options are then:

* Click **Migrate** to migrate only the selected jobs, without including dependencies.
* Click **With Dependencies** to migrate the selected jobs *and* their dependencies.
* Click **Migrate All Dependencies** to migrate the dependencies *without* the jobs that depend on them.
* Click **Migrate** against specific dependencies to migrate only those dependencies.

#### Shared jobs

On the **Shared Jobs** tab, click the **+** symbol next to any jobs you want to migrate. You can select multiple jobs. Dependencies for all selected jobs will be identified and included. The dependencies are the shared job's included jobs.

Your options are then:

* Click **Migrate** to migrate only the selected jobs, without including dependencies.
* Click **With Dependencies** to migrate the selected jobs *and* their dependencies.
* Click **Migrate All Dependencies** to migrate the dependencies *without* the jobs that depend on them.
* Click **Migrate** against specific dependencies to migrate only those dependencies.

#### Others

The **Others** tab includes API Query Profiles, Custom Connectors, and project variables.

* Click **Migrate** next to an individual item to migrate that item.
* Click **Migrate All** to migrate every item in the list.