Projects

A project is a group of resources that Designer uses when running your orchestration and transformation pipelines. From the Your projects menu, select a project, and then navigate between resource tabs to manage resources including:

Branches: used for collaborative working and version control using Git. Your default branch is main.
Environments: for configuring your connection to your data warehouse, and setting environment permissions for users.
Streaming: build streaming pipelines to capture the continuous data flow from supported sources in near real time and send it to your preferred cloud destination. To create and manage your streaming pipelines, read Create streaming pipeline and Manage streaming pipelines, respectively. You can also click View Features to access a dialog outlining the key capabilities of streaming pipelines in Maia. Additionally, you can reach out for support by filling out the Talk to an Expert form to contact sales.
Artifacts: view published versions of pipelines called ‘artifacts’ in each environment, and deploy artifacts to different environments.
Schedules: schedule your orchestration and transformation pipelines. All dates and times in the table use Greenwich Mean time (GMT) as its timezone. If you have set the time using a different timezone, view the individual schedule’s details for the selected timezone’s details.
Access: grant users access to your project.
Connections: create and manage secrets, OAuths, and cloud credentials.
Jobs Compute: run and schedule transformation pipelines on a Databricks Jobs Compute cluster.
Upgrade Assistant: migrate workloads from Matillion ETL to Maia.

Expand this box to watch our video explaining how to add and set up your first project.

Video: Setting up your first project with Maia

Add a new project

By default, users are assigned the Owner role for any projects they create. For more information, read Project roles. To add a new project:

Log in to Maia or register for an account.
Select your account.
In the left navigation, click .

Click Add project and complete the following parameters.

Parameter	Description
Project name	A unique name for the project. Max 255 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page.
Description	An optional, referential description for the project. Max 1000 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page.
Data platform	Use the drop-down to select your data platform: Snowflake, Databricks, Amazon Redshift, or Google BigQuery.

Some components may not be available, or may have limited functionality, on some data platforms. These limitations are noted in the relevant component documentation.

Click Continue.
Select how you would like your project to be configured.
- Maia managed (quick setup): Maia sets up and manages your Git repository, secrets, and Maia runners for you. This deployment model is the default option.
- Advanced settings (custom setup): Configure and optionally set up your own runner deployment (Full SaaS or Hybrid SaaS), and choose whether to host your Git repository with Maia or connect to a third-party Git repository. This option is suitable if you have specific requirements or constraints related to data residency, security, or network connectivity.
Choosing Advanced settings is considered a “custom” configuration option.You can choose to connect to a third-party repository but use a Matillion-hosted Maia runner, and conversely choose the Hybrid SaaS deployment model of using the Maia runner in your organization’s cloud infrastructure while using a Maia-hosted Git repository.

Some components may have limitations if using a Full SaaS project. These limitations are noted in the relevant component documentation.
Click Continue.

To migrate a project’s Git repository to a different repository or Git provider without recreating the project, use the Repository Migration API.

Select runner deployment

The following instructions only apply if you selected Advanced settings.

Select how you want to manage your Maia runner and secrets for this project:
- Full SaaS: Use a Matillion-hosted Maia runner.
- Hybrid SaaS: Host a Maia runner in your organization’s cloud infrastructure. If you choose this option, read Create a Maia runner in your infrastructure.
Click Continue.

Warehouse-specific setup

The steps to create your environment depend on the data platform you chose earlier.

Snowflake
Databricks
Amazon Redshift
Google BigQuery

Create an environment

Create an environment by providing the following information:

Parameter	Description
Environment name	A unique name for the environment. This name cannot be changed. Max 255 characters.
Runner	A working Maia runner. This is only required if you are using a Hybrid SaaS solution. To learn how to create a Maia runner, read Create a Maia runner.
Default environment access	Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles.

Maia runners can be restricted to specific projects and environments. If a Maia runner is not allowed for your project or environment, it will not appear in the Runner drop-down. For more information, read Restricting Maia runners.

Click Continue.
Specify your Snowflake credentials and select your data warehouse defaults — for details, read Environments.
Click Continue if you selected Advanced settings, or click Finish if you selected Maia managed.

Create an environment

Create an environment by providing the following information:

Parameter	Description
Environment name	A unique name for the environment. This name cannot be changed. Max 255 characters.
Runner	A working Maia runner. This is only required if you are using a Hybrid SaaS solution. To learn how to create a Maia runner, read Create a Maia runner.
Default environment access	Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles.

Click Continue.
Specify your Databricks credentials and select your data warehouse defaults — for details, read Environments.
Click Continue if you selected Advanced settings, or click Finish if you selected Maia managed.

Redshift requires AWS credentials to be entered at this stage, before you create the environment—except for Hybrid SaaS, where you can instead use the credentials already assigned to your Maia runner. This is a project-creation-only step: the equivalent doesn’t currently exist when adding an environment to an existing project later. The steps below depend on which project configuration and runner deployment you chose earlier.

Full SaaS
Hybrid SaaS

Specify AWS cloud credentials

In the Specify AWS cloud credentials dialog, complete the following parameters:
Parameter Description
Access Key ID Your AWS access key ID.
Secret Access Key Your AWS secret access key.
For details about creating an access key, read the AWS documentation.
Ensure the IAM user has appropriate permissions to read from and write to your S3 staging bucket. At a minimum, the user should have s3:GetObject, s3:PutObject, and s3:ListBucket.
Click Add.

Parameter	Description
Access Key ID	Your AWS access key ID.
Secret Access Key	Your AWS secret access key.

Create an environment

Create an environment by providing the following information:

Parameter	Description
Environment name	A unique name for the environment. This name cannot be changed. Max 255 characters.
Default environment access	Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles.

Click Continue.
Specify your Amazon Redshift credentials and select your data warehouse defaults — for details, read Environments.
Click Finish, or, if connecting a Git provider, click Continue.

Specify AWS cloud credentials

Select one of the following from the Credentials drop-down:

Use runner assigned credentials: No AWS credential is associated with your environment. Instead, your Maia runner‘s IAM role is used, and you can go straight to Create an environment, below.

Specify custom credentials: An AWS credential is created and associated with your environment. Complete the following parameters:

Parameter	Description
Credential name	Enter a name for your credentials.
Runner	A working Maia runner. This is only required if you are using a Hybrid SaaS solution. To learn how to create a Maia runner, read Create a Maia runner.
Access Key ID	Access key ID for an existing Amazon IAM user.
Secret Access Key	Secret access key for an existing Amazon IAM user.

Click Add.
Ensure the IAM user (or the Maia runner‘s IAM role, if using runner assigned credentials) has appropriate permissions to read from and write to your S3 staging bucket. At a minimum, the user should have s3:GetObject, s3:PutObject, and s3:ListBucket.

Create an environment

Create an environment by providing the following information:

Parameter	Description
Environment name	A unique name for the environment. This name cannot be changed. Max 255 characters.
Runner	A working Maia runner. This is only required if you are using a Hybrid SaaS solution. To learn how to create a Maia runner, read Create a Maia runner.
Default environment access	Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles.

Click Continue.
Specify your Amazon Redshift credentials and select your data warehouse defaults — for details, read Environments.
Click Continue to proceed to select your Git provider.

BigQuery requires GCP credentials to be entered explicitly at this stage—except for Hybrid SaaS, where you can instead use the credentials already assigned to your Maia runner. The steps below depend on which project configuration and runner deployment you chose earlier.

Full SaaS
Hybrid SaaS

Specify GCP cloud credentials

Paste the entire contents of your service account JSON key file, to access Google BigQuery and Google Cloud Storage. For more information, read Create and delete service account keys. An example of a service account key is displayed below:

{
  "type": "service_account",
  "project_id": "a-project-id",
  "private_key_id": "123456",
  "private_key": "the private key",
  "client_email": "service-account-name@a-project-id.iam.gserviceaccount.com",
  "client_id": "123456",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-name%a-project-id.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

Click Add.

Create an environment

Create an environment by providing the following information:

Parameter	Description
Environment name	A unique name for the environment. This name cannot be changed. Max 255 characters.
Default environment access	Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles.

Click Continue.
Select your BigQuery defaults — for details, read Environments.
If you use a Matillion Full SaaS solution, the cloud credentials associated with your environment will be used to access your Google Cloud resources, including BigQuery and any associated GCS buckets. To overwrite this role, associate different cloud credentials with this environment after you have finished creating it. You can create these credentials before or after creating the environment.
Click Finish, or, if connecting a Git provider, click Continue.

Specify GCP cloud credentials

Select one of the following from the Credentials drop-down:

Use runner assigned credentials: No Google Cloud credential is associated with your environment. Instead, application default credentials are used, and you can go straight to Create an environment, below.

Specify custom credentials: A Google Cloud credential is created and associated with your environment. Complete the following parameters:

Parameter	Description
Credentials	Select Specify custom credentials.
Credential name	Enter a name for your credentials.
Runner	A working Maia runner. This is only required if you are using a Hybrid SaaS solution. To learn how to create a Maia runner, read Create a Maia runner.
GCP Project ID	The unique, user-defined string identifier for your Google Cloud project. You can find the project ID in the Google Cloud Console on the dashboard.
Service account key	Your Google Cloud service account JSON key file.

Click Add.

Create an environment

Create an environment by providing the following information:

Parameter	Description
Environment name	A unique name for the environment. This name cannot be changed. Max 255 characters.
Runner	A working Maia runner. This is only required if you are using a Hybrid SaaS solution. To learn how to create a Maia runner, read Create a Maia runner.
Default environment access	Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles.

Click Continue.
Select your BigQuery defaults — for details, read Environments.
If you use a Hybrid SaaS solution, your new environment will inherit the Maia runner‘s execution role (service account role) to access your Google Cloud resources, including BigQuery and any associated GCS buckets. To overwrite this role, associate different cloud credentials with this environment after you have finished creating it. You can create these credentials before or after creating the environment.
Click Continue to proceed to select your Git provider.

Select Git provider

The following instructions only apply if you selected Advanced settings. To use a third-party Git provider, you must install the Maia app for that provider — see the relevant prerequisites below.

Select your Git provider.
1. Maia Hosted Git: Maia will create and host the repository for your project.
2. GitHub: Connect to one of your organization’s GitHub repositories. Read the following docs to get started:
  1. Prerequisites for connecting a GitHub repository to Maia
  2. Installing the Maia app in GitHub
  If you log in to GitHub using SSO, you are required to click Authorize and not Continue to authenticate to matillion-apps. Clicking Continue won’t complete the authorization properly, and you won’t be able to perform remote actions. To reauthorize, follow the steps documented in Permission denied to access GitHub repository.
3. Azure DevOps: Connect to one of your organization’s Azure DevOps repositories. Read the following docs to get started:
  1. Prerequisites for connecting an Azure DevOps repository to Maia
  2. Installing the Maia app in Azure
4. GitLab: Connect to one of your organization’s GitLab repositories. Read the following guide to get started:
  1. Prerequisites for connecting a GitLab repository to Maia
5. Bitbucket: Connect to one of your organization’s Bitbucket repositories. Read the following guide to get started:
  1. Prerequisites for connecting to a Bitbucket repository
Click Continue to trigger the authorization flow, if not already authorized. A new browser tab will open.
Click Authorize Maia. The browser tab will close and return you to the project flow.
For GitHub only. On successful authorization, you will see a dialog with a list of your authorized accounts. Click Continue.
Once you’ve authorized with your Git provider, you’ll be taken to the Select repository page, where you’ll find the list of repositories you have granted access to Maia. You can connect either an existing repository with multiple branches and a large history, or a brand new repository.
You can only choose a repository that already exists in your organization or project. Maia does not create a new repository in your account.
Select one of the repositories from the list. You can use the search bar to refine your search—for example, if the list contains many repositories.
Click Finish.

You will be redirected to your new project’s menu. From here, you can view your:

You can edit the project name and description at any time by clicking the pencil buttons next to those items at the top of the screen.

Delete a project

Before deleting a project, make sure any active schedules are disabled. If you have any active schedules when you try to delete a project, you’ll receive an error message. To delete a project, follow these steps:

In the Your projects page, click the three dots … next to the intended project.
Click Delete project.
The Delete project dialog will be displayed. Type the name of the project you want to delete in the Enter project name field.
To confirm, click Delete Project.

If you create another project with the same name as a deleted project, make sure the new project’s environment name is a different name to the name of the deleted project’s environment.

Project-level sampling

If necessary, you can enable and disable sampling at the project level. This is useful if your data contains personal information that cannot be viewed outside your region. Only users with project admin permissions can change this setting. To enable or disable sampling for a project:

In the Your projects page, click the three dots … next to the intended project.
Click Enable sampling or Disable sampling as required.

Platform overview

Setup

Getting started

Custom connectors

Runners

Observability

Git

Connections and credentials

Data ops

Streaming

Videos

Projects

Add a new project

Select runner deployment

Warehouse-specific setup

Create an environment

Create an environment

Specify AWS cloud credentials

Create an environment

Specify AWS cloud credentials

Create an environment

Specify GCP cloud credentials

Create an environment

Specify GCP cloud credentials

Create an environment

Select Git provider

Delete a project

Project-level sampling

​Add a new project

​Select runner deployment

​Warehouse-specific setup

​Create an environment

​Create an environment

​Specify AWS cloud credentials

​Create an environment

​Specify AWS cloud credentials

​Create an environment

​Specify GCP cloud credentials

​Create an environment

​Specify GCP cloud credentials

​Create an environment

​Select Git provider

​Delete a project

​Project-level sampling

Add a new project

Select runner deployment

Warehouse-specific setup

Create an environment

Create an environment

Specify AWS cloud credentials

Create an environment

Specify AWS cloud credentials

Create an environment

Specify GCP cloud credentials

Create an environment

Specify GCP cloud credentials

Create an environment

Select Git provider

Delete a project

Project-level sampling