Skip to main content
A project is a group of resources that uses when running your orchestration and transformation pipelines. From the Your projects menu, select a project, and then navigate between resource tabs to manage resources including:
  • Branches: used for collaborative working and version control using Git. Your default branch is main.
  • Google BigQuery Environments: for configuring your connection to your data warehouse, and setting environment permissions for users.
  • Streaming: build streaming pipelines to capture the continuous data flow from supported sources in near real time and send it to your preferred cloud destination. To create and manage your streaming pipelines, read Create streaming pipeline and Manage streaming pipelines, respectively. You can also click View Features to access a dialog outlining the key capabilities of streaming pipelines in . Additionally, you can reach out for support by filling out the Talk to an Expert form to contact sales.
  • Artifacts: view published versions of pipelines called ‘artifacts’ in each environment, and deploy artifacts to different environments.
  • Schedules: schedule your orchestration and transformation pipelines. All dates and times in the table use Greenwich Mean time (GMT) as its timezone. If you have set the time using a different timezone, view the individual schedule’s details for the selected timezone’s details.
  • Access: grant users access to your project.
  • Secrets and secret definitions: define secrets stored within your secret manager tool so you can reuse them across projects. Supports AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager.
  • Cloud credentials: set up cloud provider credentials to authenticate with your chosen cloud provider.
  • OAuth: create an OAuth entry to connect to your chosen third-party service.

​
Add a new project

By default, users are assigned the Owner role for any projects they create. For more information, read Project roles. To add a new project:
  1. Log in to or register for an account.
  2. Select your account.
  3. In the left navigation, click .
  4. Click Add project, select Build, and complete the following parameters.
    ParameterDescription
    Project nameA unique name for the project. Max 255 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page.
    DescriptionAn optional, referential description for the project. Max 1000 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page.
    Data platformThe cloud data platform to connect to. Use the drop-down to select Google BigQuery.
    Some components may not be available, or may have limited functionality, on some data platforms. These limitations are noted in the relevant component documentation.
  5. Click Continue.
  6. Choose how you would like your project to be configured.
    • managed: A full software-as-a-service (SaaS) deployment model where Matillion handles the infrastructure, maintenance, and upgrades. You can access through a web browser and focus on building data transformation and orchestration workflows without worrying about the underlying infrastructure. This deployment model is the default option for Enterprise and non-enterprise customers.
    • Advanced settings: A Hybrid SaaS deployment model that lets you manually deploy and manage s, connect to third-party repositories, and store your secrets within your own cloud infrastructure, whilst building data transformation and orchestration workflows within . It combines the flexibility of cloud deployment with the control and security of managing s, repositories, and secrets within your environment. This option is suitable if you have specific requirements or constraints related to data residency, security, or network connectivity.
    Choosing Advanced settings is considered a “custom” configuration option.You can choose to connect to a third-party repository but use a Matillion-hosted , and conversely choose the Hybrid SaaS deployment model of using the in your organization’s cloud infrastructure while using a Matillion hosted Git repository.
    Some components may have limitations if using a Full SaaS project. These limitations are noted in the relevant component documentation.
  7. Click Continue.
Once you connect a project to a Git repository (GitHub, GitLab, Azure DevOps, or Bitbucket), you cannot disconnect or change the Git repository connection.You must create a new project from scratch if you need to:
  • Disconnect from your current Git repository.
  • Connect to a different Git repository.
  • Switch between different repo host platforms (e.g., from GitHub to GitLab).
Reusing an existing Matillion ETL Git repository isn’t supported and can cause issues with your pipelines.

​
Select Runner deployment

The following instructions only apply if you selected Advanced settings. Select how you want to manage your and secrets for this project:
  • Full SaaS: Use a Matillion-hosted to manage my Runner and secrets.
  • Hybrid SaaS: Host a in your organization’s cloud infrastructure. If you choose this option, read Create a in your infrastructure.
Click Continue.

​
Specify GCP cloud credentials

Paste the entire contents of your service account JSON key file, to access Google BigQuery and Google Cloud Storage. For more information, read Create and delete service account keys. An example of a service account key is displayed below:
{
  "type": "service_account",
  "project_id": "a-project-id",
  "private_key_id": "123456",
  "private_key": "the private key",
  "client_email": "service-account-name@a-project-id.iam.gserviceaccount.com",
  "client_id": "123456",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-name%a-project-id.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}
Click Add.

​
Create an environment

You are required to create an environment for this project.
ParameterDescription
Environment nameA unique name for the environment. Max 255 characters.
Default environment accessUse the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles.
Click Continue.

​
Select BigQuery defaults

ParameterDescription
GCP Project IDThe unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation.
DatasetThe BigQuery dataset for this environment connection. For more information, read the BigQuery documentation.
Allow inherit project defaultsUse this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments.
Click Finish.You will be redirected to your new project’s menu. From here, you can view your:You can edit the project name and description at any time by clicking the pencil buttons next to those items at the top of the screen.

​
Delete a project

Prior to deleting a project, make sure any active schedules are disabled. If you have any active schedules when you try to delete a project, you’ll receive an error message. To delete a project, follow these steps:
  1. In the Your projects page, click the three dots … next to the intended project.
  2. Click Delete project.
  3. The Delete project dialog will be displayed. Type the name of the project you want to delete in the Enter project name field.
  4. To confirm, click Delete Project.
If you create another project with the same name as a deleted project, make sure the new project’s environment name is a different name to the name of the deleted project’s environment.

​
Project-level sampling

If necessary, you can enable and disable sampling at the project level. This is useful if your data contains personal information that cannot be viewed outside your region. Only users with project admin permissions can change this setting. To enable or disable sampling for a project:
  1. In the Your projects page, click the three dots … next to the intended project.
  2. Click Enable sampling or Disable sampling as required.