Google BigQuery projects

A project is a group of resources that uses when running your orchestration and transformation pipelines. From the Your projects menu, select a project, and then navigate between resource tabs to manage resources including:

Branches: used for collaborative working and version control using Git. Your default branch is main.
Google BigQuery Environments: for configuring your connection to your data warehouse, and setting environment permissions for users.
Streaming: build streaming pipelines to capture the continuous data flow from supported sources in near real time and send it to your preferred cloud destination. To create and manage your streaming pipelines, read Create streaming pipeline and Manage streaming pipelines, respectively. You can also click View Features to access a dialog outlining the key capabilities of streaming pipelines in . Additionally, you can reach out for support by filling out the Talk to an Expert form to contact sales.
Artifacts: view published versions of pipelines called ‘artifacts’ in each environment, and deploy artifacts to different environments.
Schedules: schedule your orchestration and transformation pipelines. All dates and times in the table use Greenwich Mean time (GMT) as its timezone. If you have set the time using a different timezone, view the individual schedule’s details for the selected timezone’s details.
Access: grant users access to your project.
Secrets and secret definitions: define secrets stored within your secret manager tool so you can reuse them across projects. Supports AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager.
Cloud credentials: set up cloud provider credentials to authenticate with your chosen cloud provider.
OAuth: create an OAuth entry to connect to your chosen third-party service.

Add a new project

By default, users are assigned the Owner role for any projects they create. For more information, read Project roles. To add a new project:

Log in to or register for an account.
Select your account.
In the left navigation, click .

Click Add project, select Build, and complete the following parameters.

Parameter	Description
Project name	A unique name for the project. Max 255 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page.
Description	An optional, referential description for the project. Max 1000 characters. You can edit this after creation by clicking the edit button (pencil icon) within the project page.
Data platform	The cloud data platform to connect to. Use the drop-down to select Google BigQuery.

Some components may not be available, or may have limited functionality, on some data platforms. These limitations are noted in the relevant component documentation.

Click Continue.
Choose how you would like your project to be configured.
- managed: A full software-as-a-service (SaaS) deployment model where Matillion handles the infrastructure, maintenance, and upgrades. You can access through a web browser and focus on building data transformation and orchestration workflows without worrying about the underlying infrastructure. This deployment model is the default option for Enterprise and non-enterprise customers.
- Advanced settings: A Hybrid SaaS deployment model that lets you manually deploy and manage s, connect to third-party repositories, and store your secrets within your own cloud infrastructure, whilst building data transformation and orchestration workflows within . It combines the flexibility of cloud deployment with the control and security of managing s, repositories, and secrets within your environment. This option is suitable if you have specific requirements or constraints related to data residency, security, or network connectivity.
Choosing Advanced settings is considered a “custom” configuration option.You can choose to connect to a third-party repository but use a Matillion-hosted , and conversely choose the Hybrid SaaS deployment model of using the in your organization’s cloud infrastructure while using a Matillion hosted Git repository.

Some components may have limitations if using a Full SaaS project. These limitations are noted in the relevant component documentation.
Click Continue.

To migrate a project’s Git repository to a different repository or Git provider without recreating the project, use the Repository Migration API.

Select Runner deployment

The following instructions only apply if you selected Advanced settings. Select how you want to manage your and secrets for this project:

Full SaaS: Use a Matillion-hosted to manage my Runner and secrets.
Hybrid SaaS: Host a in your organization’s cloud infrastructure. If you choose this option, read Create a in your infrastructure.

Click Continue.

Maia managed
Full SaaS
Hybrid SaaS

Specify GCP cloud credentials

Paste the entire contents of your service account JSON key file, to access Google BigQuery and Google Cloud Storage. For more information, read Create and delete service account keys. An example of a service account key is displayed below:

{
  "type": "service_account",
  "project_id": "a-project-id",
  "private_key_id": "123456",
  "private_key": "the private key",
  "client_email": "service-account-name@a-project-id.iam.gserviceaccount.com",
  "client_id": "123456",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-name%a-project-id.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

Click Add.

Create an environment

You are required to create an environment for this project.

Parameter	Description
Environment name	A unique name for the environment. Max 255 characters.
Default environment access	Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles.

Click Continue.

Select BigQuery defaults

Parameter	Description
GCP Project ID	The unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation.
Dataset	The BigQuery dataset for this environment connection. For more information, read the BigQuery documentation.
Allow inherit project defaults	Use this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments.

Click Finish.You will be redirected to your new project’s menu. From here, you can view your:

You can edit the project name and description at any time by clicking the pencil buttons next to those items at the top of the screen.

Specify GCP cloud credentials

{
  "type": "service_account",
  "project_id": "a-project-id",
  "private_key_id": "123456",
  "private_key": "the private key",
  "client_email": "service-account-name@a-project-id.iam.gserviceaccount.com",
  "client_id": "123456",
  "auth_uri": "https://accounts.google.com/o/oauth2/auth",
  "token_uri": "https://oauth2.googleapis.com/token",
  "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
  "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/service-account-name%a-project-id.iam.gserviceaccount.com",
  "universe_domain": "googleapis.com"
}

Click Add.

Create an environment

You are required to create an environment for this project.

Parameter	Description
Environment name	A unique name for the environment. Max 255 characters.
Runner	A working . This is only required if you are using a Hybrid SaaS solution. To learn how to create a , read Create a .
Default environment access	Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles.

s can be restricted to specific projects and environments. If a is not allowed for your project or environment, it will not appear in the Runner drop-down. For more information, read Restricting s.

Click Continue.

Select BigQuery defaults

Parameter	Description
GCP Project ID	The unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation.
Dataset	The BigQuery dataset for this environment connection. For more information, read the BigQuery documentation.
Allow inherit project defaults	Use this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments.

If you use a Matillion Full SaaS solution, the cloud credentials associated with your environment will be used to access your Google Cloud resources, including BigQuery and any associated GCS buckets.To overwrite this role, associate different cloud credentials with this environment after you have finished creating it. You can create these credentials before or after creating the environment.

Click Continue.

Select Git provider

The following instructions only apply if you selected Advanced settings.

Select your Git provider.
1. Hosted Git: will create and host the repository for your project.
2. GitHub: Connect to one of your organization’s GitHub repositories. Read the following docs to get started:
  1. Prerequisites for connecting a GitHub repository to
  2. Installing the app in GitHub
  If you log in to GitHub using SSO, you are required to click Authorize and not Continue to authenticate to matillion-apps. Clicking Continue won’t complete the authorization properly, and you won’t be able to perform remote actions. To reauthorize, follow the steps documented in Permission denied to access GitHub repository.
3. Azure DevOps: Connect to one of your organization’s Azure DevOps repositories. Read the following docs to get started:
  1. Prerequisites for connecting an Azure DevOps repository to
  2. Installing the app in Azure
4. GitLab: Connect to one of your organization’s GitLab repositories. Read the following guide to get started:
  1. Prerequisites for connecting a GitLab repository to
5. Bitbucket: Connect to one of your organization’s Bitbucket repositories. Read the following guide to get started:
  1. Prerequisites for connecting a Bitbucket repository to
Click Continue to trigger the authorization flow, if not already authorized. A new browser tab will open.
Click Authorize . The browser tab will close and return you to the project flow.
For GitHub only. On successful authorizaton, you will see a modal with a list of your authorized accounts. Click Continue.
Once you’ve authorized with your Git provider, you’ll be taken to the Select repository page, where you’ll find the list of repositories you have granted access to . You can connect either an existing repository with multiple branches and a large history, or a brand new repository.
You can only choose a repository that already exists in your organization or project. does not create a new repository in your account.
Select one of the repositories from the list. You can use the search bar to refine your search—for example, if the list contains many repositories.
Click Finish.

You will be redirected to your new project’s menu. From here, you can view your:

You can edit the project name and description at any time by clicking the pencil buttons next to those items at the top of the screen.

Specify GCP cloud credentials

Select the following from the Credentials drop-down to continue creating a project:

Use runner assigned credentials: This option does not have any Google Cloud credentials associated to your environment. Instead, application default credentials will be used.
Specify custom credentials: This option will allow you to specify your credentials, which will result in a Google Cloud credential being associated to your environment. For more information, read Specify custom credentials, below.

Once you’ve chosen which Google Cloud credentials to use, click Add.

Use runner assigned credentials

Go straight to the steps documented in Create an environment.

Specify custom credentials

Specify GCP cloud credentials

Add the credentials in the fields below. The Google Cloud credential will be created, and associated to your environment.

Parameter	Description
Credentials	Select Specify custom credentials.
Credential name	Enter a name for your credentials.
Runner	A working . This is only required if you are using a Hybrid SaaS solution. To learn how to create a , read Create a .
GCP Project ID	The unique, user-defined string identifier for your Google Cloud project. You can find the project ID in the Google Cloud Console on the dashboard.
Service account key	Your Google Cloud service account JSON key file.

Click Add.

Create an environment

You are required to create an environment for this project.

Parameter	Description
Environment name	A unique name for the environment. Max 255 characters.
Runner	A working . This is only required if you are using a Hybrid SaaS solution. To learn how to create a , read Create a .
Default environment access	Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles.

Click Continue.

Select BigQuery defaults

Parameter	Description
GCP Project ID	The unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation.
Dataset	The BigQuery dataset for this environment connection. For more information, read the BigQuery documentation.
Allow inherit project defaults	Use this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments.

If you use a Hybrid SaaS solution, your new environment will inherit the ‘s execution role (service account role) to access your Google Cloud resources, including BigQuery and any associated GCS buckets.To overwrite this role, associate different cloud credentials with this environment after you have finished creating it. You can create these credentials before or after creating the environment.

Click Continue.

Select Git provider

The following instructions only apply if you selected Advanced settings.

Select your Git provider.
1. Hosted Git: will create and host the repository for your project.
2. GitHub: Connect to one of your organization’s GitHub repositories. Read the following docs to get started:
  1. Prerequisites for connecting a GitHub repository to
  2. Installing the app in GitHub
  If you log in to GitHub using SSO, you are required to click Authorize and not Continue to authenticate to matillion-apps. Clicking Continue won’t complete the authorization properly, and you won’t be able to perform remote actions. To reauthorize, follow the steps documented in Permission denied to access GitHub repository.
3. Azure DevOps: Connect to one of your organization’s Azure DevOps repositories. Read the following docs to get started:
  1. Prerequisites for connecting an Azure DevOps repository to
  2. Installing the app in Azure
4. GitLab: Connect to one of your organization’s GitLab repositories. Read the following guide to get started:
  1. Prerequisites for connecting a GitLab repository to
5. Bitbucket: Connect to one of your organization’s Bitbucket repositories. Read the following guide to get started:
  1. Prerequisites for connecting a Bitbucket repository to
Click Continue to trigger the authorization flow, if not already authorized. A new browser tab will open.
Click Authorize . The browser tab will close and return you to the project flow.
For GitHub only. On successful authorizaton, you will see a modal with a list of your authorized accounts. Click Continue.
Once you’ve authorized with your Git provider, you’ll be taken to the Select repository page, where you’ll find the list of repositories you have granted access to . You can connect either an existing repository with multiple branches and a large history, or a brand new repository.
You can only choose a repository that already exists in your organization or project. does not create a new repository in your account.
Select one of the repositories from the list. You can use the search bar to refine your search—for example, if the list contains many repositories.
Click Finish.

You will be redirected to your new project’s menu. From here, you can view your:

You can edit the project name and description at any time by clicking the pencil buttons next to those items at the top of the screen.

Delete a project

Prior to deleting a project, make sure any active schedules are disabled. If you have any active schedules when you try to delete a project, you’ll receive an error message. To delete a project, follow these steps:

In the Your projects page, click the three dots … next to the intended project.
Click Delete project.
The Delete project dialog will be displayed. Type the name of the project you want to delete in the Enter project name field.
To confirm, click Delete Project.

If you create another project with the same name as a deleted project, make sure the new project’s environment name is a different name to the name of the deleted project’s environment.

Project-level sampling

If necessary, you can enable and disable sampling at the project level. This is useful if your data contains personal information that cannot be viewed outside your region. Only users with project admin permissions can change this setting. To enable or disable sampling for a project:

In the Your projects page, click the three dots … next to the intended project.
Click Enable sampling or Disable sampling as required.

Platform overview

Setup

Getting started

Projects

Custom connectors

Runners

Observability

Git

Connections and credentials

Data ops

Troubleshooting

Videos

Add a new project

Select Runner deployment

Specify GCP cloud credentials

Create an environment

Select BigQuery defaults

Specify GCP cloud credentials

Create an environment

Select BigQuery defaults

Select Git provider

Specify GCP cloud credentials

Use runner assigned credentials

Specify custom credentials

Specify GCP cloud credentials

Create an environment

Select BigQuery defaults

Select Git provider

Delete a project

Project-level sampling

​Add a new project

​Select Runner deployment

​Specify GCP cloud credentials

​Create an environment

​Select BigQuery defaults

​Specify GCP cloud credentials

​Create an environment

​Select BigQuery defaults

​Select Git provider

​Specify GCP cloud credentials

​Use runner assigned credentials

​Specify custom credentials

​Specify GCP cloud credentials

​Create an environment

​Select BigQuery defaults

​Select Git provider

​Delete a project

​Project-level sampling

Add a new project

Select Runner deployment

Specify GCP cloud credentials

Create an environment

Select BigQuery defaults

Specify GCP cloud credentials

Create an environment

Select BigQuery defaults

Select Git provider

Specify GCP cloud credentials

Use runner assigned credentials

Specify custom credentials

Specify GCP cloud credentials

Create an environment

Select BigQuery defaults

Select Git provider

Delete a project

Project-level sampling