An environment defines the connection between a project and your chosen cloud data warehouse. Environments include useful defaults—such as a default warehouse, database, and schema—that can be used to pre-populate component configurations in . Haven’t added a project yet? Read Google BigQuery projects.
We recommend using environments to separate your development and production environments:
- Use development environments for building, testing, and iterating on pipelines before they are deployed.
- Use production environments to run pipelines that are fully deployed to work on live data. Only stable and thoroughly tested pipelines should be deployed here.
- You can also use intermediate environments, such as
staging, test, or preprod, to validate pipelines before they are deployed to production. These can also be used for performance testing.
For more information, read our Unlocking Data Productivity DataOps guide.
Prerequisites
Before adding a new environment, make sure a Google Cloud credential is configured in the project. You can then choose whether to use the project-level credential or specify an environment-level value.
Add an environment
This tab lists all environments in your project. Follow these steps if you’ve already created a BigQuery project.
-
In your project, click the Environments tab.
-
Click Add environment.
The available options for adding a new environment depend on your project configuration.
Maia managed
Full SaaS
Hybrid SaaS
Specify GCP cloud credentials
You are required to provide the following GCP cloud credentials:| Parameter | Description |
|---|
| GCP credentials | Use the drop-down to select the credentials you set up when you created your project. |
| Use project defaults | Select this option to inherit the cloud credentials configured as defaults for your project. |
| Override for this environment | Select this option if you want to provide new credentials for this environment only. |
Use project defaults
If you selected Use project defaults on the previous screen, use this section to create a new environment.Create an environment
| Parameter | Description |
|---|
| Environment name | A unique name for the environment. Max 255 characters. |
| Default environment access | Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles. |
Select BigQuery defaults
| Parameter | Description |
|---|
| GCP Project ID | The unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation. |
| Dataset | The BigQuery dataset for this environment connection. For more information, read the BigQuery documentation. |
| Allow inherit project defaults | Use this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments. |
Override for this environment
If you selected Override for this environment on the previous screen, use this section to select your Google BigQuery defaults.Select BigQuery defaults
| Parameter | Description |
|---|
| GCP Project ID | The unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation. |
| Dataset | The BigQuery dataset for this environment connection. For more information, read the BigQuery documentation. |
| Allow inherit project defaults | Use this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments. |
Specify GCP cloud credentials
You are required to provide the following GCP cloud credentials:| Parameter | Description |
|---|
| GCP credentials | Use the drop-down to select the credentials you set up when you created your project. |
| Use project defaults | Select this option to inherit the cloud credentials configured as defaults for your project. |
| Override for this environment | Select this option if you want to provide new credentials for this environment only. |
Use project defaults
If you selected Use project defaults on the previous screen, use this section to create a new environment.Create an environment
| Parameter | Description |
|---|
| Environment name | A unique name for the environment. Max 255 characters. |
| Default environment access | Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles. |
Select BigQuery defaults
| Parameter | Description |
|---|
| GCP Project ID | The unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation. |
| Dataset | The BigQuery dataset for this environment connection. For more information, read the BigQuery documentation. |
| Allow inherit project defaults | Use this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments. |
If you use a Matillion Full SaaS solution, the cloud credentials associated with your environment will be used to access your Google Cloud resources, including BigQuery and any associated GCS buckets.To overwrite this role, associate different cloud credentials with this environment after you have finished creating it. You can create these credentials before or after creating the environment. Override for this environment
If you selected Override for this environment on the previous screen, use this section to create an environment.| Parameter | Description |
|---|
| Environment name | A unique name for the environment. Max 255 characters. |
| Default environment access | Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles. |
| Service Account Key | Your Google Cloud service account JSON key file. |
Select BigQuery defaults
| Parameter | Description |
|---|
| GCP Project ID | The unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation. |
| Dataset | The BigQuery dataset for this environment connection. For more information, read the BigQuery documentation. |
| Allow inherit project defaults | Use this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments. |
If you use a Matillion Full SaaS solution, the cloud credentials associated with your environment will be used to access your Google Cloud resources, including BigQuery and any associated GCS buckets.To overwrite this role, associate different cloud credentials with this environment after you have finished creating it. You can create these credentials before or after creating the environment. Specify GCP cloud credentials
Select the following from the Credentials drop-down:Once you’ve chosen which GCP cloud credentials to use, click Continue.Use runner assigned credentials
If you selected Use runner assigned credentials earlier, complete the following to create an environment.Create an environment
| Parameter | Description |
|---|
| Environment name | A unique name for the environment. Max 255 characters. |
| Runner | A working . This is only required if you are using a Hybrid SaaS solution. To learn how to create a , read Create a . |
| Default environment access | Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles. |
s can be restricted to specific projects and environments. If a is not allowed for your project or environment, it will not appear in the Runner drop-down. For more information, read Restricting s. Select BigQuery defaults
| Parameter | Description |
|---|
| GCP Project ID | The unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation. |
| Dataset | The BigQuery dataset for this environment connection. For more information, read the BigQuery documentation. |
| Allow inherit project defaults | Use this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments. |
If you use a Hybrid SaaS solution, your new environment will inherit the ‘s execution role (service account role) to access your Google Cloud resources, including BigQuery and any associated GCS buckets.To overwrite this role, associate different cloud credentials with this environment after you have finished creating it. You can create these credentials before or after creating the environment. Specify custom credentials
Specify GCP cloud credentials
If you selected Specify custom credentials earlier, use this section to specify your GCP cloud credentials.| Parameter | Description |
|---|
| Credentials | Specify custom credentials. |
| GCP credentials | The credentials you set up when you created your project, or credentials you created some other time. |
| Use project defaults | Select this option to inherit the cloud credentials configured as defaults for your project. |
| Override for this environment | Select this option if you want to provide new credentials for this environment. |
Use project defaults
If you selected Use project defaults above, create an environment.Create an environment
| Parameter | Description |
|---|
| Environment name | A unique name for the environment. Max 255 characters. |
| Runner | A working . This is only required if you are using a Hybrid SaaS solution. To learn how to create a , read Create a . |
| Default environment access | Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles. |
s can be restricted to specific projects and environments. If a is not allowed for your project or environment, it will not appear in the Runner drop-down. For more information, read Restricting s. Select BigQuery defaults
| Parameter | Description |
|---|
| GCP Project ID | The unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation. |
| Dataset | The BigQuery dataset for this environment connection. For more information, read the BigQuery documentation. |
| Allow inherit project defaults | Use this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments. |
If you use a Hybrid SaaS solution, your new environment will inherit the ‘s execution role (service account role) to access your Google Cloud resources, including BigQuery and any associated GCS buckets.To overwrite this role, associate different cloud credentials with this environment after you have finished creating it. You can create these credentials before or after creating the environment. Override for this environment
If you selected Override for this environment above, create an environment.Click Continue.Create an environment
Add the following details to override your current project environment settings:| Parameter | Description |
|---|
| Environment name | A unique name for the environment. Max 255 characters. |
| Runner | A working . This is only required if you are using a Hybrid SaaS solution. To learn how to create a , read Create a . |
| Default environment access | Use the drop-down to select the default access for all new and existing users added to the project. For more information, read Environment roles. |
| GCP Project ID | The unique, user-defined string identifier for your Google Cloud project. You can find the project ID in the Google Cloud Console on the dashboard. |
| Service Account Key | Your Google Cloud service account JSON key file. |
Select BigQuery defaults
| Parameter | Description |
|---|
| GCP Project ID | The unique, user-defined string identifier for your Google Cloud project that contains your BigQuery datasets and tables. You can find a project’s ID in the Google Cloud Console on the dashboard. For more information, read the Google Cloud documentation. |
| Dataset | The BigQuery dataset for this environment connection. For more information, read the BigQuery documentation. |
| Allow inherit project defaults | Use this toggle to manage how the environment handles variable values. When enabled (default), the environment inherits project-level default values. If disabled, you must manually provide values for each variable to ensure pipelines function successfully across environments. |
If you use a Hybrid SaaS solution, your new environment will inherit the ‘s execution role (service account role) to access your Google Cloud resources, including BigQuery and any associated GCS buckets.To overwrite this role, associate different cloud credentials with this environment after you have finished creating it. You can create these credentials before or after creating the environment.
Associate cloud provider credentials with an environment
Each environment in your project should have at least one set of cloud credentials associated with it. This allows you to access account resources on different platforms other than that hosting your project. For example, if your project is on AWS and you want to access resources in Azure, you need to associate your Azure cloud credentials with the environment.
Credentials are configured at the project level and apply to selected environments within the project. They can then be used by any pipelines that use those environments. To create and associate cloud credentials, read Cloud provider credentials.
Edit or remove an associated credential
Edit or remove which credential is in use by your environment:
- Hybrid SaaS: Removing the credential causes the environment to fall back to runner-assigned credentials.
- Full SaaS: Removing the credential leaves the environment unusable, because Full SaaS environments require an explicitly associated credential.
Manage environments
To view your environments:
- From the Your projects menu, select your project.
- Navigate to the Environments tab.
Click the column headers to sort your environments by name, default , cloud data warehouse account name, or credential type.
Edit an environment
The edit environment screen only allows you to edit default values for your GCP project and dataset.
- Click the three dots … in the row of the environment you want to edit.
- Click Edit environment.
- On the Edit Google BigQuery credentials, use the Runner drop-down to choose a different runner for your environment.
- Use the Default environment access drop-down to choose a different default role for your environment. For more information, read Environment roles.
- Click Continue to move to the next screen, where you can use the drop-down menus to update your BigQuery defaults.
- Click Update to return to the Environments tab in your project.
If you disable the Allow inherit project defaults toggle on the BigQuery defaults screen, the environment will not inherit project-level defaults. You must manually provide a default value for each variable within the environment to ensure pipelines function successfully across different environments.
Delete an environment
Deleting an environment permanently removes the environment from your project. All artifacts and schedules in the deleted environment will be inaccessible. This action cannot be undone.
- Disable any active schedules that run pipelines in this environment.
- Change the default environment of any branches that currently use this environment as their default. For more information, read Branches.
To delete an environment:
- Click the three dots … in the row of the environment you want to delete.
- Click Delete environment.
- In the confirmation dialog, enter the name of the environment you want to delete.
- Click Delete environment.
