> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maia.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Cloud provider credentials

export const m_runner = "Maia runner";

export const maia = "Maia";

Each [environment](/docs/guides/environments) in your project should have at least one set of cloud credentials associated with it. This allows you to access account resources on different platforms other than that hosting your project. For example, if your project is on AWS and you want to access resources in Azure, you need to associate your Azure cloud credentials with the environment.

* In a [Matillion Full SaaS](/docs/guides/runner-overview#matillion-full-saas) project, you **must** specify a cloud connection to access resources in your own cloud provider account.
* In a [Hybrid SaaS runner](/docs/guides/runner-overview#hybrid-saas) project that needs to connect to cloud resources outside the platform hosting your {m_runner}, you need to provide credentials to connect to that platform.

Each credential allows access to a single cloud platform. Supported platforms are:

* Amazon Web Services
* Azure
* Google Cloud

Credentials are configured at the project level and apply to selected environments within the project. They can then be used by any pipelines that use those environments.

{maia} only uses your cloud provider credentials to connect to the resources specified within your pipeline component configurations. Your credentials are not used to access or retrieve secrets.

## Understanding environment defaults

Each cloud connection should be given default values for the connection credentials. Default values are a fallback for all environments that have defaults enabled. You can then add specific overrides for each environment as needed. Pipelines will fail in any environment that lacks either a default or an override.

As you can add default values **and** specific overrides, it's important to understand how they interact, and which credentials an environment uses.

* If you only fill in the defaults for the cloud connection, you can use that connection in **any** environment. Every environment that uses the connection will use the default credentials.
* If you don't fill in the defaults for the cloud connection, you can **only** use the connection in environments that you have specified overrides for. The environment will use the override credentials.
* If you fill in the defaults for the cloud connection and **also** fill in some environmental overrides, you can use that connection in **any** environment. Every environment that you have specified overrides for will use the override credentials. Every other environment will use the default credentials.
* If you don't fill in the defaults for the cloud connection, and don't associate the connection with any environment you have specified overrides for, the connection then becomes essentially dormant, not used or usable by any environment.

***

## Creating a cloud provider credential

To set up a new cloud provider credential, follow these steps:

1. In your project, click the **Cloud credentials** tab.

   This tab lists all cloud credentials currently created.

2. Click **Add cloud credential**.

3. Choose the cloud provider and click **Continue**.

4. Complete the following fields:
   * **Name:** A unique, descriptive name for your cloud provider credential. This is used to reference this connection in pipelines. It can't be changed after creation.
   * **Provider:** This is pre-populated with the provider chosen on the previous screen and can't be changed here.
   * **Description:** An optional description of the credential. This is for your information only, and does not affect the credential.
   * **Runner:** This is only required if you host your own {m_runner} in a [Hybrid SaaS deployment](/docs/guides/runner-overview#matillion-full-saas-vs-hybrid-saas). Use the drop-down to select the {m_runner} that will be used to store these credentials securely in {maia}.

5. Enter credentials for the connection. This includes both **Default values** and as many environment-specific credentials as you require. Read [Understanding environment defaults](#understanding-environment-defaults), above, for more details.

   <Tabs>
     <Tab title="AWS">
       * **Access key ID:** Access key ID for an existing Amazon IAM user.
       * **Secret access key:** Secret access key for an existing Amazon IAM user.

       Obtain these credentials from the AWS console, as described in [Manage access keys for IAM users](https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_access-keys.html).
     </Tab>

     <Tab title="Azure">
       * **Tenant ID:** The Azure Tenant ID, also known as Directory ID.
       * **Client ID:** The Azure Client ID, also known as Application ID.
       * **Secret key:** The Azure Secret Key.

       Obtain these credentials from the Azure portal, as described in [Register a Microsoft Entra app and create a service principal](https://learn.microsoft.com/en-us/entra/identity-platform/howto-create-service-principal-portal) and [Add and manage application credentials in Microsoft Entra ID](https://learn.microsoft.com/en-us/entra/identity-platform/how-to-add-credentials).

       For Azure Databricks, an [additional step](https://learn.microsoft.com/en-us/azure/databricks/connect/unity-catalog/cloud-storage/external-locations) is required to connect to cloud storage. For further information, read [Accessing Storage using a service principal and Microsoft Entra ID](https://docs.databricks.com/aws/en/connect/storage/aad-storage-service-principal).

       To connect to Data Lake Storage Gen2 and Blob Storage, read [Connect to Azure Data Lake Storage Gen2 and Blob Storage](https://docs.databricks.com/aws/en/connect/storage/azure-storage?language=Azure%C2%A0service%C2%A0principal).
     </Tab>

     <Tab title="Google Cloud">
       * **Service account key:** Enter the service account key to access services such as Google BigQuery and Google Cloud Storage. Make sure that the key is in a valid JSON format, as in the following example:

         ```json theme={null}
         {
         "type": "service_account",
         "project_id": "abcde",
         "private_key_id": "",
         "private_key": "",
         "client_email": "abcde@appspot.gserviceaccount.com",
         "client_id": "XXXXXXXXXXXXX",
         "auth_uri": "https://accounts.google.com/o/oauth2/auth",
         "token_uri": "https://accounts.google.com/o/oauth2/token",
         "auth_provider_x509_cert_url": "https://www.googleapis.com/oauth2/v1/certs",
         "client_x509_cert_url": "https://www.googleapis.com/robot/v1/metadata/x509/abcde%40appspot.gserviceaccount.com"
         }
         ```

       Obtain this credential from the Google Cloud console, as described in [Create and delete service account keys](https://cloud.google.com/iam/docs/keys-create-delete).
     </Tab>
   </Tabs>

6. Click **Add** to move to the **Update environment defaults** screen.

7. Select the environments where this connection should act as the default credential for accessing cloud resources. Any number of environments can be selected, or the step can be skipped without making any selection. For any environments you select, ensure that you specified default credentials or environment-specific credentials in step 5. Read [Understanding environment defaults](#understanding-environment-defaults), above, for more details.

8. If you select environments to use the connection, click **Update**. Otherwise, click **Skip**.

***

## Deleting a cloud provider credential

1. In your project, click the **Cloud credentials** tab.

   This tab lists all cloud credentials currently created.

2. Click the three dots **...** in the row of the credential you want to delete.

3. Click **Delete cloud credential**.

4. Click **Yes, delete** to confirm deletion. Otherwise, click **Cancel**.

***

## Editing a cloud provider credential

1. In your project, click the **Cloud credentials** tab.

   This tab lists all cloud credentials currently created.

2. Click the three dots **...** in the row of the credential you want to edit.

3. Click **Edit cloud credential**.

4. Edit any of the details described in [Creating a cloud provider credential](#creating-a-cloud-provider-credential). You can't rename the credential here.

5. Click **Update**.

***

## Roles and permissions for Cloud Storage

To load your data directly into your preferred cloud storage destination, such as S3, Azure Blob Storage, or Google Cloud Storage, you'll need to apply the following roles and permissions to your cloud accounts:

<table>
  <tr>
    <th>Cloud Storage</th>
    <th>Role/policy</th>
    <th>Permissions</th>
  </tr>

  <tr>
    <td><a href="https://docs.aws.amazon.com/AmazonS3/latest/userguide/access-policy-language-overview.html">S3</a></td>
    <td>An appropriate policy name</td>

    <td>
      <ul>
        <li>s3:DeleteObject</li>
        <li>s3:GetObject</li>
        <li>s3:PutObject</li>
        <li>s3:ListAllMyBuckets</li>
      </ul>
    </td>
  </tr>

  <tr>
    <td rowSpan="2">Azure storage</td>

    <td>
      <p><a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/storage#storage-blob-data-contributor">Storage Blob Data Contributor</a></p>
      <p>For information about adding this role to your account, read <a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal">Assign Azure roles using the Azure portal</a>.</p>
    </td>

    <td>
      <p>Actions:</p>

      <ul>
        <li>Microsoft.Storage/storageAccounts/blobServices/containers/delete</li>
        <li>Microsoft.Storage/storageAccounts/blobServices/containers/read</li>
        <li>Microsoft.Storage/storageAccounts/blobServices/containers/write</li>
        <li>Microsoft.Storage/storageAccounts/blobServices/generateUserDelegationKey/action</li>
      </ul>

      <p>DataActions:</p>

      <ul>
        <li>Microsoft.Storage/storageAccounts/blobServices/containers/blobs/delete</li>
        <li>Microsoft.Storage/storageAccounts/blobServices/containers/blobs/read</li>
        <li>Microsoft.Storage/storageAccounts/blobServices/containers/blobs/write</li>
        <li>Microsoft.Storage/storageAccounts/blobServices/containers/blobs/move/action</li>
        <li>Microsoft.Storage/storageAccounts/blobServices/containers/blobs/add/action</li>
      </ul>
    </td>
  </tr>

  <tr>
    <td>
      <p><a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/built-in-roles/storage#storage-account-contributor">Storage Account Contributor</a></p>
      <p>For information about adding this role to your account, read <a href="https://learn.microsoft.com/en-us/azure/role-based-access-control/role-assignments-portal">Assign Azure roles using the Azure portal</a>.</p>
    </td>

    <td>
      <p>Actions:</p>

      <ul>
        <li>Microsoft.Authorization/{'*'}/read</li>
        <li>Microsoft.Insights/alertRules/{'*'}</li>
        <li>Microsoft.Insights/diagnosticSettings/{'*'}</li>
        <li>Microsoft.Network/virtualNetworks/subnets/joinViaServiceEndpoint/action</li>
        <li>Microsoft.ResourceHealth/availabilityStatuses/read</li>
        <li>Microsoft.Resources/deployments/{'*'}</li>
        <li>Microsoft.Resources/subscriptions/resourceGroups/read</li>
        <li>Microsoft.Storage/storageAccounts/{'*'}</li>
        <li>Microsoft.Support/{'*'}</li>
      </ul>
    </td>
  </tr>

  <tr>
    <td><a href="https://cloud.google.com/storage/docs/access-control/iam-roles">Google Cloud Storage</a></td>
    <td>Storage Admin</td>
    <td>storage.buckets.\*</td>
  </tr>
</table>

***

## Roles and permissions for Amazon Bedrock Prompt

The [Amazon Bedrock Prompt](/docs/components/amazon-bedrock-prompt) component requires permissions to be set to allow access to the Bedrock large language models (LLMs).

Coarse-grained permission:

* AmazonBedrockFullAccess

Fine-grained permission:

* bedrock:InvokeModel on the models that need to be accessible.

To allow all models, use:

```json theme={null}
{
  "Sid": "InvokeModel",
  "Effect": "Allow",
  "Action": [
    "bedrock:InvokeModel"
  ],
  "Resource": "arn:aws:bedrock:*::foundation-model/*"
}
```

To restrict access to a single model, for example `anthropic.claude-3-sonnet-20240229-v1:0`, use:

```json theme={null}
{
  "Sid": "InvokeModel",
  "Effect": "Allow",
  "Action": [
    "bedrock:InvokeModel"
  ],
  "Resource": "arn:aws:bedrock:*::foundation-model/anthropic.claude-3-sonnet-20240229-v1:0"
}
```