This article provides an outline of various roles, permissions, and authentication methods for managing resources and access control in cloud environments for use with .
Always follow the principle of least privilege and assign only the minimum permissions required for the specific task.
AWS permissions
This section details permissions related to AWS services like CloudFormation, S3 buckets, IAM roles, ECS (Amazon Elastic Container Service), and agent deployment.
S3 permissions
Users must have appropriate permissions defined in their IAM policies to access S3 buckets from components.
| AWS Permission | Description |
|---|
| s3:GetObject | Grants the ability to get objects from an S3 bucket. |
| s3:ListBucket | Permits listing objects within an S3 bucket. |
| s3:PutObject | Allows to upload (put) an object into an S3 bucket. |
| s3:DeleteObject | Grants permission to delete an object from an S3 bucket. |
| s3:GetBucketLocation | Permits to retrieve the location of an S3 bucket. |
| s3:ListAllMyBuckets | Allows listing buckets. Omitting this does not necessarily cause pipeline errors (although may cause validation errors), but it’s essential to verify the specific requirements for each use case. |
Redshift permissions
This section outlines the steps to manage users, databases, and schemas within the Redshift cluster for . It covers user creation, modification of user access, database creation, modification of database ownership, and schema creation.
| Action | Description | Syntax |
|---|
| User creation | | |
| Create User | Create a user with a password. | CREATE USER new_user PASSWORD 'your_password'; |
| Modify User Access | Grant unrestricted syslog access to the specified user. | ALTER USER new_user SYSLOG ACCESS UNRESTRICTED; |
| Database and schema management | | |
| Create Database | Create a new database with the specified name. | CREATE DATABASE new_database; |
| Modify Database Ownership | Change the owner of the specified database to a new user. | ALTER DATABASE new_database OWNER TO new_owner; |
| Create Schema | Create a new schema within the specified database and assign ownership to a user. | CREATE SCHEMA new_schema AUTHORIZATION new_owner; |
IAM roles
Users configuring their AWS resources for use with are likely to require permission to create IAM Roles.
| AWS Permission | Description |
|---|
| iam:CreateRole | Allows the creation of IAM roles. |
| iam:AttachRolePolicy | Enables attaching policies to an IAM role. |
| iam:PassRole | Necessary if the CloudFormation stack involves passing an IAM role to an AWS service. |
Agent deployment permissions
Users deploying an agent in will require the following permission types:
Coarse-grained:
- AmazonECS_FullAccess
- CloudWatchFullAccess
- SecretsManagerReadWrite
- IAMFullAccess
- AmazonS3FullAccess
Fine-grained:
| AWS policy | Action |
|---|
| IAM | iam:AddRoleToInstanceProfile, iam:PutRolePolicy, iam:PassRole, iam:GetRolePolicy, iam:CreateInstanceProfile, iam:DeleteInstanceProfile, iam:RemoveRoleFromInstanceProfile, iam:DetachRolePolicy, iam:AttachRolePolicy, iam:DeleteRolePolicy, iam:GetRole, iam:CreateRole, iam:DeleteRole, iam:TagPolicy, iam:UntagRole, iam:TagRole, iam:UntagPolicy, iam:UpdateRole, iam:UntagInstanceProfile, iam:TagInstanceProfile. |
| EC2 | ec2:ReplaceIamInstanceProfileAssociation, ec2:AuthorizeSecurityGroupEgress, ec2:AuthorizeSecurityGroupIngress, ec2:RevokeSecurityGroupEgress, ec2:RevokeSecurityGroupIngress, ec2:DescribeSecurityGroups, ec2:DescribeVpcs, ec2:DescribeSubnets, ec2:UpdateSecurityGroupRuleDescriptionsIngress, ec2:UpdateSecurityGroupRuleDescriptionsEgress, ec2:CreateTags, ec2:ModifySecurityGroupRules, ec2:DisassociateIamInstanceProfile, ec2:DeleteTags, ec2:CreateSecurityGroup, ec2:AssociateIamInstanceProfile, ec2:DeleteSecurityGroup. |
| Logging | logs:PutRetentionPolicy, logs:CreateLogStream, logs:TagLogGroup, logs:TagResource, logs:CreateLogGroup, logs:DeleteLogStream, logs:UntagLogGroup, logs:DeleteLogGroup, logs:UntagResource. |
| ECS | secretsmanager:DeleteSecret, secretsmanager:UpdateSecretVersionStage, secretsmanager:TagResource, secretsmanager:UntagResource, secretsmanager:CreateSecret, secretsmanager:UpdateSecret, ecs:UpdateCluster, ecs:DescribeServices, ecs:DescribeTasks, ecs:DescribeClusters, ecs:UpdateContainerInstancesState, ecs:StartTask, ecs:CreateCapacityProvider, ecs:UpdateService, ecs:RegisterTaskDefinition, ecs:StopTask, ecs:DeregisterContainerInstance, ecs:DeleteTaskDefinitions, ecs:TagResource, ecs:UpdateClusterSettings, ecs:CreateCluster, ecs:DeleteService, ecs:DeleteCluster, ecs:RegisterContainerInstance, ecs:DeleteCapacityProvider, ecs:DeregisterTaskDefinition, ecs:CreateService, ecs:RunTask, ecs:UntagResource, ecs:PutClusterCapacityProviders, ecs:UpdateCapacityProvider, ecs:UpdateContainerAgent. |
AWS service-linked roles
AWSServiceRoleForECS: This role is required to allow the deployment of an ECS (Amazon Elastic Container Service) agent in AWS. The ECS agent is often used within ecosystem for managing containerized workloads and orchestrating tasks.
Other service permissions
Below are permissions generally required for specific AWS services:
| AWS Permission | Description |
|---|
cloudformation:CreateStack | Allows the creation of CloudFormation stacks. |
rds:DescribeDBInstances | Required for the RDS Query component to display available RDS instances. |
Azure permissions
The Azure Blob permissions section provides guidance on managing access to Azure Blob Storage, which is crucial for controlling data access and ensuring security within Azure environments. Here’s a brief overview of the key points covered in this section.
Granular control with SAS or Azure Roles helps ensure data security.
Authorization methods
Shared Access Signature (SAS):
- Grants temporary access to specific blobs or containers without exposing account keys.
- Allows defining specific read, write, or list permissions with start and expiry times.
Azure roles:
- Assign built-in or custom roles based on the Azure Role-Based Access Control (RBAC) system.
- Grants access to specific containers, blobs, or entire storage accounts.
Azure blob storage
The below are recommended permissions for accessing Azure Blob Storage with .
| Permission Type | Description |
|---|
| Shared Access Signature (SAS) | |
| Blob Reader | Allows reading blob data. |
| Blob Contributor | Enables reading, writing, and deleting blob data. |
| Blob Container Reader | Grants read access to all blobs within a container. |
| Blob Container Contributor | Provides permissions to manage all blobs within a container. |
| Azure roles | |
| Storage Blob Data Reader | Allows reading blob data. |
| Storage Blob Data Contributor | Enables reading, writing, and deleting blob data. |
| Storage Account Contributor | Gives access to manage all resources within a storage account, including blobs, containers, and account settings. |
| Azure blob integration permissions | |
| Queue Storage Permissions | To interact with Azure Queue Storage, consider permissions like queue:PeekMessages, queue:DequeueMessages, and queue:SendMessage for secure message handling. |
| Data Lake Storage Permissions | To integrate with Azure Data Lake Storage, permissions such as Data Lake Storage Blob Data Owner and Data Lake Storage Blob Data Contributor may be relevant for data access and management. |
Snowflake
Your Snowflake environment in uses your role name to identify you. This role name will have certain privileges associated with it in Snowflake that you can adjust accordingly.
For ease of use and ensuring full functionality, we recommend granting all privileges to the role being used and tying that role only to schemas for use with .
Your role name must be enclosed by double quotes if it contains a space.
Roles and privileges
| Action | Syntax |
|---|
| Role creation | |
| Creating a Custom Role | CREATE ROLE "Data_Productivity_Cloud_role"; |
| Or, if the role name has spaces | CREATE ROLE "Data_Productivity_Cloud role"; |
| Granting privileges | |
| Granting Usage on Warehouse | GRANT USAGE ON WAREHOUSE <warehouse-name> TO ROLE <role-name>; |
| Granting Usage on Database | GRANT USAGE ON DATABASE <database-name> TO ROLE <role-name>; |
| Granting All Privileges on Schema | GRANT ALL ON SCHEMA <schema-name> TO ROLE <role-name>; |
| Granting All Privileges on Tables in Schema | GRANT ALL ON ALL TABLES IN SCHEMA <schema-name> TO ROLE <role-name>; |
| Granting All Privileges on Future Tables in Schema | GRANT ALL ON FUTURE TABLES IN SCHEMA <schema-name> TO ROLE <role-name>; |
| Granting Specific Privilege on Future Tables | GRANT <privilege> ON FUTURE TABLES IN SCHEMA <schema-name> TO ROLE <role-name>; |
| Defining role hierarchy | |
| Assigning Roles | GRANT Data <role-name> TO USER <username>; |
| Revoking privileges | |
| Revoking Privileges | REVOKE <privilege> ON ALL TABLES IN SCHEMA <schema-name> FROM ROLE <role-name>; |
Authentication methods
- Users can authenticate using their Snowflake username and password.
- Users can authenticate using a keypair generated for their Snowflake account.
- Multi-factor authentication connections are not supported for projects. It’s advisable to use dedicated Snowflake Service Account Users for projects.
