The adapter and the tutorial expect different message formats. This page describes the format for the adapter. For the tutorial’s format, read Messages.
Prerequisites
To use the adapter, you need:- A Maia runner deployed on AWS with Terraform, using the
runner/aws/ecsorrunner/aws/eksconfiguration from the deployment library. For EKS, read the EKS deployment guide. - Maia API credentials (a client ID and client secret). If you’re not familiar with API credentials in Maia, read Authenticate to the Maia API.
- An AWS account with permission to create Lambda functions, SQS queues, DynamoDB tables, and Amazon Elastic Container Registry (ECR) repositories.
- A published pipeline to trigger. The adapter can only run published pipelines.
Copy the adapter image to your own ECR repository
The adapter is published as a container image in Amazon ECR Public atpublic.ecr.aws/matillion/maia-queue-adapter. AWS Lambda can’t create a function from an ECR Public image, or from a multi-architecture image. Before you deploy, copy a single-architecture image into a private ECR repository in the same AWS region as the Lambda function.
Use adapter version 1.1.5 or later. Earlier versions can’t start pipelines. The following tags are available:
The Terraform configuration expects an arm64 image, so copy either the
<version>-arm64 tag or the current tag.
To copy the image:
-
Create a private ECR repository in the same AWS region as the Lambda function, if you don’t already have one:
-
Pull the arm64 image from ECR Public:
-
Tag the image for your private repository:
-
Authenticate Docker to your private registry:
-
Push the image:
Store your API credentials
The adapter authenticates to Maia using OAuth client credentials. Create a secret in AWS Secrets Manager that contains your client ID and client secret in the following format:matillion-dpc. To use a different name, set sqs_adapter_secret_name when you deploy.
Deploy the adapter with Terraform
To deploy the adapter, setenable_sqs_pipeline_trigger = true in your terraform.tfvars file for runner/aws/ecs or runner/aws/eks, and set the image URI you noted earlier:
terraform init, terraform plan, and terraform apply as you would for any change to your runner deployment.
The adapter is disabled by default, so existing deployments are unaffected until you enable it.
Variables
Add project mappings
The adapter uses a DynamoDB table to look up the project ID for the project name and environment in each message. Add one item to the table for each project and environment combination you want to trigger pipelines in:
The adapter converts the
project and environment values in each message to lowercase before it looks them up, so store PK and SK in lowercase. The names in the message can use any case.
Send a message to trigger a pipeline
Send a message to the queue in JSON format. Each message contains the following fields:project: The name of the project that contains the pipeline.environment: The name of the environment to run the pipeline in.job: The folder path and name of the pipeline, without a file extension, for examplefolder/PipelineName. Don’t add.orch.yaml, because the adapter rejects a value that includes a file extension. The value can contain letters, numbers, underscores, hyphens, spaces, and forward slashes (/), and can’t start or end with a slash.variables: The scalar variables to pass to the pipeline.gridVariables: The grid variables to pass to the pipeline.