Skip to main content
The SQS pipeline trigger adapter is a packaged, supported AWS Lambda function that triggers a Maia pipeline when a message arrives on an Amazon Simple Queue Service (SQS) queue. Use it to run pipelines in response to events in other systems without writing and maintaining your own Lambda function. You deploy the adapter with Terraform, alongside your Maia runner on AWS. If you’d rather build and maintain the Lambda function yourself, read Tutorial: Triggering a pipeline via SQS.
The adapter and the tutorial expect different message formats. This page describes the format for the adapter. For the tutorial’s format, read Messages.

Prerequisites

To use the adapter, you need:
  • A Maia runner deployed on AWS with Terraform, using the runner/aws/ecs or runner/aws/eks configuration from the deployment library. For EKS, read the EKS deployment guide.
  • Maia API credentials (a client ID and client secret). If you’re not familiar with API credentials in Maia, read Authenticate to the Maia API.
  • An AWS account with permission to create Lambda functions, SQS queues, DynamoDB tables, and Amazon Elastic Container Registry (ECR) repositories.
  • A published pipeline to trigger. The adapter can only run published pipelines.

Copy the adapter image to your own ECR repository

The adapter is published as a container image in Amazon ECR Public at public.ecr.aws/matillion/maia-queue-adapter. AWS Lambda can’t create a function from an ECR Public image, or from a multi-architecture image. Before you deploy, copy a single-architecture image into a private ECR repository in the same AWS region as the Lambda function. Use adapter version 1.1.5 or later. Earlier versions can’t start pipelines. The following tags are available: The Terraform configuration expects an arm64 image, so copy either the <version>-arm64 tag or the current tag. To copy the image:
  1. Create a private ECR repository in the same AWS region as the Lambda function, if you don’t already have one:
  2. Pull the arm64 image from ECR Public:
  3. Tag the image for your private repository:
  4. Authenticate Docker to your private registry:
  5. Push the image:
When the copy is complete, note the URI of the image in your private repository. You use it in the Deploy the adapter with Terraform step.

Store your API credentials

The adapter authenticates to Maia using OAuth client credentials. Create a secret in AWS Secrets Manager that contains your client ID and client secret in the following format:
The adapter only runs published pipelines. If the credentials have an account role other than Super Admin, they need access to the project (Viewer or higher) and at least the Operator role on the environment. Operator is the lowest environment role that can run published pipelines—Viewer can’t. Read project roles and environment roles. The secret must exist before you deploy. Terraform doesn’t create it. By default, the adapter looks for a secret named matillion-dpc. To use a different name, set sqs_adapter_secret_name when you deploy.

Deploy the adapter with Terraform

To deploy the adapter, set enable_sqs_pipeline_trigger = true in your terraform.tfvars file for runner/aws/ecs or runner/aws/eks, and set the image URI you noted earlier:
Then run terraform init, terraform plan, and terraform apply as you would for any change to your runner deployment. The adapter is disabled by default, so existing deployments are unaffected until you enable it.

Variables

The default API URL is for the EU1 region. If your account is in a different region, replace eu1 in sqs_adapter_matillion_api_url with the value for your region.

Add project mappings

The adapter uses a DynamoDB table to look up the project ID for the project name and environment in each message. Add one item to the table for each project and environment combination you want to trigger pipelines in: The adapter converts the project and environment values in each message to lowercase before it looks them up, so store PK and SK in lowercase. The names in the message can use any case.

Send a message to trigger a pipeline

Send a message to the queue in JSON format. Each message contains the following fields:
  • project: The name of the project that contains the pipeline.
  • environment: The name of the environment to run the pipeline in.
  • job: The folder path and name of the pipeline, without a file extension, for example folder/PipelineName. Don’t add .orch.yaml, because the adapter rejects a value that includes a file extension. The value can contain letters, numbers, underscores, hyphens, spaces, and forward slashes (/), and can’t start or end with a slash.
  • variables: The scalar variables to pass to the pipeline.
  • gridVariables: The grid variables to pass to the pipeline.
For example:
The adapter supports both FIFO and standard queues.

Handle failed messages

If the adapter can’t process a message, the message moves to the dead-letter queue according to the redrive policy of the source queue. When Terraform creates the queue, it also creates the dead-letter queue. If you use an existing queue, configure its redrive policy yourself.