> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maia.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# SQS pipeline trigger adapter

The SQS pipeline trigger adapter is a packaged, supported AWS Lambda function that triggers a Maia pipeline when a message arrives on an Amazon Simple Queue Service (SQS) queue. Use it to run pipelines in response to events in other systems without writing and maintaining your own Lambda function.

You deploy the adapter with Terraform, alongside your Maia runner on AWS. If you'd rather build and maintain the Lambda function yourself, read [Tutorial: Triggering a pipeline via SQS](https://docs.maia.ai/docs/guides/triggering-a-pipeline-via-sqs.md).

> [!NOTE]
>
> The adapter and the tutorial expect different message formats. This page describes the format for the adapter. For the tutorial's format, read [Messages](https://docs.maia.ai/docs/guides/triggering-a-pipeline-via-sqs.md#messages).

***

## Prerequisites

To use the adapter, you need:

* A Maia runner deployed on AWS with Terraform, using the `runner/aws/ecs` or `runner/aws/eks` configuration from the [deployment library](https://github.com/matillion-public/deployment-library). For EKS, read the [EKS deployment guide](https://docs.maia.ai/docs/guides/eks-deployment-guide.md).
* Maia API credentials (a client ID and client secret). If you're not familiar with API credentials in Maia, read [Authenticate to the Maia API](https://docs.maia.ai/docs/api-reference/maia-api-authentication.md).
* An AWS account with permission to create Lambda functions, SQS queues, DynamoDB tables, and Amazon Elastic Container Registry (ECR) repositories.
* A published pipeline to trigger. The adapter can only run published pipelines.

***

## Copy the adapter image to your own ECR repository

The adapter is published as a container image in Amazon ECR Public at `public.ecr.aws/matillion/maia-queue-adapter`. AWS Lambda can't create a function from an ECR Public image, or from a multi-architecture image. Before you deploy, copy a single-architecture image into a private ECR repository in the same AWS region as the Lambda function.

Use adapter version 1.1.5 or later. Earlier versions can't start pipelines. The following tags are available:

| Tag | Description |
| - | - |
| `<version>-arm64` | A specific version, built for arm64. |
| `<version>-amd64` | A specific version, built for amd64. |
| `<version>` | A specific version. This is the same image as `<version>-arm64`. |
| `current` | The newest version, built for arm64. |
| `current-amd64` | The newest version, built for amd64. |

The Terraform configuration expects an arm64 image, so copy either the `<version>-arm64` tag or the `current` tag.

To copy the image:

1. Create a private ECR repository in the same AWS region as the Lambda function, if you don't already have one:

   ```bash theme={null}
   aws ecr create-repository --repository-name <repository> --region <region>
   ```

2. Pull the arm64 image from ECR Public:

   ```bash theme={null}
   docker pull --platform linux/arm64 public.ecr.aws/matillion/maia-queue-adapter:<version>-arm64
   ```

3. Tag the image for your private repository:

   ```bash theme={null}
   docker tag public.ecr.aws/matillion/maia-queue-adapter:<version>-arm64 <account-id>.dkr.ecr.<region>.amazonaws.com/<repository>:<version>
   ```

4. Authenticate Docker to your private registry:

   ```bash theme={null}
   aws ecr get-login-password --region <region> | docker login --username AWS --password-stdin <account-id>.dkr.ecr.<region>.amazonaws.com
   ```

5. Push the image:

   ```bash theme={null}
   docker push <account-id>.dkr.ecr.<region>.amazonaws.com/<repository>:<version>
   ```

When the copy is complete, note the URI of the image in your private repository. You use it in the [Deploy the adapter with Terraform](#deploy-the-adapter-with-terraform) step.

***

## Store your API credentials

The adapter authenticates to Maia using OAuth client credentials. Create a secret in AWS Secrets Manager that contains your client ID and client secret in the following format:

```json theme={null}
{
  "client_id": "<your-client-id>",
  "client_secret": "<your-client-secret>"
}
```

The adapter only runs published pipelines. If the credentials have an account role other than **Super Admin**, they need access to the project (Viewer or higher) and at least the Operator role on the environment. Operator is the lowest environment role that can run published pipelines—Viewer can't. Read [project roles](https://docs.maia.ai/docs/administration/project-roles.md) and [environment roles](https://docs.maia.ai/docs/administration/environment-roles.md).

The secret must exist before you deploy. Terraform doesn't create it. By default, the adapter looks for a secret named `matillion-dpc`. To use a different name, set `sqs_adapter_secret_name` when you deploy.

***

## Deploy the adapter with Terraform

To deploy the adapter, set `enable_sqs_pipeline_trigger = true` in your `terraform.tfvars` file for `runner/aws/ecs` or `runner/aws/eks`, and set the image URI you noted earlier:

```hcl theme={null}
enable_sqs_pipeline_trigger = true
sqs_adapter_image_uri       = "<account-id>.dkr.ecr.<region>.amazonaws.com/<repository>:<tag>"
```

Then run `terraform init`, `terraform plan`, and `terraform apply` as you would for any change to your runner deployment.

The adapter is disabled by default, so existing deployments are unaffected until you enable it.

### Variables

| Variable | Default | Description |
| - | - | - |
| `enable_sqs_pipeline_trigger` | `false` | Deploys the adapter. |
| `sqs_adapter_image_uri` | None | The URI of the arm64 adapter image in your private ECR repository, in the same region as the Lambda function. Required when the adapter is enabled. |
| `sqs_adapter_secret_name` | `matillion-dpc` | The name of the existing AWS Secrets Manager secret that contains your client ID and client secret. |
| `sqs_adapter_create_queue` | `true` | Creates the source queue and a dead-letter queue. Set to `false` to use an existing queue. |
| `sqs_adapter_queue_name` | `matillion-dpc-requests` | The name of the queue to create. Used when `sqs_adapter_create_queue` is `true`. |
| `sqs_adapter_existing_queue_arn` | None | The ARN of the existing queue. Used when `sqs_adapter_create_queue` is `false`. |
| `sqs_adapter_existing_queue_url` | None | The URL of the existing queue. Used when `sqs_adapter_create_queue` is `false`. |
| `sqs_adapter_create_mapping_table` | `true` | Creates the DynamoDB table that maps project names to project IDs. Set to `false` to use an existing table. |
| `sqs_adapter_mapping_table_name` | `matillion-project-mappings` | The name of the table to create. Used when `sqs_adapter_create_mapping_table` is `true`. |
| `sqs_adapter_existing_mapping_table_name` | None | The name of the existing table. Used when `sqs_adapter_create_mapping_table` is `false`. |
| `sqs_adapter_existing_mapping_table_arn` | None | The ARN of the existing table. Used when `sqs_adapter_create_mapping_table` is `false`. |
| `sqs_adapter_matillion_api_url` | `https://eu1.api.matillion.com/dpc/v1` | The base URL of the Maia pipeline execution API. |
| `sqs_adapter_matillion_token_url` | `https://id.core.matillion.com/oauth/dpc/token` | The Maia OAuth token endpoint. |

> [!WARNING]
>
> The default API URL is for the EU1 region. If your account is in a different region, replace `eu1` in `sqs_adapter_matillion_api_url` with the value for your region.

***

## Add project mappings

The adapter uses a DynamoDB table to look up the project ID for the project name and environment in each message. Add one item to the table for each project and environment combination you want to trigger pipelines in:

| Attribute | Value |
| - | - |
| `PK` | The project name, in lowercase. |
| `SK` | The environment name, in lowercase. |
| `project_id` | The ID of the project. |

The adapter converts the `project` and `environment` values in each message to lowercase before it looks them up, so store `PK` and `SK` in lowercase. The names in the message can use any case.

***

## Send a message to trigger a pipeline

Send a message to the queue in JSON format. Each message contains the following fields:

* `project`: The name of the project that contains the pipeline.
* `environment`: The name of the environment to run the pipeline in.
* `job`: The folder path and name of the pipeline, without a file extension, for example `folder/PipelineName`. Don't add `.orch.yaml`, because the adapter rejects a value that includes a file extension. The value can contain letters, numbers, underscores, hyphens, spaces, and forward slashes (`/`), and can't start or end with a slash.
* `variables`: The scalar variables to pass to the pipeline.
* `gridVariables`: The grid variables to pass to the pipeline.

For example:

```json theme={null}
{
  "project": "CustomerDataPipeline",
  "environment": "dev",
  "job": "folder/PipelineName",
  "variables": {
    "sourceBucket": "s3://data-bucket"
  },
  "gridVariables": {}
}
```

The adapter supports both FIFO and standard queues.

***

## Handle failed messages

If the adapter can't process a message, the message moves to the dead-letter queue according to the redrive policy of the source queue. When Terraform creates the queue, it also creates the dead-letter queue. If you use an existing queue, configure its redrive policy yourself.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.