> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maia.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Triggering a pipeline via an Azure queue

You can trigger a [pipeline](https://docs.maia.ai/docs/guides/pipelines.md) in Maia automatically when a message arrives on an Azure queue. The Maia queue adapter reads messages from an Azure Service Bus queue or an Azure Queue Storage queue, and starts the pipeline named in each message.

This is the Azure equivalent of the [SQS pipeline trigger adapter](https://docs.maia.ai/docs/guides/sqs-pipeline-trigger-adapter.md) for AWS. Matillion provides the adapter as a packaged container image, so you don't write any function code.

The adapter runs in drain-and-exit mode: each execution processes every message currently on the queue, then stops. A new execution starts when more messages arrive.

> [!NOTE]
>
> You deploy and run the adapter in your own Azure subscription, so you're responsible for the cost and upkeep of the resources it uses.

***

## How it works

1. A system you own sends a message to an Azure queue.
2. Kubernetes Event-driven Autoscaling (KEDA) detects messages on the queue and starts an adapter execution. On a Container App Job, a KEDA scale rule does this. On Azure Kubernetes Service (AKS), a KEDA `ScaledJob` does it.
3. The adapter reads the OAuth credentials for the Maia API from Azure Key Vault, using its managed identity.
4. The adapter looks up the project in an Azure Table Storage mapping table.
5. For each message, the adapter calls the Maia API to run the pipeline, and then removes the message from the queue.

***

## Prerequisites

You need the following:

* A Maia account with [API credentials](https://docs.maia.ai/docs/api-reference/maia-api-authentication.md).
* A published pipeline to trigger. The adapter can only run published pipelines.
* An Azure subscription where you can create managed identities, role assignments, and either Azure Container Apps or an AKS cluster.
* An Azure Key Vault, an Azure Table Storage account, and a queue (Azure Service Bus or Azure Queue Storage).
* Terraform, if you deploy to Azure Container Apps with the Terraform module. For AKS, you need Helm and the [runner Helm chart](https://docs.maia.ai/docs/guides/aks-deployment-guide.md). To deploy with the Maia runner ARM template, you need a Maia runner installed with the [ARM template](https://docs.maia.ai/docs/guides/azure-arm-runner-install.md).

To deploy on AKS, you must also enable the KEDA and Workload Identity add-ons on the cluster.

***

## Choose a queue backend

The adapter supports two backends. You select one when you deploy it.

| Backend | Value | Use when |
| - | - | - |
| Azure Service Bus | `servicebus` | You need message ordering, deduplication, and dead-letter queue (DLQ) behavior equivalent to the AWS adapter. |
| Azure Queue Storage | `storage` | You want a lighter, lower-cost queue. |

***

## Message format

Each message must be JSON in the following format. The format is the same as the AWS adapter.

```json theme={null}
{"project": "CustomerDataPipeline", "environment": "dev", "job": "folder/PipelineName", "variables": {}, "gridVariables": {}}
```

* `project`: The project that contains the pipeline.
* `environment`: The environment to run the pipeline in.
* `job`: The folder path and name of the pipeline, without a file extension, for example `folder/PipelineName`. Don't add `.orch.yaml`, because the adapter rejects a value that includes a file extension. The value can contain letters, numbers, underscores, hyphens, spaces, and forward slashes (`/`). The value can't start or end with a slash.
* `variables`: Scalar variables to pass to the pipeline. Use an empty object if you have none.
* `gridVariables`: Grid variables to pass to the pipeline. Use an empty object if you have none.

***

## Container image

The adapter image is `matillion.azurecr.io/maia-queue-adapter`.

* The image supports amd64 only. Don't use it on Arm node pools.
* The registry allows anonymous pulls, so you don't need registry credentials.

Use adapter version 1.1.5 or later. Earlier versions can't start pipelines. Available tags:

* `<version>`: A specific release.
* `current`: The latest release.

> [!WARNING]
>
> Container App Jobs and KEDA `ScaledJob` resources start a new container for every execution. If you use the `current` tag, the image can change between two executions. In production, pin a specific `<version>` tag.

***

## Prepare Azure resources

### Store the Maia credentials

Create a Key Vault secret that contains the OAuth credentials for the Maia API as JSON:

```json theme={null}
{"client_id": "<your-client-id>", "client_secret": "<your-client-secret>"}
```

For details of creating credentials, read [Authenticate to the Maia API](https://docs.maia.ai/docs/api-reference/maia-api-authentication.md).

The adapter only runs published pipelines. If the credentials have an account role other than **Super Admin**, they need access to the project (Viewer or higher) and at least the Operator role on the environment. Operator is the lowest environment role that can run published pipelines—Viewer can't. Read [project roles](https://docs.maia.ai/docs/administration/project-roles.md) and [environment roles](https://docs.maia.ai/docs/administration/environment-roles.md).

### Create the project mapping table

The adapter reads project mappings from an Azure Table Storage table. The default table name is `matillionprojectmappings`.

If you deploy as a Container App Job with Terraform or the ARM template, you can let the deployment create the storage account and table, or you can use your own. If you deploy on AKS, you create the table yourself.

The table contains one entity for each project and environment combination, with the following properties:

| Property | Value |
| - | - |
| `PartitionKey` | The project name, in lowercase. |
| `RowKey` | The environment name, in lowercase. |
| `project_id` | The Maia project ID. |

The adapter converts the `project` and `environment` values in each message to lowercase before it looks them up. The table must therefore store the partition key and row key in lowercase, but the names in the message can use any case.

### Assign roles to the managed identity

The adapter authenticates to Azure with a managed identity. If you deploy as a Container App Job with Terraform, the module creates the identity and assigns the roles below, scoped to the queue and the table. If you deploy with the ARM template, the adapter uses the Maia runner's managed identity, and the template assigns the roles for any queue and table it creates. If you deploy on AKS, assign these roles yourself to the identity that you set in `queueAdapter.serviceAccount.clientId`.

| Role | Scope | Backend |
| - | - | - |
| Key Vault Secrets User | The Key Vault | Both |
| Storage Table Data Reader | The project mapping table | Both |
| Azure Service Bus Data Receiver | The queue | `servicebus` |
| Storage Queue Data Contributor | The source queue and the `<queueName>-poison` queue | `storage` |

> [!WARNING]
>
> For the `storage` backend, don't use the Storage Queue Data Message Processor role. It can't hide a message again for retry or write to the poison queue, so failed messages aren't handled correctly.

***

## Deploy the adapter

Matillion supports three ways to deploy the adapter:

* [As an Azure Container App Job, using Terraform](#deploy-as-an-azure-container-app-job).
* [As part of an Maia runner installation on Azure Container Apps, using the ARM template (or Bicep)](#deploy-using-the-arm-template).
* [On AKS, using the runner Helm chart](#deploy-on-aks).

### Deploy as an Azure Container App Job

Use the [`queue-adapter-job` Terraform module](https://github.com/matillion-public/deployment-library/tree/main/modules/azure/queue-adapter-job) in the Matillion deployment library. It runs the adapter as an event-driven Container App Job in your existing Azure Container Apps environment. A KEDA scale rule starts an execution when the queue has messages, and the job scales to zero when the queue is empty.

The module creates:

* A user-assigned managed identity for the job, with the role assignments described in [Assign roles to the managed identity](#assign-roles-to-the-managed-identity).
* Optionally, a Service Bus namespace and queue, with duplicate detection and dead-lettering. Alternatively, you can use your own queue.
* Optionally, a storage account and project mapping table. Alternatively, you can use your own.
* On the `storage` backend, the source queue and its `<queueName>-poison` queue.
* The Container App Job, configured with the adapter image and the KEDA scale rule.

You always create the Key Vault secret yourself.

Set the following required inputs:

* `location`: The Azure region.
* `resource_group_name`: The resource group to create the adapter resources in.
* `container_app_environment_id`: The ID of the Container Apps environment to run the job in.
* `image`: The adapter image URI and tag.
* `key_vault_id`: The resource ID of the Key Vault.
* `key_vault_url`: The URL of the Key Vault.
* `secret_name`: The name of the secret that contains the Maia credentials.

If you use the public image at `matillion.azurecr.io/maia-queue-adapter`, leave `container_acr_id` unset. The registry allows anonymous pulls, so the job needs no registry credentials. Set `container_acr_id` only if you mirror the image to your own Azure Container Registry.

You can also set the following optional inputs:

| Input | Default | Description |
| - | - | - |
| `queue_backend` | `servicebus` | `servicebus` or `storage`. |
| `create_queue` | `true` | Set to `false` to use your own Service Bus queue. If you do, also set `existing_servicebus_namespace_id` and `existing_servicebus_namespace_fqdn`. |
| `queue_name` | `matillion-maia-requests` | The queue to read from. |
| `create_mapping_table` | `true` | Set to `false` to use your own storage account. If you do, also set `existing_storage_account_id` and `existing_storage_account_url`. On the `storage` backend, also set `existing_storage_queue_url`. |
| `mapping_table_name` | `matillionprojectmappings` | The project mapping table. |
| `matillion_api_url` | `https://eu1.api.matillion.com/dpc/v1` | The Maia API URL for your region. Read [Set the region](#set-the-region) for the values to use. |
| `matillion_token_url` | `https://id.core.matillion.com/oauth/dpc/token` | The Maia token URL. |
| `min_executions` | 0 | The minimum number of KEDA executions. |
| `max_executions` | 10 | The maximum number of concurrent KEDA executions. |
| `polling_interval` | 30 | How often KEDA checks the queue depth, in seconds. |
| `messages_per_execution` | 5 | The number of messages per execution. |
| `storage_queue_length` | 5 | On the `storage` backend, the KEDA target number of messages per execution. |
| `existing_storage_queue_url` | None | The Queue endpoint of your own storage account, for example `https://<account>.queue.core.windows.net`. Required on the `storage` backend when `create_mapping_table` is `false`. |
| `servicebus_sku` | `Standard` | The Service Bus namespace SKU, `Standard` or `Premium`. Duplicate detection requires Standard or higher. |
| `max_delivery_count` | 3 | The number of times Service Bus delivers a message before it dead-letters the message, for a queue that the module creates. |
| `workload_profile_name` | `agentpool` | The workload profile to run the job on. The value must match a workload profile in your Container Apps environment. |
| `cpu` | 0.5 | The vCPU for the job container. |
| `memory` | `1Gi` | The memory for the job container. |

For example:

```hcl theme={null}
module "queue_adapter" {
  source = "../../modules/azure/queue-adapter-job"

  location                     = var.location
  resource_group_name          = var.resource_group_name
  container_app_environment_id = module.container_apps.container_app_environment_id
  image                        = "matillion.azurecr.io/maia-queue-adapter:<version>"
  key_vault_id                 = module.container_apps.key_vault_id
  key_vault_url                = module.container_apps.key_vault_uri
  secret_name                  = "matillion-maia"
}
```

> [!WARNING]
>
> The KEDA scaler on a Container App Job needs a connection string to read the queue depth. For a queue that the module creates, the module generates a listen-only credential. If you use your own queue, you must supply one in `scaler_connection_string`. Either way, the value is stored in plaintext in your Terraform state, so keep the state encrypted and restrict access to it.

> [!NOTE]
>
> For the `storage` backend, the scaler needs a storage account connection string, which can't be restricted to listen-only access. For this backend, deploy on AKS instead, where the scaler uses workload identity and needs no connection string.

### Deploy using the ARM template

Both Azure Maia runner templates—dedicated and consumption—include an optional queue adapter. It's off by default, so the template creates no adapter resources unless you turn it on. When enabled, the adapter runs as a Container App Job, scaled by KEDA, alongside your Maia runner. It uses the Maia runner's managed identity and reads the OAuth credentials from the Maia runner's Key Vault.

To use it, follow the steps in [Maia runner installation using an ARM template](https://docs.maia.ai/docs/guides/azure-arm-runner-install.md), and set the following parameters on the **Custom deployment** page:

| Parameter | Default | Description |
| - | - | - |
| `enableQueueAdapter` | `false` | Set to `true` to deploy the adapter. |
| `queueAdapterBackend` | `servicebus` | `servicebus` or `storage`. |
| `queueAdapterImageURL` | `matillion.azurecr.io/maia-queue-adapter:current` | The adapter image. Pin a version in production. |
| `createQueueAdapterResources` | `true` | Set to `true` to create the queue, storage account, and project mapping table. Set to `false` to use your own. |
| `queueAdapterQueueName` | `matillion-maia-requests` | The queue to read from. The name can be at most 56 characters. |
| `queueAdapterMappingTableName` | `matillionprojectmappings` | The project mapping table. |
| `queueAdapterSecretName` | `matillion-maia-oauth` | The name of the Key Vault secret that holds the Maia credentials. |
| `queueAdapterApiUrl` | `https://eu1.api.matillion.com/dpc/v1` | The Maia API URL for your region. Read [Set the region](#set-the-region). |
| `queueAdapterTokenUrl` | `https://id.core.matillion.com/oauth/dpc/token` | The Maia token URL. |

If you set `createQueueAdapterResources` to `false`, also set the following parameters for your own resources:

* `existingServiceBusNamespaceFqdn`: The fully qualified Service Bus namespace, for example `<namespace>.servicebus.windows.net`.
* `existingStorageTableEndpoint`: The Table endpoint of your storage account, for example `https://<account>.table.core.windows.net/`.
* `existingStorageQueueUrl`: On the `storage` backend, the Queue endpoint of your storage account, for example `https://<account>.queue.core.windows.net/`.
* `queueAdapterScalerConnectionString`: A listen-only Service Bus connection string that the KEDA scaler uses to read the queue depth.

The template doesn't assign roles on your own resources. Instead, it returns a `queueAdapterRequiredRoles` output that lists the roles to grant to the Maia runner's managed identity. For the full list, read [Assign roles to the managed identity](#assign-roles-to-the-managed-identity).

> [!WARNING]
>
> The `queueAdapterScalerConnectionString` value is a secret. Keep your deployment history and any parameter files restricted.

### Deploy on AKS

Enable the queue adapter in the Maia runner Helm chart. The adapter is disabled by default, so the chart creates no adapter resources until you enable it. When enabled with `cloudProvider: azure`, the chart deploys a KEDA `ScaledJob` that starts an adapter job when the queue has messages.

Set the following values:

| Value | Default | Description |
| - | - | - |
| `cloudProvider` | None | Set to `azure`. |
| `queueAdapter.enabled` | `false` | Set to `true` to deploy the adapter. |
| `queueAdapter.backend` | `servicebus` | `servicebus` or `storage`. |
| `queueAdapter.image.repository` | None | `matillion.azurecr.io/maia-queue-adapter`, or a mirror in your own Azure Container Registry. |
| `queueAdapter.image.tag` | `current` | The image tag. Pin a version in production. |
| `queueAdapter.serviceAccount.clientId` | None | The workload identity client ID of the managed identity that the adapter runs as. |
| `queueAdapter.keyVault.url` | None | The URL of the Key Vault, for example `https://<vault>.vault.azure.net/`. |
| `queueAdapter.keyVault.secretName` | `matillion-maia` | The name of the secret that contains the Maia credentials. |
| `queueAdapter.storage.accountName` | None | The short name of the storage account. |
| `queueAdapter.storage.accountUrl` | None | The Table endpoint of the storage account, for example `https://<account>.table.core.windows.net`. |
| `queueAdapter.storage.tableName` | `matillionprojectmappings` | The project mapping table. |
| `queueAdapter.matillionApiUrl` | `https://eu1.api.matillion.com/dpc/v1` | The Maia API URL for your region. Read [Set the region](#set-the-region). |
| `queueAdapter.matillionTokenUrl` | `https://id.core.matillion.com/oauth/dpc/token` | The Maia token URL. |

Then set the values for your backend.

For the `servicebus` backend, set the following values:

* `queueAdapter.serviceBus.namespace`: The fully qualified namespace, for example `<namespace>.servicebus.windows.net`.
* `queueAdapter.serviceBus.queueName`: The queue to read from. The default is `matillion-maia-requests`.

For the `storage` backend, set the following values:

* `queueAdapter.storage.queueUrl`: The Queue endpoint of the storage account, for example `https://<account>.queue.core.windows.net`.
* `queueAdapter.storage.queueName`: The queue to read from. The default is `matillion-maia-requests`.

For example, to use the `servicebus` backend:

```yaml theme={null}
cloudProvider: azure
queueAdapter:
  enabled: true
  backend: servicebus
  image:
    repository: matillion.azurecr.io/maia-queue-adapter
    tag: <version>
  serviceAccount:
    clientId: <managed-identity-client-id>
  serviceBus:
    namespace: <namespace>.servicebus.windows.net
    queueName: matillion-maia-requests
  keyVault:
    url: https://<vault>.vault.azure.net/
    secretName: matillion-maia
  storage:
    accountName: <storage-account-name>
    accountUrl: https://<storage-account-name>.table.core.windows.net
    tableName: matillionprojectmappings
```

#### Scaling

KEDA scales the adapter on queue depth. You can adjust the following values under `queueAdapter.scaling`:

| Value | Default | Description |
| - | - | - |
| `minReplicaCount` | 0 | The minimum number of adapter jobs. With the default, no jobs run when the queue is empty. |
| `maxReplicaCount` | 10 | The maximum number of concurrent adapter jobs. |
| `pollingInterval` | 30 | How often KEDA checks the queue depth, in seconds. |
| `messagesPerExecution` | 5 | For the `servicebus` backend, the number of messages per job. |
| `queueLength` | 5 | For the `storage` backend, the number of messages per job. |

You can also set CPU and memory requests and limits for the adapter under `queueAdapter.resources`.

For general guidance on deploying Maia runners on AKS, read the [Azure AKS deployment guide](https://docs.maia.ai/docs/guides/aks-deployment-guide.md).

#### Add a second federated credential for KEDA

> [!WARNING]
>
> On AKS, the KEDA operator reads the queue depth using the adapter's managed identity. You must add a federated credential to that identity for the KEDA operator's service account. If you don't, the `ScaledJob` reports a Ready status but never starts a job, and messages accumulate on the queue with no error.

The adapter's managed identity needs two federated credentials:

* A credential for the adapter's own service account, which the Helm chart configures.
* A credential for the KEDA operator's service account, `kube-system/keda-operator` on the AKS add-on.

KEDA uses this credential to assume the adapter's identity and read the queue depth. For example, using the Azure CLI:

```bash theme={null}
az identity federated-credential create \
  --name keda-operator \
  --identity-name <managed-identity-name> \
  --resource-group <resource-group> \
  --issuer "$(az aks show --name <cluster-name> --resource-group <resource-group> --query oidcIssuerProfile.issuerUrl -o tsv)" \
  --subject system:serviceaccount:kube-system:keda-operator \
  --audiences api://AzureADTokenExchange
```

***

## Set the region

The adapter calls the Maia API for your region. Set the API URL to `https://<region>.api.matillion.com/dpc/v1`, where `<region>` is one of `eu1`, `eu2`, `us1`, or `au1`.

The setting is called `matillion_api_url` in Terraform, `queueAdapter.matillionApiUrl` in the Helm chart, and `queueAdapterApiUrl` in the ARM template.

***

## Test the setup

1. Publish the pipeline that you want to trigger in Maia.
2. Send a test message in the [message format](#message-format) to the queue.
3. Check the **Your activity** page in Maia. The pipeline run appears within a few moments. For more information, read [Pipeline run history](https://docs.maia.ai/docs/guides/pipeline-run-history.md).

If the pipeline doesn't start, check the following:

* On AKS, confirm that you added the second federated credential for KEDA.
* Confirm that the managed identity has all the roles for your backend.
* Confirm that the project, environment, and pipeline names in the message match your Maia project. Project and environment names aren't case-sensitive, but the project mapping table must contain a matching entity in lowercase.
* Check the poison queue for messages that the adapter couldn't process.

### Monitor failed messages

When the adapter can't process a message, it moves the message to the dead-letter queue (`servicebus`) or the `<queueName>-poison` queue (`storage`).

On a Container App Job, a run that dead-letters a message exits with an error, and Container Apps retries it. The retry finds the queue already cleared and exits successfully, so the execution shows as `Succeeded` even though a message was discarded. Alert on the depth of the poison queue or on the adapter's `MessagesDeadLettered` metric, not on the job execution status.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.