Skip to main content
You can trigger a pipeline in Maia automatically when a message arrives on an Azure queue. The Maia queue adapter reads messages from an Azure Service Bus queue or an Azure Queue Storage queue, and starts the pipeline named in each message. This is the Azure equivalent of the SQS pipeline trigger adapter for AWS. Matillion provides the adapter as a packaged container image, so you don’t write any function code. The adapter runs in drain-and-exit mode: each execution processes every message currently on the queue, then stops. A new execution starts when more messages arrive.
You deploy and run the adapter in your own Azure subscription, so you’re responsible for the cost and upkeep of the resources it uses.

How it works

  1. A system you own sends a message to an Azure queue.
  2. Kubernetes Event-driven Autoscaling (KEDA) detects messages on the queue and starts an adapter execution. On a Container App Job, a KEDA scale rule does this. On Azure Kubernetes Service (AKS), a KEDA ScaledJob does it.
  3. The adapter reads the OAuth credentials for the Maia API from Azure Key Vault, using its managed identity.
  4. The adapter looks up the project in an Azure Table Storage mapping table.
  5. For each message, the adapter calls the Maia API to run the pipeline, and then removes the message from the queue.

Prerequisites

You need the following:
  • A Maia account with API credentials.
  • A published pipeline to trigger. The adapter can only run published pipelines.
  • An Azure subscription where you can create managed identities, role assignments, and either Azure Container Apps or an AKS cluster.
  • An Azure Key Vault, an Azure Table Storage account, and a queue (Azure Service Bus or Azure Queue Storage).
  • Terraform, if you deploy to Azure Container Apps with the Terraform module. For AKS, you need Helm and the runner Helm chart. To deploy with the Maia runner ARM template, you need a Maia runner installed with the ARM template.
To deploy on AKS, you must also enable the KEDA and Workload Identity add-ons on the cluster.

Choose a queue backend

The adapter supports two backends. You select one when you deploy it.

Message format

Each message must be JSON in the following format. The format is the same as the AWS adapter.
  • project: The project that contains the pipeline.
  • environment: The environment to run the pipeline in.
  • job: The folder path and name of the pipeline, without a file extension, for example folder/PipelineName. Don’t add .orch.yaml, because the adapter rejects a value that includes a file extension. The value can contain letters, numbers, underscores, hyphens, spaces, and forward slashes (/). The value can’t start or end with a slash.
  • variables: Scalar variables to pass to the pipeline. Use an empty object if you have none.
  • gridVariables: Grid variables to pass to the pipeline. Use an empty object if you have none.

Container image

The adapter image is matillion.azurecr.io/maia-queue-adapter.
  • The image supports amd64 only. Don’t use it on Arm node pools.
  • The registry allows anonymous pulls, so you don’t need registry credentials.
Use adapter version 1.1.5 or later. Earlier versions can’t start pipelines. Available tags:
  • <version>: A specific release.
  • current: The latest release.
Container App Jobs and KEDA ScaledJob resources start a new container for every execution. If you use the current tag, the image can change between two executions. In production, pin a specific <version> tag.

Prepare Azure resources

Store the Maia credentials

Create a Key Vault secret that contains the OAuth credentials for the Maia API as JSON:
For details of creating credentials, read Authenticate to the Maia API. The adapter only runs published pipelines. If the credentials have an account role other than Super Admin, they need access to the project (Viewer or higher) and at least the Operator role on the environment. Operator is the lowest environment role that can run published pipelines—Viewer can’t. Read project roles and environment roles.

Create the project mapping table

The adapter reads project mappings from an Azure Table Storage table. The default table name is matillionprojectmappings. If you deploy as a Container App Job with Terraform or the ARM template, you can let the deployment create the storage account and table, or you can use your own. If you deploy on AKS, you create the table yourself. The table contains one entity for each project and environment combination, with the following properties: The adapter converts the project and environment values in each message to lowercase before it looks them up. The table must therefore store the partition key and row key in lowercase, but the names in the message can use any case.

Assign roles to the managed identity

The adapter authenticates to Azure with a managed identity. If you deploy as a Container App Job with Terraform, the module creates the identity and assigns the roles below, scoped to the queue and the table. If you deploy with the ARM template, the adapter uses the Maia runner‘s managed identity, and the template assigns the roles for any queue and table it creates. If you deploy on AKS, assign these roles yourself to the identity that you set in queueAdapter.serviceAccount.clientId.
For the storage backend, don’t use the Storage Queue Data Message Processor role. It can’t hide a message again for retry or write to the poison queue, so failed messages aren’t handled correctly.

Deploy the adapter

Matillion supports three ways to deploy the adapter:

Deploy as an Azure Container App Job

Use the queue-adapter-job Terraform module in the Matillion deployment library. It runs the adapter as an event-driven Container App Job in your existing Azure Container Apps environment. A KEDA scale rule starts an execution when the queue has messages, and the job scales to zero when the queue is empty. The module creates:
  • A user-assigned managed identity for the job, with the role assignments described in Assign roles to the managed identity.
  • Optionally, a Service Bus namespace and queue, with duplicate detection and dead-lettering. Alternatively, you can use your own queue.
  • Optionally, a storage account and project mapping table. Alternatively, you can use your own.
  • On the storage backend, the source queue and its <queueName>-poison queue.
  • The Container App Job, configured with the adapter image and the KEDA scale rule.
You always create the Key Vault secret yourself. Set the following required inputs:
  • location: The Azure region.
  • resource_group_name: The resource group to create the adapter resources in.
  • container_app_environment_id: The ID of the Container Apps environment to run the job in.
  • image: The adapter image URI and tag.
  • key_vault_id: The resource ID of the Key Vault.
  • key_vault_url: The URL of the Key Vault.
  • secret_name: The name of the secret that contains the Maia credentials.
If you use the public image at matillion.azurecr.io/maia-queue-adapter, leave container_acr_id unset. The registry allows anonymous pulls, so the job needs no registry credentials. Set container_acr_id only if you mirror the image to your own Azure Container Registry. You can also set the following optional inputs: For example:
The KEDA scaler on a Container App Job needs a connection string to read the queue depth. For a queue that the module creates, the module generates a listen-only credential. If you use your own queue, you must supply one in scaler_connection_string. Either way, the value is stored in plaintext in your Terraform state, so keep the state encrypted and restrict access to it.
For the storage backend, the scaler needs a storage account connection string, which can’t be restricted to listen-only access. For this backend, deploy on AKS instead, where the scaler uses workload identity and needs no connection string.

Deploy using the ARM template

Both Azure Maia runner templates—dedicated and consumption—include an optional queue adapter. It’s off by default, so the template creates no adapter resources unless you turn it on. When enabled, the adapter runs as a Container App Job, scaled by KEDA, alongside your Maia runner. It uses the Maia runner‘s managed identity and reads the OAuth credentials from the Maia runner‘s Key Vault. To use it, follow the steps in Maia runner installation using an ARM template, and set the following parameters on the Custom deployment page: If you set createQueueAdapterResources to false, also set the following parameters for your own resources:
  • existingServiceBusNamespaceFqdn: The fully qualified Service Bus namespace, for example <namespace>.servicebus.windows.net.
  • existingStorageTableEndpoint: The Table endpoint of your storage account, for example https://<account>.table.core.windows.net/.
  • existingStorageQueueUrl: On the storage backend, the Queue endpoint of your storage account, for example https://<account>.queue.core.windows.net/.
  • queueAdapterScalerConnectionString: A listen-only Service Bus connection string that the KEDA scaler uses to read the queue depth.
The template doesn’t assign roles on your own resources. Instead, it returns a queueAdapterRequiredRoles output that lists the roles to grant to the Maia runner‘s managed identity. For the full list, read Assign roles to the managed identity.
The queueAdapterScalerConnectionString value is a secret. Keep your deployment history and any parameter files restricted.

Deploy on AKS

Enable the queue adapter in the Maia runner Helm chart. The adapter is disabled by default, so the chart creates no adapter resources until you enable it. When enabled with cloudProvider: azure, the chart deploys a KEDA ScaledJob that starts an adapter job when the queue has messages. Set the following values: Then set the values for your backend. For the servicebus backend, set the following values:
  • queueAdapter.serviceBus.namespace: The fully qualified namespace, for example <namespace>.servicebus.windows.net.
  • queueAdapter.serviceBus.queueName: The queue to read from. The default is matillion-maia-requests.
For the storage backend, set the following values:
  • queueAdapter.storage.queueUrl: The Queue endpoint of the storage account, for example https://<account>.queue.core.windows.net.
  • queueAdapter.storage.queueName: The queue to read from. The default is matillion-maia-requests.
For example, to use the servicebus backend:

Scaling

KEDA scales the adapter on queue depth. You can adjust the following values under queueAdapter.scaling: You can also set CPU and memory requests and limits for the adapter under queueAdapter.resources. For general guidance on deploying Maia runners on AKS, read the Azure AKS deployment guide.

Add a second federated credential for KEDA

On AKS, the KEDA operator reads the queue depth using the adapter’s managed identity. You must add a federated credential to that identity for the KEDA operator’s service account. If you don’t, the ScaledJob reports a Ready status but never starts a job, and messages accumulate on the queue with no error.
The adapter’s managed identity needs two federated credentials:
  • A credential for the adapter’s own service account, which the Helm chart configures.
  • A credential for the KEDA operator’s service account, kube-system/keda-operator on the AKS add-on.
KEDA uses this credential to assume the adapter’s identity and read the queue depth. For example, using the Azure CLI:

Set the region

The adapter calls the Maia API for your region. Set the API URL to https://<region>.api.matillion.com/dpc/v1, where <region> is one of eu1, eu2, us1, or au1. The setting is called matillion_api_url in Terraform, queueAdapter.matillionApiUrl in the Helm chart, and queueAdapterApiUrl in the ARM template.

Test the setup

  1. Publish the pipeline that you want to trigger in Maia.
  2. Send a test message in the message format to the queue.
  3. Check the Your activity page in Maia. The pipeline run appears within a few moments. For more information, read Pipeline run history.
If the pipeline doesn’t start, check the following:
  • On AKS, confirm that you added the second federated credential for KEDA.
  • Confirm that the managed identity has all the roles for your backend.
  • Confirm that the project, environment, and pipeline names in the message match your Maia project. Project and environment names aren’t case-sensitive, but the project mapping table must contain a matching entity in lowercase.
  • Check the poison queue for messages that the adapter couldn’t process.

Monitor failed messages

When the adapter can’t process a message, it moves the message to the dead-letter queue (servicebus) or the <queueName>-poison queue (storage). On a Container App Job, a run that dead-letters a message exits with an error, and Container Apps retries it. The retry finds the queue already cleared and exits successfully, so the execution shows as Succeeded even though a message was discarded. Alert on the depth of the poison queue or on the adapter’s MessagesDeadLettered metric, not on the job execution status.