You deploy and run the adapter in your own Azure subscription, so you’re responsible for the cost and upkeep of the resources it uses.
How it works
- A system you own sends a message to an Azure queue.
- Kubernetes Event-driven Autoscaling (KEDA) detects messages on the queue and starts an adapter execution. On a Container App Job, a KEDA scale rule does this. On Azure Kubernetes Service (AKS), a KEDA
ScaledJobdoes it. - The adapter reads the OAuth credentials for the Maia API from Azure Key Vault, using its managed identity.
- The adapter looks up the project in an Azure Table Storage mapping table.
- For each message, the adapter calls the Maia API to run the pipeline, and then removes the message from the queue.
Prerequisites
You need the following:- A Maia account with API credentials.
- A published pipeline to trigger. The adapter can only run published pipelines.
- An Azure subscription where you can create managed identities, role assignments, and either Azure Container Apps or an AKS cluster.
- An Azure Key Vault, an Azure Table Storage account, and a queue (Azure Service Bus or Azure Queue Storage).
- Terraform, if you deploy to Azure Container Apps with the Terraform module. For AKS, you need Helm and the runner Helm chart. To deploy with the Maia runner ARM template, you need a Maia runner installed with the ARM template.
Choose a queue backend
The adapter supports two backends. You select one when you deploy it.Message format
Each message must be JSON in the following format. The format is the same as the AWS adapter.project: The project that contains the pipeline.environment: The environment to run the pipeline in.job: The folder path and name of the pipeline, without a file extension, for examplefolder/PipelineName. Don’t add.orch.yaml, because the adapter rejects a value that includes a file extension. The value can contain letters, numbers, underscores, hyphens, spaces, and forward slashes (/). The value can’t start or end with a slash.variables: Scalar variables to pass to the pipeline. Use an empty object if you have none.gridVariables: Grid variables to pass to the pipeline. Use an empty object if you have none.
Container image
The adapter image ismatillion.azurecr.io/maia-queue-adapter.
- The image supports amd64 only. Don’t use it on Arm node pools.
- The registry allows anonymous pulls, so you don’t need registry credentials.
<version>: A specific release.current: The latest release.
Prepare Azure resources
Store the Maia credentials
Create a Key Vault secret that contains the OAuth credentials for the Maia API as JSON:Create the project mapping table
The adapter reads project mappings from an Azure Table Storage table. The default table name ismatillionprojectmappings.
If you deploy as a Container App Job with Terraform or the ARM template, you can let the deployment create the storage account and table, or you can use your own. If you deploy on AKS, you create the table yourself.
The table contains one entity for each project and environment combination, with the following properties:
The adapter converts the
project and environment values in each message to lowercase before it looks them up. The table must therefore store the partition key and row key in lowercase, but the names in the message can use any case.
Assign roles to the managed identity
The adapter authenticates to Azure with a managed identity. If you deploy as a Container App Job with Terraform, the module creates the identity and assigns the roles below, scoped to the queue and the table. If you deploy with the ARM template, the adapter uses the Maia runner‘s managed identity, and the template assigns the roles for any queue and table it creates. If you deploy on AKS, assign these roles yourself to the identity that you set inqueueAdapter.serviceAccount.clientId.
Deploy the adapter
Matillion supports three ways to deploy the adapter:- As an Azure Container App Job, using Terraform.
- As part of an Maia runner installation on Azure Container Apps, using the ARM template (or Bicep).
- On AKS, using the runner Helm chart.
Deploy as an Azure Container App Job
Use thequeue-adapter-job Terraform module in the Matillion deployment library. It runs the adapter as an event-driven Container App Job in your existing Azure Container Apps environment. A KEDA scale rule starts an execution when the queue has messages, and the job scales to zero when the queue is empty.
The module creates:
- A user-assigned managed identity for the job, with the role assignments described in Assign roles to the managed identity.
- Optionally, a Service Bus namespace and queue, with duplicate detection and dead-lettering. Alternatively, you can use your own queue.
- Optionally, a storage account and project mapping table. Alternatively, you can use your own.
- On the
storagebackend, the source queue and its<queueName>-poisonqueue. - The Container App Job, configured with the adapter image and the KEDA scale rule.
location: The Azure region.resource_group_name: The resource group to create the adapter resources in.container_app_environment_id: The ID of the Container Apps environment to run the job in.image: The adapter image URI and tag.key_vault_id: The resource ID of the Key Vault.key_vault_url: The URL of the Key Vault.secret_name: The name of the secret that contains the Maia credentials.
matillion.azurecr.io/maia-queue-adapter, leave container_acr_id unset. The registry allows anonymous pulls, so the job needs no registry credentials. Set container_acr_id only if you mirror the image to your own Azure Container Registry.
You can also set the following optional inputs:
For example:
For the
storage backend, the scaler needs a storage account connection string, which can’t be restricted to listen-only access. For this backend, deploy on AKS instead, where the scaler uses workload identity and needs no connection string.Deploy using the ARM template
Both Azure Maia runner templates—dedicated and consumption—include an optional queue adapter. It’s off by default, so the template creates no adapter resources unless you turn it on. When enabled, the adapter runs as a Container App Job, scaled by KEDA, alongside your Maia runner. It uses the Maia runner‘s managed identity and reads the OAuth credentials from the Maia runner‘s Key Vault. To use it, follow the steps in Maia runner installation using an ARM template, and set the following parameters on the Custom deployment page:
If you set
createQueueAdapterResources to false, also set the following parameters for your own resources:
existingServiceBusNamespaceFqdn: The fully qualified Service Bus namespace, for example<namespace>.servicebus.windows.net.existingStorageTableEndpoint: The Table endpoint of your storage account, for examplehttps://<account>.table.core.windows.net/.existingStorageQueueUrl: On thestoragebackend, the Queue endpoint of your storage account, for examplehttps://<account>.queue.core.windows.net/.queueAdapterScalerConnectionString: A listen-only Service Bus connection string that the KEDA scaler uses to read the queue depth.
queueAdapterRequiredRoles output that lists the roles to grant to the Maia runner‘s managed identity. For the full list, read Assign roles to the managed identity.
Deploy on AKS
Enable the queue adapter in the Maia runner Helm chart. The adapter is disabled by default, so the chart creates no adapter resources until you enable it. When enabled withcloudProvider: azure, the chart deploys a KEDA ScaledJob that starts an adapter job when the queue has messages.
Set the following values:
Then set the values for your backend.
For the
servicebus backend, set the following values:
queueAdapter.serviceBus.namespace: The fully qualified namespace, for example<namespace>.servicebus.windows.net.queueAdapter.serviceBus.queueName: The queue to read from. The default ismatillion-maia-requests.
storage backend, set the following values:
queueAdapter.storage.queueUrl: The Queue endpoint of the storage account, for examplehttps://<account>.queue.core.windows.net.queueAdapter.storage.queueName: The queue to read from. The default ismatillion-maia-requests.
servicebus backend:
Scaling
KEDA scales the adapter on queue depth. You can adjust the following values underqueueAdapter.scaling:
You can also set CPU and memory requests and limits for the adapter under
queueAdapter.resources.
For general guidance on deploying Maia runners on AKS, read the Azure AKS deployment guide.
Add a second federated credential for KEDA
The adapter’s managed identity needs two federated credentials:- A credential for the adapter’s own service account, which the Helm chart configures.
- A credential for the KEDA operator’s service account,
kube-system/keda-operatoron the AKS add-on.
Set the region
The adapter calls the Maia API for your region. Set the API URL tohttps://<region>.api.matillion.com/dpc/v1, where <region> is one of eu1, eu2, us1, or au1.
The setting is called matillion_api_url in Terraform, queueAdapter.matillionApiUrl in the Helm chart, and queueAdapterApiUrl in the ARM template.
Test the setup
- Publish the pipeline that you want to trigger in Maia.
- Send a test message in the message format to the queue.
- Check the Your activity page in Maia. The pipeline run appears within a few moments. For more information, read Pipeline run history.
- On AKS, confirm that you added the second federated credential for KEDA.
- Confirm that the managed identity has all the roles for your backend.
- Confirm that the project, environment, and pipeline names in the message match your Maia project. Project and environment names aren’t case-sensitive, but the project mapping table must contain a matching entity in lowercase.
- Check the poison queue for messages that the adapter couldn’t process.
Monitor failed messages
When the adapter can’t process a message, it moves the message to the dead-letter queue (servicebus) or the <queueName>-poison queue (storage).
On a Container App Job, a run that dead-letters a message exits with an error, and Container Apps retries it. The retry finds the queue already cleared and exits successfully, so the execution shows as Succeeded even though a message was discarded. Alert on the depth of the poison queue or on the adapter’s MessagesDeadLettered metric, not on the job execution status.