> ## Documentation Index > Fetch the complete documentation index at: https://docs.maia.ai/llms.txt > Use this file to discover all available pages before exploring further. # Maia runner installation using an ARM template export const s_runner = "Streaming runner"; export const m_runner = "Maia runner"; export const maia = "Maia"; export const RunnerMetadata = ({runnerType, platforms = []}) => { return
Runner type {runnerType}
Runner platform
{platforms.map((platform, i) => {platform} ✅ )}
; }; This topic details how to install a Matillion {m_runner} in your Azure infrastructure using an ARM template provided by Matillion. You can also, if you wish, [download the supplied ARM template](/docs/guides/amend-azure-arm-template) and edit as needed. If you are installing a **{s_runner}**, read [{s_runner} installation on Azure](/docs/streaming/azure-streaming-agent-install). If you require multiple {m_runner}s, each one must be installed into a different resource group. For troubleshooting and frequently asked questions encountered in this process, read [Azure {m_runner} troubleshooting](/docs/guides/azure-runner-troubleshooting). *** ## Prerequisites We recommend that you read the documentation and prerequisites before beginning this process. You may require input from your organization's cloud administrator for access and permissions. To get started, you'll need: * An Azure subscription with appropriate permissions to provision cloud resources in the Azure environment and manage access control, specifically for managing resource groups, virtual networks, key vaults, and container apps. Read [ARM template permissions](/docs/guides/arm-template-permissions) for details. * A suitable [NAT gateway](#create-a-nat-gateway) already defined in your Azure environment. Using a NAT gateway is recommended because it doesn't require a public IP address. Azure can dynamically rotate the public IP for each container, which can restart the {m_runner}, interrupt pipelines, and complicate IP whitelisting. * A suitable [resource group](#create-a-resource-group) already defined in your Azure environment. * A suitable [virtual network](#create-a-virtual-network) already defined in your Azure environment, to secure access between the {m_runner} and other resources, reducing the scope of possible access to those resources. The virtual network must: * Be fully configured as desired, including routing to on-premises resources. * Allow egress to Matillion's [{maia} IP ranges](/docs/security/network-access-and-ip-allowlist-requirements/). * Have room for one additional subnet with at least `/27` IP range. For purposes of the {m_runner}, `/27` provides sufficient usable IPs for use by {m_runner} container replicas for normal operation, and to allow extra IPs for new nodes to come online in a rolling restart/update. Read [Environment selection](https://learn.microsoft.com/en-us/azure/container-apps/networking?tabs=consumption-only-env%2Cazure-cli#environment-selection) in the Azure documentation for further details. * Optionally, a suitable [key vault](#create-a-key-vault) already defined in your environment. This isn't a mandatory requirement, as the installation process will create a new key vault by default. * Minimum permissions to include the following: * Create: * Subnet * Managed identity * Key Vault * Log Analytics Workspace * Container App & Container App Environment * Modify: * Subnet delegation * Permissions: * Role assignments in the resource group * Key Vault Read [ARM template permissions](/docs/guides/arm-template-permissions) for full details of the fine-grained permissions required. This document includes instructions for creating a [resource group](#create-a-resource-group), [virtual network](#create-a-virtual-network), [key vault](#create-a-key-vault), and [NAT gateway](#create-a-nat-gateway), if you don't already have these resources. In all cases, consult Azure's own documentation for further details. The container app for the {m_runner} *must* be deployed with a **Dedicated** workload profile type. Otherwise, the {m_runner} will be unable to update without manual intervention when Matillion issues an updated version. SSL inspection and rewriting for outbound connections (e.g. by network security appliances) can prevent the Matillion {m_runner} from launching or connecting to critical external endpoints, such as the Matillion control plane. If your environment uses appliances that rewrite SSL connections, make sure that any root certificates used for this SSL rewriting are provided to the {m_runner}. For more information, read [Customizing {m_runner} networking and connectivity](/docs/guides/customizing-runner-networking-and-connectivity). *** ## Completing the installation 1. Ensure you have created the appropriate [prerequisites](#prerequisites) in your Azure environment. 2. On the **Virtual networks** screen, locate your virtual network. 3. Click **Service endpoints**. 4. Click **+ Add** and select the required service endpoints from the **Service** drop-down. These are required if the subnet doesn't have access to the Internet (via Internet gateway or NAT gateway). As a minimum, the following service endpoints will be required: * Microsoft.KeyVault * Microsoft.Storage * Microsoft.ContainerRegistry 5. Create a NAT gateway. For instructions, read [create a NAT gateway](#create-a-nat-gateway). 6. Select a subnet from the **Subnets** drop-down. The subnet must have been previously created via the Azure CLI using the following command: ``` az network vnet subnet create \ --resource-group \ --vnet-name \ --name \ --address-prefix 10.0.2.0/27 \ --service-endpoints Microsoft.Storage Microsoft.KeyVault Microsoft.ContainerRegistry \ --delegations Microsoft.App/environments \ --default-outbound-access false \ --nat-gateway ``` The `--default-outbound-access false` flag prevents Azure from automatically assigning a public IP address to the {m_runner}. This must be configured *before* {m_runner} deployment. If you add this flag after the {m_runner} is already deployed, you must redeploy the {m_runner} for the setting to take effect. Note the subnet's full ID. 7. If desired, add a [NAT gateway](#create-a-nat-gateway). Apply any security groups and user-defined routing that your network configuration may require; for example, to achieve connectivity to the Internet, Azure resources, and resources available via site-to-site VPN. For a full discussion of these topics, read [Networking in Azure Container Apps environment](https://learn.microsoft.com/en-us/azure/container-apps/networking?tabs=workload-profiles-env%2Cazure-cli) in the Azure documentation. 8. In {maia}, follow the steps to [create a new {m_runner}](/docs/guides/create-a-runner), with **Azure** as the cloud provider and **Container App** as the deployment type. 9. On the **Runner details** screen, click **Reveal credentials** in the **Credentials** section and note the **client\_id** and **client\_secret**. 10. Click the **Launch** button in the **Install using ARM** section. This will open the Azure portal at the **Custom deployment** page in a new browser tab. 11. Complete the following details on the **Custom deployment** page. | Property | Description | | -------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | | Subscription | Select the subscription you will deploy the {m_runner} into. | | Resource group | Select the [resource group you created previously](#create-a-resource-group). | | Region | Select the Azure region you will deploy the {m_runner} to, or leave the default value to deploy the {m_runner} in the same region as the selected resource group. | | Create Key Vault | Select **true** if you want the ARM template to create a new key vault. This is the recommended method. If you prefer to use a key vault you have created yourself, select **false**. | | Create Managed Identity | Select **true** if you want the ARM template to create a new [managed identity](https://learn.microsoft.com/en-us/entra/identity/managed-identities-azure-resources/overview). | | Resource Name | Enter a unique and descriptive name for the {m_runner} resource. The given name will be used for the other required resources (for example, Log Analytics Workspace) and optional resources (for example, Key Vault, Identity). | | Location | This is set automatically by the template and shouldn't be changed. | | Matillion Cloud Region | This is set automatically by the template and shouldn't be changed. It should be the same as the **MATILLION\_REGION** parameter shown on the **Runner details** screen. | | Managed Identity Name | The name of the managed identity that will be associated to the container app. Must be in the same resource group. | | Container App Name | This is set automatically by the template and shouldn't be changed. | | Key Vault Name | If you are using an existing key vault, enter its name here. If you are allowing the ARM template to create the key vault, **don't** edit the parameter string displayed in this field. | | Key Vault Name Resource Group | If you are using an existing key vault, enter the name of its resource group here. If you are allowing the ARM template to create the key vault, **don't** edit the parameter string displayed in this field. | | Subnet ID | Enter the ID of the subnet you created previously. | | Account ID | This is set automatically by the template and shouldn't be changed. It should be the same as the **ACCOUNT\_ID** parameter shown on the **Runner details** screen. | | {m_runner} ID | This is set automatically by the template and shouldn't be changed. It should be the same as the **AGENT\_ID** parameter shown on the **Runner details** screen. | | Client ID | Enter the **client\_id** credential you obtained from the **Runner details** screen. | | Client Secret | Enter the **client\_secret** credential you obtained from the **Runner details** screen. | | Container Image URL | The full path used to locate and pull a container image. A typical container image URL structure is `/:`. For example, `myregistry.azurecr.io/myapp/webapp:latest`. We strongly suggest using the default value to leverage Matillion's published images (for Azure, this is `matillion.azurecr.io/cloud-runner:`, where `` refers to one of our image tracks). You may need or prefer to use a self-hosted copy of the Matillion image. This requires some careful planning related to {m_runner} management (such as starting, stopping, automatic updates, and manually updating). If self hosting, the Container Image URL to use will be **your** repo or container image's URL, and you must provide a container registry username and password (see below) if applicable. | | Container Registry Username | This can be left blank, as it's not required for Matillion's repo. It may be required if using a self-hosted image (as described above). | | Container Registry Password Secret Ref | This can be left blank, as it's not required for Matillion's repo. It may be required if using a self-hosted image (as described above). | | Tags | Optionally, add any tags that your organization's tagging policy requires. Tags should be added as a JSON string, for example: `{"Category": "Development","Business Unit": "Data Pipelines","Owner": "owner@organization.com"}`. | The following properties are optional and should only be set if needed. For more information on these, read [Optional {m_runner} parameters](/docs/guides/optional-runner-parameters). | Property | Description | | --------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | | Proxy HTTP | If using a proxy, enter the HTTP proxy details. For example: `http://myproxy.com:8080`. | | Proxy HTTPS | If using a proxy, enter the HTTPS proxy details. For example: `https://myproxy.com:8443`. | | Proxy Excludes | If using a proxy, specify any addresses to exclude. Separate a list of addresses with the pipe character `\|`. For example: `example.com\|example.net`. | | Custom Certificate Location | If using custom certificates, specify the storage location. For example: `my_storage/my_certs`. The location refers to an Azure Blob Storage folder's URI that the managed identity has been or will be given access to. | | Extension Library Location | Specify the location of any additional Python libraries you want to use. The location refers to an Azure Blob Storage folder's URI that the managed identity has been or will be given access to. | | External Driver Location | Specify the location of any external driver files you want to use. The location refers to an Azure Blob Storage folder's URI that the managed identity has been or will be given access to. | When specifying a value for `External Driver Location`, the driver files and manifest file must be placed at the root level of the container. Placing these files within a subfolder will cause the {m_runner} installation to fail. For more information on setting up driver files correctly, read [Managing external drivers](/docs/guides/uploading-external-drivers). 12. Click **Review & Create**. It will take at least 10 minutes for the container app to be created and {m_runner} application to load and initialize. Return to the **Runners** list in {maia}, and ensure that the {m_runner}'s status is **Running**. ### Deploying multiple Maia runner instances You can rerun the template to create additional {m_runner}s in the same resource group using a previously created managed identity and role assignment. Alternatively, you can pre-create the managed identity and custom roles with the desired permissions as a prerequisite step, then select the pre-created identity during template launch. * Multiple {m_runner}s can be deployed into the same resource group, but each {m_runner} must be deployed into a separate subnet due to Azure limitations. * If using an existing managed identity, verify that it has the necessary permissions before proceeding. *** ## Create a resource group You need an Azure resource group that will contain all of the Azure resources for the {m_runner}. The {m_runner} deployment template requests the name of the resource group to deploy into and will create all resources there. If you don't have a suitable existing resource group, create a new one as follows: 1. Log in to the [Azure Portal](https://portal.azure.com/#home). 2. Use the search bar to search for "resource groups", and click the **Resource groups** result. 3. On the **Resource groups** screen, click **+ Create**. 4. On the **Create resource group** screen, select the **Subscription** you want to use for the {m_runner} from the drop-down. 5. Enter a new name for the **Resource group**. 6. Select an Azure **Region** for your resource group. 7. If your organization has a policy that mandates resource group tags, click **Tags** and assign tags as required. 8. Click **Review + create**, and then click **Create** if you are satisfied with your resource group configuration. For more information, read [Manage Azure resource groups by using the Azure portal](https://learn.microsoft.com/en-us/azure/azure-resource-manager/management/manage-resource-groups-portal) in the Microsoft developer documentation. *** ## Create a virtual network You need a virtual network to secure access between the {m_runner} and other resources, reducing the scope of possible access to those resources. If you don't have a suitable existing virtual network, create a new one as follows: 1. Log in to the [Azure Portal](https://portal.azure.com/#home). 2. Use the search bar to search for "virtual networks", and click the **Virtual networks** result. 3. On the **Virtual networks** screen, click **+ Create**. 4. On the **Create virtual network** screen, select the [resource group](#create-a-resource-group) you created previously. 5. Enter a new **Virtual network name**. 6. Select an Azure **Region** for your virtual network. By default, this should be the same region as your resource group. 7. If your organization has a policy that mandates virtual network tags, click **Tags** and assign tags as required. 8. Click **Review + create**, and then click **Create** if you are satisfied with your virtual network configuration. *** ## Create a key vault The {m_runner} requires access to an Azure key vault. However, you don't need to create a key vault yourself; the deployment template will automatically create a key vault in the same resource group as the {m_runner}. If you prefer, the {m_runner} can reference a different key vault you have previously created. The existing key vault may be in the same or a different resource group. You might want to reference a key vault in a different resource group so that you can have multiple {m_runner}s referencing the same secrets. If you are using an existing key vault, you will need to complete the configuration steps described in [Configuring a key vault for Azure {m_runner}](/docs/guides/azure-arm-runner-key-vault). *** ## Create a NAT gateway 1. Log in to the [Azure Portal](https://portal.azure.com/#home). 2. Use the search bar to search for "nat gateways", and click the **NAT gateways** result. 3. Click **Create NAT gateway**. 4. On the **Basics** tab, enter details of the **Subscription**, **Resource group**, and **Region**, ensuring that these match the details you have used for your [virtual network](#create-a-virtual-network). Enter a suitable **Name** for the gateway, and keep the suggested defaults for **Availability zone** (No Zone) and **TCP idle timeout** (4 minutes). 5. Click **Next: Outbound IP**. 6. On the **Outbound IP tab** drop-down, click **Create a new public IP address**. Provide a **Name** for the IP address and click **OK**. 7. Click **Next: Subnet**. 8. On the **Subnet** tab, select the **Virtual network** containing the target subnet. In the list of subnets that appears, select the required subnet. 9. If your organization has a policy that mandates resource group tags, click **Next: Tags** and assign tags as required. 10. Click **Review + create**, and then click **Create** if you are satisfied with the configuration.