Setup guide - Hybrid SaaS Databricks on Azure

This document describes the necessary steps to follow to set up your first working project in Maia for the following configuration options:

---

​

Video example

Expand this box to watch our video on how to setup Hybrid SaaS on Azure.

Video

---

​

Prerequisites

​

Azure requirements

• An Azure subscription with appropriate permissions to provision cloud resources in the Azure environment and manage access control, specifically for managing:
  • Resource groups.
  • Virtual networks.
  • Key vaults.
  • Container apps.
• A suitable resource group already defined in your Azure environment.
• Optionally, a suitable NAT gateway already defined in your Azure environment. We recommend defining a NAT gateway, but this isn’t a mandatory requirement, as Azure automatically assigns the Maia runner a public IP address if one isn’t present, which it can dynamically rotate for each container, restarting the Maia runner, interrupting pipelines, and complicating IP allowlisting.
• A suitable virtual network already defined in your Azure environment. The virtual network must:
  • Be fully configured, including routing to on-premises resources.
  • Allow egress to Matillion’s Maia IP ranges.
  • Have room for one additional subnet with at least /27 IP range.
  • Have default outbound access disabled on the subnet, to prevent Azure from automatically assigning a public IP address to the Maia runner.
• A suitable key vault. You can use an existing key vault or a new one will be created as part of the setup process.
• The container app for the Maia runner must be deployed with a Dedicated workload profile type. Otherwise, updates require manual intervention when Matillion issues a new version.
• Minimum permissions to include the following:
  • Create subnet.
  • Create managed identity.
  • Create key vault.
  • Create log analytics workspace.
  • Create container app and container app environment.
  • Modify subnet delegation.
• Role assignments in the resource group and key vault.

​

Databricks requirements

• A Databricks account with the following information:
  • Your Databricks instance name.
  • Your Databricks personal access token.

​

Connectivity requirements

• Access enabled for the IP addresses listed under the Hybrid SaaS section of Network access and IP Allowlist requirements.

​

Git requirements

If you choose to use your own Git provider instead of the Maia Hosted Git option, you need the following:

• The Maia app installed in your organization’s account with one of the supported Git providers:
  • GitHub.
  • Azure DevOps.
  • GitLab.
  • Bitbucket.

---

​

Setup steps

1. Register for a Maia account.
2. Create accounts for users and admins who will be active in Maia.
3. Create a Maia runner in Maia.
4. Deploy a Container App Maia runner in Azure using the recommended ARM template.
5. Create a project, making the following choices:
   • Select Advanced settings.
   • Select the Maia runner you created and deployed previously.
   • Select the Git provider you wish to use.
6. Create an environment using your Databricks credentials.
7. Set up secret definitions for passwords, API keys, and tokens.
8. Create a Git branch in which to begin pipeline work.
9. Create your first pipeline.