Skip to main content

Documentation Index

Fetch the complete documentation index at: https://docs.maia.ai/llms.txt

Use this file to discover all available pages before exploring further.

This topic explains how to create a in your own infrastructure. For a discussion of why you might want to deploy in your own infrastructure rather than use a Matillion Full SaaS solution, read overview. We currently support s running in Snowflake, AWS, or Azure environments. There are two different types of : This article covers s only. For s, read Create a . The installation process is similar for both types of , but you must select the correct type for the type of pipeline you want to run. You can install both s in the same infrastructure if you intend to run both types of pipeline. Read Scaling best practices to understand how to size your for the level of performance you need.

​
Prerequisites

  • A account. To register, read Registration. Once you have signed up, log in to .
  • An account in either Azure or AWS to host the .

​
Create a Maia runner

  1. In the left navigation, click s. Then, select Runners from the menu. You will see a list of all s currently created.
  2. Click Add runner.
  3. Click .
  4. Complete the following properties:
    • Runner name: A unique name for your new . Maximum 30 characters. Accepts both uppercase and lowercase A-z, 0-9, whitespace (not the first character), hyphens and underscores.
    • Description: Optionally enter a brief description of the .
    • Version track: Select the version track that this will use: Current or Stable. Read Version tracks for details of this option.
    • Cloud provider: The cloud platform that the will be deployed to. Currently, Snowflake, AWS, and Azure are supported.
    • Deployment: The supported deployment method for the given cloud provider. Currently, Native App for Snowflake, Fargate for AWS, and Container App/ACI for Azure are supported.
    • Auto update: Select whether you want your to automatically update or not. Read Auto update for details of this option.
  5. Click Create runner.

​
Set up the Maia runner in your cloud infrastructure

When created with the above process, the needs to be set up in your cloud infrastructure. There are several different ways of doing this, and you can use whichever method suits you: To complete these processes, you will require certain details of the created . To obtain these details, locate the in the Runners list of s, and click the three dots … next to it, then click Runner details. The parameters and values in the sections Runner image URI, Runner environment variables, and Credentials are required when configuring your in your cloud infrastructure.

​
Check Maia runner status

After deploying the in your cloud infrastructure, you should return to to verify that it’s correctly connected and running.
  1. Click the menu button in the top left of any page, then click Manage runners.
  2. Locate the in the list and check the status:
    • Running: The is connected and ready for development and pipeline tasks.
    • Unknown: The is in an unknown state.
    • Pending: The has been created but has not yet connected to .
    • Stopped: The has been stopped.
  3. When the status shows Running, it’s ready to use.

​
Restarting your Maia runner

There are a number of reasons why you may need to restart your , such as receiving updates. To restart a , follow the steps in Restart a . In some cases, you may need or choose to pause the before restarting.

​
Deleting a Maia runner

To delete a from , locate the in the Runners list, then click the three dots … → Remove runner. Deleting the from the Runners list doesn’t remove the underlying Snowflake, AWS, or Azure resources. You should go into the Snowpark Container Services, AWS Console, or Azure Portal, and clean up any resources that you no longer require.
  • Snowflake: Use Snowpark Container Services to manage and clean up any Snowflake resources associated with the .
  • AWS: Use the AWS Console to identify and delete any resources associated with the , such as EC2 instances and IAM roles.
  • Azure: Use the Azure Portal to locate and remove any related resources, such as virtual machines and resource groups.
Deleting a that is currently running may interrupt scheduled pipelines or pipelines that are currently running. Therefore, you should always stop the service before deleting it.

​
Using the Maia runner to develop and run pipelines

To use a for developing and then running pipelines, you need to select the when you create an environment. See below for details of how to make s available to specific environments. You can change which an environment uses at any time by editing the environment and selecting a different . This is safe to do, as the change will only affect pipelines that are run after the change. So:
  • Your open session won’t change.
  • Any pipelines that are currently running will continue to run using the old .
  • Any existing schedules will continue to run using the old . To use the new , you would have to change the schedule.

​
Restricting a Maia runner

You can restrict which s are available to a specific project or environment. This is useful if you have multiple s and want to ensure that a project only uses specific ones. An unrestricted (the default state when a is created) can always be used by any project or environment. A restricted can only be used by the projects and environments you specify. If you restrict a but don’t specify any projects, the won’t be accessible to any project. For details of how to restrict a , read Restricting s.

​
Video