> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maia.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Maia runner overview

export const s_runner = "Streaming runner";

export const m_runner = "Maia runner";

export const designer = "Designer";

export const maia = "Maia";

export const RunnerMetadata = ({runnerType, platforms = []}) => {
  return <div style={{
    background: 'var(--colors-background-light, #f9fafb)',
    border: '1px solid var(--colors-border-default, #e5e7eb)',
    borderRadius: '12px',
    padding: '20px 28px',
    marginBottom: '28px'
  }}>
      <table style={{
    width: '100%',
    borderCollapse: 'collapse'
  }}>
        <tbody>
          <tr>
            <td style={{
    fontWeight: '600',
    paddingRight: '32px',
    paddingBottom: '14px',
    whiteSpace: 'nowrap',
    verticalAlign: 'middle',
    width: '180px'
  }}>Runner type</td>
            <td style={{
    paddingBottom: '14px',
    verticalAlign: 'middle'
  }}>{runnerType}</td>
          </tr>
          <tr>
            <td style={{
    fontWeight: '600',
    paddingRight: '32px',
    whiteSpace: 'nowrap',
    verticalAlign: 'middle'
  }}>Runner platform</td>
            <td style={{
    verticalAlign: 'middle'
  }}>
              <div style={{
    display: 'flex',
    flexWrap: 'wrap',
    gap: '8px'
  }}>
                {platforms.map((platform, i) => <span key={i} style={{
    background: '#dcfce7',
    color: '#15803d',
    border: '1px solid #bbf7d0',
    borderRadius: '9999px',
    padding: '3px 12px',
    fontSize: '0.85rem',
    fontWeight: '500',
    whiteSpace: 'nowrap'
  }}>
                    {platform} ✅
                  </span>)}
              </div>
            </td>
          </tr>
        </tbody>
      </table>
    </div>;
};

<RunnerMetadata runnerType={`${maia} Hybrid | ${maia} Full SaaS`} platforms={["AWS", "Azure", "Google Cloud", "Snowflake", "Matillion"]} />

The {m_runner} is a component within {maia}. It serves as a bridge between the user's data plane and the centralized platform, enabling the execution and scheduling of compatible pipelines.

The key characteristics of a {m_runner} include:

* **Execution and scheduling:** The {m_runner} enables the execution of pipelines within {maia}. It acts as the engine that carries out the tasks defined in the pipelines.
* **Data locality:** Your data will remain in the region your {m_runner} is running in, allowing you to conform to data locality limitations set by your organization or law.
* **Scalability:** The {m_runner} can scale up by running multiple instances concurrently, allowing for increased workload handling based on the user's requirements.

<Note>
  You need a {s_runner} to run [Streaming pipelines](/docs/streaming/streaming-pipelines/). This {s_runner} type has its own architecture considerations and installation procedure, which are covered under [Create a {s_runner}](/docs/streaming/create-streaming-agent).
</Note>

***

## Video example

Expand this box to watch our video about {m_runner}s.

<iframe width="560" height="315" src="https://www.youtube.com/embed/EmOrXY6chbE?si=nY_HWFiKp1_EGONi&enablejsapi=1" title="YouTube video player" frameBorder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture; web-share; fullscreen" referrerPolicy="strict-origin-when-cross-origin" allowFullScreen />

***

## Architecture

<img src="https://mintcdn.com/matillion/I06T4ygCZYYqgb0_/images/agent-overview/agent-overview-01.png?fit=max&auto=format&n=I06T4ygCZYYqgb0_&q=85&s=acd07cfcd2f35dcb925bf8052b83e686" alt="Runner deployment architecture" width="2706" height="882" data-path="images/agent-overview/agent-overview-01.png" />

The {m_runner} is required for {designer} to schedule and execute the pipelines you create. While {m_runner}s can be [installed manually on your cloud platform](/docs/guides/create-a-runner), a Matillion [Full SaaS](#matillion-full-saas) deployment is provided for you when you create your first {designer} project and this {m_runner} (hosted by Matillion) can be seen from the **Manage runners** page in {maia}. Thus, you don't need to install your own {m_runner} if you don't want to.

The {m_runner} is based on Ubuntu Linux and deployed to supported runtimes, including AWS, Azure, and Google Cloud.

***

## What is a data plane?

A data plane refers to an abstract concept that represents an execution environment. This execution environment, such as AWS Fargate, is used in conjunction with a {m_runner} provided by Matillion as a Docker image.

The data plane serves as the infrastructure or environment where the {m_runner} instances operate. The {m_runner} instances, when deployed within the data plane, collectively form the {m_runner}. Each {m_runner} instance is responsible for executing individual pipeline tasks.

These pipeline tasks typically involve interactions with various data sources, including a data warehouse. The data plane provides the necessary resources and infrastructure for the {m_runner} instances to execute the assigned pipeline tasks and perform data-related operations.

***

## What is Maia runner?

The {m_runner} is a software component responsible for executing the pipeline tasks within the specified environment. It serves as the execution engine for the pipelines and interacts with the runner gateway to receive the pipeline tasks that need to be executed. The {m_runner} communicates with the runner gateway using an egress-only method, meaning it can send requests and receive responses but can't accept incoming connections.

The pipeline tasks that are sent to the {m_runner} originate from {maia} platform. These tasks define the specific operations and transformations to be performed on the data within the pipeline.

***

## What is Maia runner manager?

The runner manager is an application or tool that facilitates the creation, configuration, and management of {m_runner}s within {maia} ecosystem. It is typically accessed through {maia}, where you can manage various aspects of the data pipelines.

With the runner manager, users can create and define configurations for installing and deploying {m_runner}s. This includes specifying settings such as the {m_runner} image, environment variables, resource allocation, and any other required parameters. The runner manager provides a user-friendly interface or set of commands to simplify the process of configuring and setting up {m_runner}s.

<Note>
  The runner manager offers functionality for managing {m_runner}s in compatible environments like AWS ECS Fargate. This includes features such as **automatic updates**, where the runner manager handles the process of updating {m_runner}s to newer versions seamlessly. This ensures that {m_runner}s deployed in compatible environments remain up to date with the latest features, bug fixes, and security patches. This means {m_runner} updates are controlled by Matillion, and as such, the user isn't required to manually update the {m_runner} themselves.
</Note>

***

## Matillion Full SaaS vs Hybrid SaaS

{maia} provides two deployment models: Full SaaS and Hybrid SaaS.

* With Full SaaS, Matillion manages the entire infrastructure, including {m_runner} deployment and security measures. Matillion ensures seamless updates and robust security protocols. The Matillion-hosted {m_runner} handles the execution of tasks, and customer secrets are securely stored in a Matillion-hosted secrets manager.
* With Hybrid SaaS, you deploy and manage your own {m_runner}s within your own cloud infrastructure. This gives you full control over security measures, network isolation, access control, and where your secrets are stored. This option is available for [Enterprise](/docs/administration/editions) edition customers only.

Regardless of the deployment model, your data remains secure at all times and is never visible to Matillion. Read [Security overview](/docs/security/security-overview) for a more detailed explanation of Matillion's security framework.

Each deployment model offers distinct features and benefits, allowing organizations to choose the option that best aligns with their needs and infrastructure preferences. Each deployment model also comes with its own security considerations. For a full discussion of the deployment architecture and security considerations, read [Deployment options](/docs/security/deployment-options).

Every {maia} account includes a Full SaaS {m_runner}, hosted by Matillion, offering a zero-install and zero-maintenance experience. By starting with Full SaaS, you can quickly leverage {maia}'s capabilities without any setup. As your requirements become more sophisticated, you can move to the Hybrid SaaS model and customize it to your unique infrastructure, if you wish.

Due to considerations such as security and data flow, some [pipeline components](/docs/guides/components-overview) can only be used in a Hybrid SaaS environment. This will be clearly stated in the component documentation. Components that are currently only available in Hybrid SaaS include:

* Custom [JDBC](/docs/components/jdbc) drivers.
* [SAP ODP](/docs/components/sap-odp/).
* [NetSuite SuiteAnalytics](/docs/components/netsuite-suiteanalytics).
* [Python Script](/docs/components/python-script). Use [Python Pushdown](/docs/components/python-pushdown) instead in a Full SaaS deployment (Snowflake only).

### Matillion Full SaaS

Every {maia} account is provided with a {m_runner} hosted by Matillion, providing a true zero-install experience.

If your projects are set up to use Matillion's Full SaaS infrastructure then there is no need to be concerned with managing the {m_runner}—this is handled by Matillion.

There's no setup needed for a Full SaaS solution—it's available for you to use straight away.

If you need to allow any access from the Matillion Full SaaS {m_runner} to your data plane, you'll find the IP addresses [here](/docs/security/network-access-and-ip-allowlist-requirements#full-saas).

<Warning>
  Certain pipeline components, listed above, are not available for use in a Full SaaS deployment. In addition, some cloud-specific components (for example, AWS [SNS Message](/docs/components/sns-message) and [SQS Message](/docs/components/sqs-message)), as well as data ingestion and staging operations, may require extra work in setting up [cloud provider credentials](/docs/guides/cloud-credentials) and ensuring that your cloud data warehouse has the appropriate access permissions.
</Warning>

### Hybrid SaaS

If your account is on our Enterprise plan, you also have the option to install our {m_runner} software within your own cloud infrastructure and data plane. This may be beneficial if:

* Your data locality requirements need the {m_runner} to run in a specific region that Matillion doesn't currently provide {m_runner}s in.
* You aim to achieve faster processing speeds by locating the {m_runner} close to your applications, databases, and cloud data warehouse.
* You need to access systems (such as database or file storage) that only have network access from within your VPC/VNet.
* You have specific [{m_runner} scaling requirements](/docs/guides/scaling-best-practices) that a Matillion Full SaaS {m_runner} doesn't support.
* You need to use any of the Hybrid SaaS pipeline components listed above.

Currently, {maia} allows {m_runner}s to be hosted in AWS, Azure, Google Cloud, or Snowflake infrastructure. Instructions for hosting a {m_runner} in your own cloud infrastructure are given in [Create a {m_runner} in your infrastructure](/docs/guides/create-a-runner).

You can choose to have the {m_runner} update automatically when a new version is released by Matillion, or you can manually control the updating of your {m_runner}s. Read [{m_runner} updates](/docs/guides/runner-updates) for details.

### Migrating from Full SaaS to Hybrid SaaS

Projects are either Full SaaS or Hybrid SaaS, and can't switch between the two. If you want to move Full SaaS workloads to Hybrid SaaS infrastructure, you need to perform the following steps:

1. [Install](/docs/guides/create-a-runner) the Hybrid SaaS {m_runner} in your infrastructure.
2. [Create a new project](/docs/guides/projects) that uses your Hybrid SaaS {m_runner}.
3. Recreate any [secrets](/docs/guides/secrets-and-secret-definitions) in the new project.
4. [Export and Import](/docs/guides/pipelines#importing-and-exporting-pipelines) the pipelines from the Full SaaS project to your new Hybrid SaaS project.
5. Recreate any [schedules](/docs/guides/schedules) in the new project.

***

## Maia runner version tracks

In a Hybrid SaaS deployment, it's important that you always use a currently supported version of the {m_runner}, which means you will be required to update your {m_runner} from time to time. As we recognize that it's not always possible or convenient to update as soon as a new {m_runner} version is released, we offer a choice of release cadences, which we call version tracks. The choices are Current and Stable:

* **Current:** Supports the latest {maia} features and has a faster cadence for releases. This is usually twice a week, typically on a Tuesday and Thursday, though this may vary. These releases may include new features, bug fixes, and security patches. This track is ideal if you want to access the latest features as soon as they are available, and are able to update your {m_runner} frequently.
* **Stable:** Has a slower, more predictable cadence for releases. This is once per month, on the 1st of the month. Features added to a Stable release have previously been available on a Current release. We take a cut of the Current track on the first day of each month, and after a period of testing and verification we release this as the Stable version on the first day of the *subsequent* month. This means that any new features in a Stable release have been available for at least one month on the Current track.

If you need to change the track that an existing {m_runner} is on, you can use the [Update a {m_runner}](/api-reference/agents/update-an-agent) public API endpoint. However, take note of the following:

* Changing the {m_runner}'s version track does *not* update the {m_runner} itself. You must update the image URI for the {m_runner} service in your cloud provider. If you don't, the {m_runner} version may show as "Out of support".
* If you change the {m_runner} state from **Current** to **Stable** and update the installed {m_runner} service, pipelines created using a newer (**Current**) {m_runner} version might not run as expected on an older (**Stable**) version.

Choose the track which suits your needs, or even use a combination of different tracks for different {m_runner}s. For example, this would allow you to run your production {m_runner}s on the Stable track, and run your test or development {m_runner}s on the Current track, giving you a month to undertake testing and verification of a Current version before it's adopted as Stable.

The decision between choosing the Current or Stable tracks should be based on how the {m_runner} update cadence fits with your operational practices, and how eager you are to take advantage of new features.

You choose the {m_runner}'s version track when you first [create](/docs/guides/create-a-runner) the {m_runner}. If you have multiple {m_runner}s, they can be on different version tracks.

Critical security patches are always applied to the supported {m_runner} releases on both tracks.

Due to the nature of the Snowflake Native App release process, [Matillion {m_runner} for Snowflake](/docs/guides/snowflake-runner-install) can only use the Stable version track.

The Matillion-hosted [Full SaaS {m_runner}](/docs/guides/runner-overview#matillion-full-saas) is on the Current version track, and so always uses the latest available {m_runner} version.

Read [{m_runner} updates](/docs/guides/runner-updates) for details of how to update your {m_runner}.