Skip to main content
The is a component within that serves as a bridge between the source database and the target cloud data lake or cloud storage, enabling the execution and scheduling of streaming pipelines. The will be hosted in your own infrastructure, using a Hybrid SaaS solution. Once the is configured and started, it operates autonomously without requiring much intervention. The continuously monitors all changes occurring in the source database, consumes those changes from the low-level logs, and delivers them to the designated target data lake or storage. This ensures a continuous and reliable change data capture process. This topic explains how to create a in your Google Cloud infrastructure.
Each can run only one Streaming pipeline. Each Streaming pipeline requires a new installation.

​
Prerequisites

  • A account. To register, read Registration. Once you have signed up, log in to .
  • An account in Google Cloud to host the .
  • Access to a cloud secrets service, which is a secure storage system for storing authentication and connection secrets. These secrets are used by the to authenticate itself with the source database and establish a secure connection for capturing the data changes.

​
Create a Streaming runner

  1. In the left navigation, click , then in the menu, click Runners. The Runners list lists all s currently created, showing their Status, Platform, and Type.
  2. Click Add runner.
  3. Click Streaming.
  4. Complete the following properties:
    • Runner name: A unique name for your new . Maximum 30 characters. Accepts both uppercase and lowercase A-z, 0-9, whitespace (not the first character), hyphens and underscores.
    • Description: Optionally enter a brief description of the .
    • Cloud provider: The cloud platform that the will be deployed to. Select GCP.
    • Deployment: The deployment method. Choose from GCE (Google Compute Engine) or GKE (Google Kubernetes Engine).
  5. Click Create runner.
  6. This creates a definition in , and displays the following parameters on the Runner details page. You will need these parameters to set up the in your Google Cloud infrastructure.
    • ACCOUNT_ID
    • AGENT_ID
    • MATILLION_REGION
    • OAUTH_CLIENT_ID
    • OAUTH_CLIENT_SECRET
    The screen will also show any optional environment variables needed by the . Click Reveal to make the OAUTH_CLIENT_ID and OAUTH_CLIENT_SECRET values visible before copying.
  7. The ‘s status is set to Pending, which means it is not yet ready to run pipelines. The next step is to deploy the application into your Google Cloud infrastructure, as described below.

​
Set up the Streaming runner in your Google Cloud infrastructure

Setting up s requires access to Google Cloud services, as well as a degree of familiarity with that platform. We recommend going through this process with your Google Cloud administrator.
Google Cloud resources may come with their own pricing independent of any billing you receive from Matillion.
Recommendations:
  • Create new resources specifically for streaming use rather than attempt to use existing cloud resources.
  • Set up a resource group for your new resources for better organization and billing ease.
  • Consult your cloud/network administrator for advice on Google Cloud permissions, roles, access and other considerations such as Google Cloud regions.
  • Keep resources in the same Google Cloud region. Note that all resources and services may not be available in all regions; it is recommended you research your desired region in advance.

​
Check streaming runner status

After deploying the in your Google Cloud infrastructure, you should return to to verify that it’s correctly connected and running.
  1. In the left navigation, click , then in the menu, click Runners. The Runners list lists all s currently created.
  2. Locate the in the list and check the status:
    • Pending: The has been created but has not yet connected to .
    • Running: The is connected and available for running Streaming pipelines, or is connected and already running a Streaming pipeline.
    • Stopped: The has been stopped.
    • Unknown: The is in an unknown state. This typically means the has lost connection to without being stopped, for example due to networking issues.
  3. When the status shows Running, it’s ready to use. It can be selected in the Runner drop-down when you create a new Streaming pipeline, as long as a pipeline is not already assigned.