Google Cloud GKE deployment for Maia runners

Documentation Index

Fetch the complete documentation index at: https://docs.maia.ai/llms.txt

Use this file to discover all available pages before exploring further.

​

What you get with GKE deployment

• Managed Kubernetes control plane. Google handles the Kubernetes API server, etcd, and control plane upgrades.
• Workload Identity. Credential-free authentication from pods to Google Cloud services using federated identity credentials.
• Flexible node pools. Configurable machine types with autoscaling support.
• Google Cloud integration. Native support for Cloud Monitoring, VPC networking, and Google Cloud Load Balancers.
• Horizontal Pod Autoscaler. Scale pods based on metrics.
• Cluster Autoscaler. Automatically adjust node capacity in node pools.

​

Prerequisites and readiness

​

Google Cloud requirements

• Google Kubernetes Engine API enabled in your target project.
• Billing enabled for your GCP project.
• Sufficient compute quotas for node pool VMs.
• VPC with subnet configuration.

• Create and manage GKE clusters and node pools.
• Create and manage Compute Engine instances.
• Create service accounts and IAM bindings (including Workload Identity bindings).
• Manage VPC resources (subnets, routes, Cloud NAT, firewall rules).
• Access Secret Manager (for storing OAuth credentials).
• Create GCS buckets (for staging data).
• Configure Cloud Logging and Cloud Monitoring.

​

Matillion account setup

• Account ID: Your Matillion organization identifier.
• Agent ID: The unique identifier for this (auto-generated).
• OAuth Client ID and Secret: authentication credentials.
• Region: us1 (United States), eu1 (Europe), or au1 (Australia/Asia-Pacific).

​

Required tools

• Terraform 1.0+ for infrastructure provisioning.
• Google Cloud SDK (gcloud) configured with credentials (gcloud auth application-default login).
• gke-gcloud-auth-plugin for kubectl authentication to GKE.
• kubectl for Kubernetes cluster management.
• Helm 3.x for application deployment.

# Verify Google Cloud authentication
gcloud auth list
gcloud config get-value project

# Verify tool versions
terraform --version
kubectl version --client
helm version
gke-gcloud-auth-plugin --version

​

Architecture decision points

​

1. VPC and networking

• VPC with subnets and secondary IP ranges for GKE pods and services.
• Cloud NAT (when enable_cloud_nat = true) for outbound internet access from private nodes.
• Firewall rules for cluster communication.

​

2. Public vs private cluster

• Cloud NAT is enabled so private nodes can pull container images and reach external endpoints.
• Deployment workstation has VPN or bastion access to VPC.
• Authorized IP ranges include your access points.

​

3. Authentication strategy

• Workload Identity (recommended):
  • pods authenticate to Google Cloud APIs using federated identity credentials.
  • No static credentials stored in the cluster.
  • Automatic token rotation by Google Cloud STS.
  • Best-practice security model for GKE.
• Static OAuth credentials:
  • OAuth credentials stored in Kubernetes Secrets.
  • Use only if Workload Identity cannot be implemented (not recommended for GKE).

​

4. Node pool strategy

• Transformation-heavy workloads: SQL generation tasks, low CPU usage → Smaller machines sufficient.
• Data ingestion/scripting workloads: High data transfer, processing on → Larger machines needed.
• Pod density: Larger machines allow more pods per node, reducing operational overhead.

​

5. Scaling strategy

• Fixed number of pods (for example, 2, 5, 10).
• Predictable capacity and costs.
• Suitable for steady-state workloads.

• Automatically scales pods based on workload metrics.
• Configure min/max replicas (for example, min: 2, max: 10).
• Responds to workload spikes dynamically.

• Automatically adds/removes nodes in the node pool based on pod scheduling needs.
• Works in tandem with HPA.
• Optimizes infrastructure costs.

​

Container images

• US region: us-docker.pkg.dev/maia-492711/maia-runners/maia-runner
• EU region: europe-docker.pkg.dev/maia-492711/maia-runners/maia-runner
• AU region: australia-southeast1-docker.pkg.dev/maia-492711/maia-runners/maia-runner

• :stable—Slower release cycle, maximum stability, recommended for production.
• :current—Faster release cycle, earlier access to new features.

​

Deployment journey

​

Expected timeline

• Phase 1— registration: 10 minutes (Matillion console).
• Phase 2—Infrastructure provisioning: 15-20 minutes (Terraform: VPC, GKE cluster, Workload Identity).
• Phase 3—Configure kubectl access: 2 minutes (gcloud CLI + kubectl).
• Phase 4— deployment: 5-10 minutes (Helm chart).
• Phase 5—Validation: 15-30 minutes (pre-deployment checks + testing).

​

Phase 1: Maia runner registration (Matillion console)

• Account ID.
• Agent ID.
• OAuth Client ID and Secret.
• Region (us1, eu1, or au1).

​

Phase 2: Infrastructure provisioning (Terraform)

1. GKE cluster:
   • Managed Kubernetes control plane (API server, etcd, controller manager).
   • GKE-managed upgrades and patching.
   • Cloud Logging and Cloud Monitoring integration.
2. Node pool:
   • Managed instance group with configurable machine types.
   • Shielded nodes with Secure Boot enabled.
   • Kubernetes node labels (if configured).
3. Workload Identity:
   • GCP service account for workloads.
   • IAM binding linking the GCP service account to the Kubernetes service account.
   • Role assignments for Secret Manager and GCS access.
4. VPC and networking:
   • VPC with subnets and secondary IP ranges for pods and services.
   • Cloud NAT for outbound internet from private nodes.
   • Firewall rules for cluster communication.
5. Supporting services:
   • GCS bucket for staging storage.
   • Secret Manager secret for credential storage.

• project_id: Your GCP project ID.
• region: GCP region (for example, us-central1, europe-west1, australia-southeast1).
• name: Cluster name prefix (for example, matillion-runner).
• desired_node_count: Initial node count (for example, 2).
• machine_type: Node pool machine type (for example, e2-standard-4).
• is_private_cluster: true or false.
• master_ipv4_cidr_block: CIDR for the private cluster control plane (for example, 172.16.0.0/28).
• authorized_ip_ranges: List of CIDRs allowed to access the API server.
• enable_cloud_nat: true (required when is_private_cluster = true).
• labels: Resource labels for cost allocation.

terraform output cluster_name
terraform output runner_workload_sa_email

​

Phase 3: Configure kubectl access

gcloud components install gke-gcloud-auth-plugin

gcloud container clusters get-credentials <cluster-name> \
  --region <region> \
  --project <project-id>

kubectl get nodes
kubectl get namespaces

​

Phase 4: Maia runner deployment (Helm)

1. pods:
   • Deployment with configurable replica count (default: 2).
   • Each pod runs the binary.
   • Resource requests and limits derived from runnerSize t-shirt sizing.
2. ServiceAccount:
   • Kubernetes ServiceAccount configured for Workload Identity.
   • Annotated with the GCP service account email (from Phase 2).
3. ConfigMaps:
   • configuration (account ID, Agent ID, region, default GCP project).
   • Environment-specific settings.
4. Secrets:
   • OAuth Client ID and Secret for Matillion control plane authentication.
5. Service:
   • Kubernetes service exposing Prometheus metrics endpoint (port 8080).
   • Annotated for Prometheus service discovery.

• defaultGcpProject is the fallback vault used when a secret definition does not specify a project.
• Additional GCP projects can be granted access so they also appear in the Matillion UI. See Phase 4a below if you need multi-project access.
• For dev and preprod environments, set matillionEnv in the dpcAgent.dpcAgent.env block. Without it the defaults to prod, which will cause it to connect to the wrong environment.
• The optional environment variables (proxyHttp, customCertLocation, etc.) must be explicitly set to empty strings if not used. Leaving them as template placeholders (for example, <CustomCertLocation>) causes the to exit immediately on startup with no further error.
• The Helm release name and Kubernetes namespace must match. The Workload Identity binding in Terraform is created for {namespace}/{namespace}-sa. If the namespace differs from the release name, authentication will silently fail.

• Helm chart documentation.
• values.yaml reference.
• values-gcp.yaml example.

​

Phase 4a (Optional): Granting access to additional GCP projects

gcloud services enable \
  secretmanager.googleapis.com \
  cloudresourcemanager.googleapis.com \
  --project=ADDITIONAL_PROJECT_ID

​

Option 1: Terraform (recommended)

additional_gcp_projects = [
  "second-project-id",
  "third-project-id"
]

​

Option 2: Manual (gcloud)

RUNNER_SA="your-runner-sa@primary-project.iam.gserviceaccount.com"
ADDITIONAL_PROJECT="second-project-id"
for ROLE in roles/secretmanager.secretAccessor roles/secretmanager.viewer roles/browser; do
  gcloud projects add-iam-policy-binding "$ADDITIONAL_PROJECT" \
    --member="serviceAccount:$RUNNER_SA" \
    --role="$ROLE"
done

​

Phase 5: Validation and testing

# From deployment library root
./runner/helm/checks/run-check.sh --namespace matillion-runner --release matillion-runner

• Python 3 and Java runtime available.
• Filesystem permissions correct.
• Environment variables set (ACCOUNT_ID, AGENT_ID, etc.).
• cgroup CPU and memory limits applied.
• Network connectivity to Matillion control plane.
• Security agents that might interfere (Crowdstrike, Prisma Cloud).

1. Matillion Console: Navigate to Manage runners. Verify status shows “Connected”.
2. Test pipeline: Create a simple pipeline (for example, “Hello World” transformation) and execute.
3. Prometheus metrics: Verify metrics available at http://<pod-ip>:8080/actuator/prometheus.

gcloud logging read \
  "resource.type=k8s_container AND resource.labels.cluster_name=<cluster-name>" \
  --project=<project-id> --limit=50

​

Maia runner architecture on GKE

​

Workload Identity

1. Kubernetes ServiceAccount is annotated with the GCP service account email.
2. GKE OIDC issuer allows Kubernetes to issue tokens trusted by Google Cloud IAM.
3. pod requests a Google Cloud access token using the projected service account token.
4. Google Cloud STS exchanges the token for short-lived access credentials (valid one hour, auto-refreshed).
5. accesses Google Cloud services (Secret Manager, GCS) without storing credentials.

• No long-lived credentials in cluster.
• Automatic token rotation (every hour).
• Least-privilege access (GCP service account scoped to specific resources).
• Pod-level isolation (each pod authenticates independently).

• GCP service account for workloads.
• Workload Identity IAM binding between the GCP service account and the Kubernetes service account.
• Role assignments for Secret Manager (secretAccessor, viewer) and GCS.
• GKE Workload Identity pool configuration on the cluster.

terraform output -raw runner_workload_sa_email

​

Task capacity and throughput

• Two pods (default) = 40 concurrent tasks.
• Five pods = 100 concurrent tasks.
• 10 pods = 200 concurrent tasks.

• For transformation workloads: Tasks generate SQL executed by data warehouse. CPU/memory usage is low. Fewer pods needed.
• For data ingestion workloads: Tasks transfer and process data on . CPU/memory usage is high. More pods needed.

​

Monitoring and observability

​

Native Prometheus metrics

• Endpoint: http://<pod-ip>:8080/actuator/prometheus.
• Service: Automatically created by Helm chart with Prometheus annotations.

• app_version_info: version and build metadata.
• app_agent_status: status (1 = running, 0 = stopped).
• app_active_task_count: Current number of executing tasks.

prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/actuator/prometheus"

​

Google Cloud Monitoring integration

• Cluster-level metrics (CPU, memory, network).
• Pod-level metrics (resource usage per pod).
• Node-level metrics (VM health, disk usage).

• Centralized log aggregation across all pods.
• Query with the Logs Explorer or gcloud CLI.
• Set up log-based alerts on error patterns.

• pod restarts > threshold.
• pods in CrashLoopBackOff state.
• Task execution failures (requires custom metric from logs).
• Node pool CPU/memory > 80%.

​

Security best practices

​

Network security

• Deploy pods in private node pools for enhanced security.
• Use Cloud NAT for outbound internet access (required for Matillion control plane and image pulls when is_private_cluster = true).
• Restrict firewall rules to minimum required ingress/egress.

• HTTPS (443) to Matillion control plane (region-specific endpoints).
• HTTPS/JDBC to data warehouse endpoints (Snowflake, BigQuery).
• HTTPS (443) to Google Cloud APIs (Secret Manager, GCS, IAM, Artifact Registry).
• HTTP (80) to Snowflake endpoints.
• Ingress: No inbound traffic required ( initiates all connections).

• API server accessible only from VPC (or authorized VPN/bastion).
• Requires VPN or Identity-Aware Proxy for kubectl access.
• CI/CD pipelines need VPC connectivity or VPN access.

​

Pod security standards

• Run as non-root user (UID 65534).
• Read-only root filesystem.
• No privilege escalation.
• Drop all Linux capabilities.
• Seccomp profile: RuntimeDefault.

securityContext:
  runAsNonRoot: true
  runAsUser: 65534
  fsGroup: 65534
  seccompProfile:
    type: RuntimeDefault

containers:
  - securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]

​

Secrets management

• Google Cloud Secret Manager (recommended):
  • Store OAuth credentials in Secret Manager.
  • Use External Secrets Operator to sync to Kubernetes Secrets.
  • Automatic rotation support.
  • Centralized secret management across environments.
• Kubernetes Secrets (default):
  • Credentials provided via Helm values.
  • Stored as base64-encoded Kubernetes Secret.
  • Not encrypted at rest by default (enable GKE application-layer secret encryption with Cloud KMS).

​

Scaling considerations

​

When to scale

• Task queue depth consistently > 0 (check the Task history or metrics).
• Pipeline execution time increases due to task queuing.
• More concurrent pipelines being executed.
• Workload characteristics change (more data ingestion vs transformation).

• Task queue depth consistently = 0.
• Pipeline execution times stable.
• Workload primarily transformation (SQL generation).

​

Horizontal Pod Autoscaler (HPA)

• Kubernetes HPA monitors pod metrics.
• Automatically scales Deployment replicas within configured min/max range.
• Evaluates every 15 seconds (default), scales up/down based on thresholds.

• Hard cap: 20. Each pod runs a maximum of 20 concurrent tasks. Values above 20 mean the HPA can never reach the target.
• Recommended range: 15–17.
  • 15—proactive (spiky or latency-sensitive workloads, more headroom).
  • 16—balanced (recommended default).
  • 17—reactive (steady workloads, some queueing acceptable, lower cost).

​

Cluster autoscaler

• Monitors pods in Pending state (unable to schedule due to insufficient node capacity).
• Automatically adds VMs to node pool (managed instance group).
• Removes underutilized nodes after 10 minutes of low usage.

1. HPA scales pods based on metrics.
2. If pods can’t schedule (no node capacity), cluster autoscaler adds nodes.
3. pods schedule on new nodes.
4. When load decreases, HPA scales down pods, cluster autoscaler removes empty nodes.

​

Vertical scaling

• Useful when individual tasks require more resources than current pod limits.
• Requires pod restart to apply new resource limits.
• Consider workload characteristics (transformation vs ingestion).

​

Cost optimization

​

Cost optimization strategies

• Right-size nodes: Match machine type to workload (transformation-heavy = smaller, ingestion-heavy = larger).
• Use Cluster Autoscaler: Automatically remove unused nodes during low-usage periods.
• Consider Committed Use Discounts: For predictable baseline capacity (1-year or 3-year commitment).
• Monitor data transfer: Ensure data warehouses in the same region to avoid cross-region egress charges.
• Use Spot VMs for non-critical workloads: Configure node pool with Spot VMs for cost savings on interruptible workloads.

​

Troubleshooting

​

Maia runner exits immediately with no error (exit code 1)

​

BeanCreationException: gcpSecretManager

​

Unknown Matillion region

​

Workload Identity binding—identity pool does not exist

​

gke-gcloud-auth-plugin not found

export PATH="/opt/homebrew/share/google-cloud-sdk/bin:$PATH"

​

Maia runner gateway connection unhealthy

​

Additional resources

​

Implementation and deployment

• GCP directory.
• GKE Terraform module.
• Helm charts.

​

General Kubernetes guide

​

Matillion documentation

• For deployment models, read overview.
• For registration, read Create a .
• For capacity planning, read Scaling best practices.

​

Google Cloud documentation

• For GKE concepts and operations, read Google Kubernetes Engine documentation.
• For Workload Identity setup, read Use Workload Identity.
• For automatic node scaling, read Cluster Autoscaler.