Azure AKS deployment guide for Matillion agents

​

What you get with AKS deployment

• Managed Kubernetes control plane. Azure handles the Kubernetes API server, etcd, and control plane upgrades.
• Workload Identity/Managed Identity. Allows credential-free authentication to Azure services.
• Flexible node pools. VM scale sets with configurable instance types.
• Azure integration. Native support for Azure Monitor, VNet networking, and Azure Load Balancers.
• Horizontal Pod Autoscaler. Scale agent pods based on metrics.
• Cluster Autoscaler. Automatically adjust VM capacity in node pools.

​

When to choose AKS

• You have existing Azure infrastructure and expertise.
• You need Managed Identity for secure, credential-free access to Azure services (Storage, Key Vault).
• You require integration with Azure monitoring and security tools (Azure Monitor, Sentinel).
• You want Kubernetes operational flexibility with Azure managed services.
• You plan to deploy agents across multiple availability zones for high availability.

​

Prerequisites and readiness

​

Azure account requirements

• Azure Kubernetes Service enabled in your subscription.
• Sufficient VM quotas for worker nodes.
• Virtual Network (VNet) with subnet configuration.

• Create and manage AKS clusters and node pools.
• Create and manage virtual machines and VM scale sets.
• Create Managed Identities and role assignments.
• Manage VNet resources (subnets, route tables, network security groups).
• Access Azure Key Vault (for storing OAuth credentials).
• Create Storage Accounts (for agent staging data).
• Configure Azure Monitor and Log Analytics.

​

Matillion account setup

• Account ID: Your Matillion organization identifier.
• Agent ID: The unique identifier for this agent (auto-generated).
• OAuth Client ID and Secret: Agent authentication credentials
• Region: us1 (United States) or eu1 (Europe).

​

Required tools

• Terraform 1.0+ for infrastructure provisioning.
• Azure CLI configured with credentials (az login).
• kubectl for Kubernetes cluster management.
• Helm 3.x for application deployment.

# Verify Azure CLI authentication
az account show

# Verify tool versions
terraform --version
kubectl version --client
helm version

​

Architecture decision points

​

1. VNet strategy

​

2. Subnet strategy

​

3. Public vs private cluster

• Deployment workstation has VPN or bastion access to VNet.
• CI/CD runners can access cluster API server.
• Authorized IP ranges include your access points.

​

4. Authentication strategy

• Workload Identity (recommended for AKS 1.25+):
  • Agent pods authenticate to Azure using federated identity credentials.
  • No secrets stored in cluster.
  • Azure AD-backed, automatic token rotation.
  • Best-practice security model for AKS.
• Managed Identity (Azure AD Pod Identity):
  • Agent pods assume Managed Identity without storing credentials.
  • Supports AKS clusters older than 1.25.
  • Automatic credential rotation by Azure.
  • Terraform module can configure either approach.
• Static OAuth Credentials:
  • OAuth credentials stored in Kubernetes Secrets.
  • Use only if identity-based authentication unavailable (not recommended for AKS).

​

5. Node pool strategy

• Transformation-heavy workloads: SQL generation tasks, low agent CPU usage → Smaller VMs sufficient.
• Data ingestion/scripting workloads: High data transfer, processing on agent → Larger VMs needed.
• Pod density: Larger VMs allow more agent pods per node, reducing operational overhead.

​

6. Scaling strategy

• Fixed number of agent pods (e.g., 2, 5, 10).
• Predictable capacity and costs.
• Suitable for steady-state workloads.

• Automatically scales agent pods based on CPU, memory, or custom metrics.
• Configure min/max replicas (e.g. min: 2, max: 10).
• Responds to workload spikes dynamically.

• Automatically adds/removes VMs in node pool based on pod scheduling needs.
• Works in tandem with HPA.
• Optimizes infrastructure costs.

​

Container images

• :stable - Slower release cycle, maximum stability, recommended for production.
• :current - Faster release cycle, earlier access to new features.

​

Deployment journey

​

Expected timeline

• Phase 1 - Agent registration: 10 minutes (Matillion console).
• Phase 2 - Infrastructure provisioning: 15-20 minutes (Terraform: VNet, AKS cluster, Managed Identity).
• Phase 3 - Configure kubectl access: 2 minutes (Azure CLI + kubectl).
• Phase 4 - Agent deployment: 5-10 minutes (Helm chart).
• Phase 5 - Validation: 15-30 minutes (Pre-deployment checks + testing).

​

Phase 1: Agent registration (Matillion console)

• Account ID.
• Agent ID.
• OAuth Client ID and Secret.
• Region (us1 or eu1).

​

Phase 2: Infrastructure provisioning (Terraform)

1. Azure AKS cluster:
   • Managed Kubernetes control plane (API server, etcd, controller manager).
   • AKS-managed upgrades and patching.
   • Azure Monitor integration for control plane logs.
2. Node Pools:
   • VM Scale Set with configurable VM sizes.
   • Azure-managed node lifecycle and upgrades.
   • Kubernetes node labels and taints (if configured).
3. Managed Identity/Workload Identity:
   • User-assigned Managed Identity for agent workloads.
   • Role assignments for Storage Account access and Key Vault access.
   • Federated identity credentials (for Workload Identity).
4. VNet and Networking (if creating new):
   • Virtual Network with subnets across availability zones.
   • Network security groups for cluster communication.
   • Route tables for outbound connectivity.
5. Network Security Groups:
   • Rules for control plane and node communication.
   • HTTPS outbound to Matillion control plane.

• azure_subscription_id: Your Azure subscription ID.
• azure_tenant_id: Your Azure AD tenant ID.
• resource_group_name: Resource group name (e.g. matillion-agent-rg).
• location: Azure region (e.g. East US, West Europe).
• name: Cluster name prefix (e.g. matillion-agent).
• vm_size: Node pool VM size (e.g. Standard_D4s_v4).
• desired_node_count: Initial node count (e.g. 3).
• is_private_cluster: true or false.
• authorized_ip_ranges: List of CIDRs allowed to access API server.
• workload_identity_enabled: true (recommended) or false.
• tags: Resource tags for cost allocation.

terraform output cluster_name
terraform output managed_identity_client_id

​

Phase 3: Configure kubectl access

az aks get-credentials --resource-group <resource-group> --name <cluster-name>

kubectl get nodes
kubectl get namespaces

​

Phase 4: Agent deployment (Helm)

1. Agent Pods:
   • Deployment with configurable replica count (default: 2).
   • Each pod runs the Matillion agent binary.
   • Resource requests and limits for CPU and memory.
2. ServiceAccount:
   • Kubernetes ServiceAccount configured for Workload Identity or Managed Identity.
   • Annotated with Managed Identity client ID (from Phase 2).
3. ConfigMaps:
   • Agent configuration (account ID, agent ID, region).
   • Environment-specific settings.
4. Secrets:
   • OAuth Client ID and Secret for Matillion control plane authentication.
5. Service:
   • Kubernetes service exposing Prometheus metrics endpoint (port 8080).
   • Annotated for Prometheus service discovery.

• Helm Chart Documentation.
• values.yaml Reference.

​

Phase 5: Validation and testing

# From deployment library root
./agent/helm/checks/run-check.sh --namespace matillion --release matillion-agent

• Python 3 and Java runtime available.
• Filesystem permissions correct.
• Environment variables set (ACCOUNT_ID, AGENT_ID, etc.).
• cgroup CPU and memory limits applied.
• Network connectivity to Matillion control plane.
• Security agents that might interfere (Crowdstrike, Prisma Cloud).

1. Matillion Console: Navigate to Manage → Agents. Verify agent status shows “Connected”.
2. Test Pipeline: Create a simple pipeline (e.g. “Hello World” transformation) and execute.
3. Prometheus Metrics: Verify metrics available at http://<pod-ip>:8080/actuator/prometheus.

• Log Analytics workspace query: ContainerLog|where PodName contains "matillion-agent"

​

Agent architecture on AKS

​

How data flows

​

Workload Identity / Managed Identity

1. Kubernetes ServiceAccount is annotated with Managed Identity client ID.
2. AKS OIDC Issuer allows Kubernetes to issue tokens trusted by Azure AD.
3. Agent pod requests Azure AD token using projected service account token.
4. Azure AD exchanges token for access token (valid 1 hour, auto-refreshed).
5. Agent accesses Azure services (Storage, Key Vault) without storing credentials.

• No long-lived credentials in cluster.
• Automatic token rotation (every hour).
• Least-privilege access (Managed Identity scoped to specific resources).
• Pod-level isolation (each pod authenticates independently).

• User-assigned Managed Identity.
• Federated identity credential (Workload Identity).
• Role assignments for Storage Account, Key Vault.
• AKS cluster OIDC issuer configuration.

​

Task capacity and throughput

• 2 pods (default) = 40 concurrent tasks.
• 5 pods = 100 concurrent tasks.
• 10 pods = 200 concurrent tasks.

• For transformation workloads: Tasks generate SQL executed by data warehouse. Agent CPU/memory usage is low. Fewer pods needed.
• For data ingestion workloads: Tasks transfer and process data on agent. Agent CPU/memory usage is high. More pods needed.

​

Monitoring and observability

​

Native Prometheus metrics

• Endpoint: http://<pod-ip>:8080/actuator/prometheus.
• Service: Automatically created by Helm chart with Prometheus annotations.

• app_version_info: Agent version and build metadata.
• app_agent_status: Agent status (1 = running, 0 = stopped).
• app_active_task_count: Current number of executing tasks.
• app_active_request_count: Active HTTP requests to agent.
• app_open_sessions_count: Open connections to data warehouses.

prometheus.io/scrape: "true"
prometheus.io/port: "8080"
prometheus.io/path: "/actuator/prometheus"

​

Azure Monitor integration

• Cluster-level metrics (CPU, memory, network).
• Pod-level metrics (resource usage per agent pod).
• Node-level metrics (VM health, disk usage).

• Centralized log aggregation across all pods.
• Query with Kusto Query Language (KQL).
• Set up alerts on error patterns.

• Agent pod restarts > threshold.
• Agent pods in CrashLoopBackOff state.
• Task execution failures (requires custom metric from agent logs).
• Node pool CPU/memory > 80%.

​

Recommended monitoring setup

1. Use Prometheus for agent-specific metrics (task count, session count, agent status).
2. Use Azure Monitor for infrastructure metrics (cluster health, node capacity, pod restarts).
3. Set up Grafana dashboards combining Prometheus and Azure Monitor metrics.
4. Configure alerts:
   • Agent connectivity to Matillion control plane lost.
   • Task queue depth increasing (capacity insufficient).
   • Agent pod memory usage approaching limits.

​

Security best practices

​

Network security

• Deploy agent pods in private subnets for enhanced security.
• Use NAT Gateway or default route for outbound internet access (required for Matillion control plane).
• Restrict network security groups to minimum required ingress/egress.

• HTTPS (443) to Matillion control plane (region-specific endpoints).
• HTTPS/JDBC to data warehouse endpoints (Snowflake, Synapse, BigQuery).
• HTTPS (443) to Azure APIs (Storage, Key Vault, Azure AD for Workload Identity).

• Egress: Allow HTTPS (443) to specific endpoints only.
• Ingress: No inbound traffic required (agent initiates all connections).

• API server accessible only from VNet (or authorized VPN/bastion).
• Requires VPN or Azure Bastion for kubectl access.
• CI/CD pipelines need VNet connectivity or VPN access.

​

Pod security standards

• Run as non-root user (UID 65534).
• Read-only root filesystem.
• No privilege escalation.
• Drop all Linux capabilities.
• Seccomp profile: RuntimeDefault.

securityContext:
  runAsNonRoot: true
  runAsUser: 65534
  fsGroup: 65534
  seccompProfile:
    type: RuntimeDefault

containers:
  - securityContext:
      allowPrivilegeEscalation: false
      capabilities:
        drop: ["ALL"]

​

Secrets management

• Azure Key Vault (Recommended):
  • Store OAuth credentials in Azure Key Vault.
  • Use External Secrets Operator or Azure Key Vault Provider for CSI Driver to sync to Kubernetes Secrets.
  • Automatic rotation support.
  • Centralized secret management across environments.
• Kubernetes Secrets (Default):
  • Credentials provided via Helm values.
  • Stored as base64-encoded Kubernetes Secret.
  • Not encrypted at rest by default (enable Azure Disk encryption).

​

Scaling considerations

​

When to scale

• Task queue depth consistently > 0 (check the Task history or metrics).
• Pipeline execution time increases due to task queuing.
• More concurrent pipelines being executed.
• Workload characteristics change (more data ingestion vs transformation).

• Task queue depth consistently = 0.
• Agent pod CPU < 60%, memory < 70%.
• Pipeline execution times stable.
• Workload primarily transformation (SQL generation).

​

Horizontal Pod Autoscaler (HPA)

• Kubernetes HPA monitors pod metrics (CPU, memory, or custom metrics).
• Automatically scales Deployment replicas within configured min/max range.
• Evaluates every 15 seconds (default), scales up/down based on thresholds.

• Min replicas: 2 (baseline availability).
• Max replicas: 10 (cost control).
• Target CPU: 70% (scale up when average CPU > 70%).

​

Cluster autoscaler

• Monitors pods in Pending state (unable to schedule due to insufficient node capacity).
• Automatically adds VMs to node pool (VM Scale Set).
• Removes underutilized nodes after 10 minutes of low usage.

1. HPA scales agent pods based on metrics.
2. If pods can’t schedule (no node capacity), cluster autoscaler adds VMs.
3. Agent pods schedule on new VMs.
4. When load decreases, HPA scales down pods, cluster autoscaler removes empty VMs.

​

Vertical scaling

• Useful when individual tasks require more resources than current pod limits.
• Requires pod restart to apply new resource limits.
• Consider workload characteristics (transformation vs ingestion).

​

Cost optimization

​

AKS pricing components

• AKS Control Plane: Free (for standard tier).
• VM Node Pool: Pay for VMs (varies by VM size and region):
  • Standard_D2s_v4: ~$70/month (2 vCPU, 8GB RAM)
  • Standard_D4s_v4: ~$140/month (4 vCPU, 16GB RAM)
  • Standard_D8s_v4: ~$280/month (8 vCPU, 32GB RAM)
• Data Transfer: Outbound data transfer charges (data warehouse connections, Matillion control plane).
• Managed Disks: For node OS disks (~$5-20/month per node depending on disk type).

​

Cost optimization strategies

• Right-size VMs: Match VM type to workload (transformation-heavy = smaller, ingestion-heavy = larger).
• Use Cluster Autoscaler: Automatically remove unused VMs during low-usage periods.
• Consider Azure Reserved Instances: For predictable baseline capacity (1-year or 3-year commitment).
• Use Spot VMs for non-critical workloads: Up to 80% cost savings (with eviction risk).
• Monitor data transfer: Ensure data warehouses in same region to avoid cross-region charges.

​

Additional resources

​

Implementation and deployment

• Azure agent directory.
• AKS Terraform module.
• Helm charts.

​

General Kubernetes guide

​

Matillion documentation

• For deployment models, read Agent overview.
• For agent registration, read Create an agent.
• For capacity planning, read Scaling best practices.

​

Azure documentation

• For AKS concepts and operations, read Azure Kubernetes service documentation.
• For Azure AD Workload Identity for AKS, read Workload Identity.
• For automatic node scaling, read AKS Cluster Autoscaler.