> ## Documentation Index
> Fetch the complete documentation index at: https://docs.maia.ai/llms.txt
> Use this file to discover all available pages before exploring further.

# Streaming runner logs

export const s_runner = "Streaming runner";

export const maia = "Maia";

export const RunnerMetadata = ({runnerType, platforms = []}) => {
  return <div style={{
    background: 'var(--colors-background-light, #f9fafb)',
    border: '1px solid var(--colors-border-default, #e5e7eb)',
    borderRadius: '12px',
    padding: '20px 28px',
    marginBottom: '28px',
    boxShadow: '0 1px 4px rgba(0,0,0,0.10)'
  }}>
      <table style={{
    width: '100%',
    borderCollapse: 'collapse'
  }}>
        <tbody>
          <tr>
            <td style={{
    fontWeight: '600',
    paddingRight: '32px',
    paddingBottom: '14px',
    whiteSpace: 'nowrap',
    verticalAlign: 'middle',
    width: '180px'
  }}>Runner type</td>
            <td style={{
    paddingBottom: '14px',
    verticalAlign: 'middle'
  }}>{runnerType}</td>
          </tr>
          <tr>
            <td style={{
    fontWeight: '600',
    paddingRight: '32px',
    whiteSpace: 'nowrap',
    verticalAlign: 'middle'
  }}>Runner platform</td>
            <td style={{
    verticalAlign: 'middle'
  }}>
              <div style={{
    display: 'flex',
    flexWrap: 'wrap',
    gap: '8px'
  }}>
                {platforms.map((platform, i) => <span key={i} style={{
    background: '#dcfce7',
    color: '#15803d',
    border: '1px solid #bbf7d0',
    borderRadius: '9999px',
    padding: '3px 12px',
    fontSize: '0.85rem',
    fontWeight: '500',
    whiteSpace: 'nowrap'
  }}>
                    {platform} ✅
                  </span>)}
              </div>
            </td>
          </tr>
        </tbody>
      </table>
    </div>;
};

<RunnerMetadata runnerType="Streaming" platforms={["AWS", "Azure", "Google Cloud"]} />

The activity of your {s_runner} can be logged in your cloud provider's logging service, either [AWS CloudWatch](#aws-cloudwatch) or [Azure Log Analytics](#azure-log-analytics), to help you address any issues you encounter. Analyzing the logs generated by your {s_runner} enables you to identify patterns, troubleshoot issues, and gain insights into the behavior of your deployment.

***

## AWS CloudWatch

You can retrieve, inspect, and analyze log data using AWS CloudWatch in the AWS console.

By default, logs are kept for an indefinite time, but you can set up a retention period based on your requirements.

### Prerequisites

* You must have a valid AWS account to access and use AWS CloudWatch.
* Your [{s_runner}](/docs/streaming/create-streaming-agent) must be deployed in the instance for which you want to generate logs. The {s_runner} captures and forwards log data to AWS CloudWatch for monitoring and analysis.
* Your AWS account must have the necessary IAM (Identity and Access Management) permissions to access CloudWatch and retrieve logs. At a minimum, the account should have the `cloudwatch:DescribeLogGroups` and `cloudwatch:FilterLogEvents` permissions for the specific log group and log streams you want to access.
* Configure a log group in CloudWatch to store the logs generated by the {s_runner}.
* Configure the log retention settings for the log group to ensure that logs are retained for the desired period.

### How to generate logs in AWS

To generate logs from AWS CloudWatch for a deployed {s_runner} in an instance, follow these steps.

1. In the [AWS Management Console](https://aws.amazon.com/console/), navigate to the **CloudWatch** service from the services menu or from the search bar at the top of the console screen.

2. In the **CloudWatch** dashboard, locate and click **Log Groups** in the left-hand navigation menu under the **Logs** category.

3. In the **Log Groups** page, search for the log group associated with your {s_runner}'s logs. The log group should have a name or pattern specific to your deployment, which you will have set when you installed the {s_runner}.

   <Note>
     If a log group for the {s_runner} does not already exist, create a new log group by following the process in [Log groups](/docs/streaming/aws-streaming-agent-manual-setup/#log-groups-optional).
   </Note>

4. Click on the log group to access its details. You will see a list of log streams within that group.

5. Identify the log stream that corresponds to the specific instance where your {s_runner} is deployed. Click on the log stream to view its logs.

6. Within the log stream, you can locate specific logs by applying filters, searching for keywords, or adjusting the time range.

***

## Azure Log Analytics

In the Azure console, you can retrieve, inspect, and analyze log data using the Azure Log Analytics service. Azure Log Analytics provides a centralized location for log storage, analysis, and visualization, enabling you to monitor and troubleshoot your Azure environment effectively.

Azure Log Analytics also allows you to create custom dashboards and set up alerts based on log data. These features can help you monitor and receive notifications for specific events or log conditions.

### Prerequisites

* You must have a valid Azure subscription to access and use Azure Log Analytics.
* You must have Azure Container Instances (ACI) or an Azure Kubernetes Service (AKS) cluster where the {s_runner} is deployed in a container. The container must be running and generating logs.
* Ensure that an Azure Log Analytics workspace has been created. This should have been created when you [installed the {s_runner}](/docs/streaming/azure-streaming-agent-install/#resources-deployed).
* You must [enable container monitoring](https://learn.microsoft.com/en-us/azure/azure-monitor/vm/vminsights-enable-portal) for the Azure Container Instances or Azure Kubernetes Service cluster where the {s_runner} is deployed. This ensures that the logs generated by the container are collected and forwarded to Azure Log Analytics.

### How to generate logs in Azure

To generate logs from Azure Log Analytics for a deployed {s_runner} in an instance, follow these steps.

1. In the [Azure Portal](https://portal.azure.com), navigate to the **Log Analytics** service from the services menu or from the search bar at the top of the portal screen.
2. In the **Log Analytics** dashboard, search for the workspace associated with your {s_runner}'s logs. The workspace should have a name or pattern specific to your deployment. Click the workspace name to enter that workspace.
3. In the Log Analytics workspace, click **Activity Log**. By default, you will see the **Logs** section where you can run queries and view log data.
4. In the **Logs** section, you can write and [execute queries](https://learn.microsoft.com/en-us/azure/azure-monitor/logs/get-started-queries) using the **Azure Monitor Query Language (KQL)** to retrieve specific logs. You can filter logs based on time range, specific instance, or other relevant parameters. See the example below.

### Query example

```kql theme={null}
// Retrieve logs with specific table
  ContainerInstanceLog_CL
  | project TimeGenerated, Message
```

Where `TimeGenerated` and `Message` are fields from the `ContainerInstanceLog_CL` table.

This query will retrieve the `timestamp` (TimeGenerated) and the `log message` (Message) from the logs generated by container instances.

***

## Export logs

With the export logs feature enabled, logs from the {s_runner} are automatically sent to Matillion's systems for support. They are stored in the same cloud provider and geolocation as your {maia} account.

### Enable log export

To enable log export, set the `EXPORT_LOGS` optional {s_runner} environment variable to `true`. By default, `EXPORT_LOGS` is set to `false`.

### Log retention

Logs are stored for 30 days, after which they are removed.

### Stored data

The same information that is visible in your local {s_runner} logs will be available to Matillion for support purposes. Matillion cannot access any data originating from your data sources or cloud data warehouse.

### Further information

For more information on data gathering at Matillion, read [Data gathering](/docs/security/data-gathering/#types-of-data) and our [Security overview](/docs/security/security-overview).