Managed Kubernetes Service - Audit logs forwarding

Learn how to enable the forwarding of audit logs from your OVHcloud Managed Kubernetes Service (MKS) cluster to Logs Data Platform (LDP), a platform that helps you store, archive, query, and visualize your logs.

If you want to discover Logs Data Platform before reading this guide, please refer to the Log Data Platform introduction guide.

Glossary

• Logs Data Platform: a fully managed and secured log management platform by OVHcloud. Find more information on the Logs Data Platform service page.
• Data Stream: a logical partition of logs that you create in an LDP account that you will use when ingesting, visualizing, or querying your logs. Multiple sources can be stored in the same data stream, and it is the unit for defining a logs pipeline (retention policy, archiving, live streaming...), access rights, and alert policies.
• Logs forwarding: feature integrated into an OVHcloud product to ingest logs from its services to a Data Stream of an LDP account in the same OVHcloud account. The feature has to be enabled by the customer and per service.
• Logs forwarding Subscription: when enabling the logs forwarding for a given OVHcloud service to a given LDP Data Stream, a Subscription is created and attached to the Data Stream for further management by the customer.
• Request Stage and Audit Level: audit records begin their lifecycle inside the kube-apiserver component. Each request at each stage of its execution generates an audit event, which is then pre-processed according to the policy defined by the OVHcloud Managed Kubernetes Service. This policy defines which audit events as well as which audit level (meaning which events data) are forwarded through the Audit logs of your Kubernetes cluster. For more details, refer to the Kubernetes Auditing documentation.

Concept

What are the audit logs of a Managed Kubernetes cluster?

Managed Kubernetes audit logs provide a security-relevant, chronological set of records documenting the sequence of actions in your cluster.

The cluster audits the activities generated by users, by applications that use the Kubernetes API, and by the control plane itself. Auditing allows cluster administrators to know what happened, when, who initiated it, from where was it initiated, on which service it happened, and where was it going. For further details about cluster audits, refer to the Kubernetes auditing documentation.

The OVHcloud Managed Kubernetes Service has defined an audit policy, enabling you to retrieve logs about:

• requests to authorization resources to help troubleshoot authentication issues
• configmap and secret changes in all namespaces at the Metadata audit level
• changes to resources at RequestResponse level (maximum verbosity) to create, patch, update, and delete verbs
• changes to resources at the Request level for other verbs
• all other requests at the Metadata level

Note:

• No logs for requests by the "system:kube-proxy" on endpoints or services
• Low level of verbosity (level Metadata) for endpoints containing sensitive data like tokenreview
• No logs for any request in the RequestReceived stage
• No logs for health checks and requests for apiserver metrics

For details about information captured in Kubernetes Audit logs, you can refer to the Kubernetes public documentation.

Below is an example of an audit log generated by a Kubernetes cluster. Note the example is not exhaustive.

{
   "kind":"Event",
   "apiVersion":"audit.k8s.io/v1",
   "level":"Metadata",
   "auditID":"XXXXXXX-XXXX-XXXX-XXXX-XXXXXXXX",
   "stage":"ResponseStarted",
   "requestURI":"/api/v1/namespaces/searching-worker/configmaps?......",
   "verb":"watch",
   "user":{
      "username":"system:node:search-workers-node-81097c",
      "groups":[
         "system:nodes",
         "system:authenticated"
      ]
   },
   "sourceIPs":[
      "10.10.0.43"
   ],
   "userAgent":"kubelet/v1.26.4 (linux/amd64) kubernetes/f89670c",
   "objectRef":{
      "resource":"configmaps",
      "namespace":"searching-worker",
      "name":"kube-root-ca.crt",
      "apiVersion":"v1"
   },
   "responseStatus":{
      "metadata":{

      },
      "code":200
   },
   "requestReceivedTimestamp":"2024-01-01T11:59:06.428015Z",
   "stageTimestamp":"2024-01-01T11:59:06.428962Z",
   "annotations":{
      "authorization.k8s.io/decision":"allow",
      "authorization.k8s.io/reason":""
   }
}

Requirements

• A Logs Data Platform (LDP) account with at least one active Stream configured (see our Quick Start for Logs Data Platform guide for the necessary steps)

[primary]If you are not familiar with all the LDP Stream configuration possibilities, simply create a new one with the default options (indexing and websocket enabled, long-term storage disabled) for the purpose of this guide.

• An up-and-running Managed Kubernetes Service (MKS) cluster (see our Creating a Kubernetes Cluster guide for the necessary steps)
• The LDP account and MKS cluster must belong to the same OVHcloud account

Instructions

Enabling Audit Log Forwarding using the OVHcloud Control Panel

Step 1 - Access the Audit Logs section of your Managed Kubernetes Service cluster

Log in to the OVHcloud Control Panel, go to the Public Cloud section and select the Public Cloud project concerned.

Under Containers & Orchestration select the Managed Kubernetes Service in the left-hand menu.

Select the name of your cluster.

The next page will show you your cluster details.

Step 2 - Create a Logs Data Platform Subscription

In the Audit Logs tab, you can view live audit logs from your cluster.

To activate logs forwarding to LDP, simply click the Subscribe button on the right-hand side of your screen.

Step 3 - Select your LDP Account and Data stream

From the dropdown list select the LDP Accounts you want, then select the Data Stream you would like to use from the list and click the Subscribe button.

Your subscription is now created and your MKS audit logs are forwarded!

Enabling Audit Log Forwarding using APIs

You will have to define the targeted Stream of one of your LDP accounts on which you want your logs to be forwarded to. The enablement of the forwarding will create a subscription for this stream ID. Note that the forwarding activation is free of charge, but you will be charged for the usage of the Logs Data Platform service as per the standard price plan. For LDP pricing, refer to the Logs Data Platform product page.

You can retrieve the API specifications in the OVHcloud API Portal.

Step 1 - Retrieve your target Stream (and ID)

List data streams of your Logs Data Platform account:

Get the details of a data stream:

Step 2 - Create your subscription

As in the example above, the POST request has a payload containing a streamId, which is the target data stream of your LDP account where you want your Kubernetes cluster audit logs to be forwarded. You also need to specify the 'kind' of log you want to forward. Note that the only currently supported value for Managed Kubernetes Service is 'audit' (you can find available kinds using the dedicated API call):

{
  "kind": "string", // The only supported value currently is 'audit'.
  "streamId": "18d602ec-af40-4000-8e59-41ecc8c23f80" // The streamID of the targeted Stream.
}

Result:

{
  "operationId": "f550aa1c-89ab-4b1a-81ae-4fba4959966f",
  "serviceName": "ldp-xx-xxxxx"
}

Retrieve the subscriptionId of the subscription you just created using your serviceName and kubeId.

Then you can retrieve the subscriptionId for further management purposes using Logs Data Platform read operation endpoint:

Result:

{
  "createdAt": "2024-01-31T15:45:25.286Z",
  "kind": "string",
  "resource": {
    "name": "string",
    "type": "string"
  },
  "serviceName": "string",
  "streamId": "string",
  "subscriptionId": "18d60324-b260-4000-83db-b484f4db6e80",
  "updatedAt": "2024-01-31T15:45:25.286Z"
}

How to use your Kubernetes Audit logs?

Now that your Kubernetes instance Audit logs are ingested and stored in your Logs Data Platform data stream, you can query your logs and build dashboards to have a graphical representation of your logs using the web-based UI of Graylog.

• Through the OVHcloud Control Panel, retrieve the LDP username (ex: logs-xx-xxxxx) and its password from your Logs Data Platform account home page. You can refer to our Quick Start for Logs Data Platform documentation.
• Open the Graylog web-ui. You can retrieve the link in your account home page or using your Access point depending on your account region (for example: the Beauharnois, Canada, region is https://bhs2.logs.ovh.com/).
• Log into Graylog using your Logs Data Platform username and password.
• Search through your logs across the data stream of your Logs Data Platform account. You can refer to Graylog writing search queries documentation for details on search syntax.

Refer to our visualizing, querying, and exploiting your logs documentation for more details about how to use your logs with Logs Data Platform, including how to:

• setup alerts,
• view the logs in real time through a WebSocket,
• build visualization with OpenSearch Dashboards,
• integrate with OpenSearch API,
• connect with Grafana.

How to manage your subscriptions?

At any point, you can retrieve subscriptions attached to your Logs Data Platform data stream and choose to disable the forwarding by canceling your subscription on your stream so that your Logs Data Platform stream doesn't receive your audit logs anymore.

Note that this doesn't delete the logs that have been stored before the subscription cancellation, as data stored in a logs stream is immutable unless you delete the entire stream.

To delete your subscription you can use the dedicated section in the OVHcloud Control Panel. Go to the Audit logs tab of your MKS cluster and click on Unsubscribe or use the following API route:

Result:

{
  "operationId": "a550aa1c-89ab-4b1a-81ae-4fba4959966c",
  "serviceName": "string"
}

Go further

For more information and tutorials, please see our other Managed Kubernetes or Platform as a Service guides. You can also explore the guides for other OVHcloud products and services.

If you need training or technical assistance to implement our solutions, contact your sales representative or click on this link to get a quote and ask our Professional Services experts for a custom analysis of your project.