Unlock the Power of Tracee: Real-time Kubernetes Security with eBPF

As part of our Kubernetes security series, we recently explained OPA Gatekeeper, Falco, Tetragon, and KubeArmor, covering the solution, the type of policies provided by the agent, and the observability data it exposes.

This article will focus on another runtime agent, Tracee, which also detects suspicious security behavior within our environment using eBPF. We will start with an introduction to Tracee, then move on to Tracee’s policies, and finally, its observability.

Introduction to Tracee

Tracee is an open source runtime agent provided by Aqua Security. Over the years, Aqua Security has built many valuable solutions to secure our K8S environment, like the Trivy Operator, that identifies vulnerabilities in the containers deployed in our cluster. More than exposing the vulnerabilities, it also controls that your deployment respects security best practices.

However, as mentioned in the episode about Falco, detecting vulnerabilities is excellent but not enough. That is why we need runtime agents that detect suspicious events. Aqua Security has built Tracee for that.

Tracee architecture

When you deploy Tracee, the daemonset deploys the runtime agents using an eBPF probe to capture kernel events. It also adds a Kubernetes deployment with the Tracee operator, adding a CRD to our cluster called Policy to configure your agents by building policies.

The eBPF probe generates events that are, by default, structured in JSON format. This allows you to see the detected events by looking directly at the logs produced by the Tracee agent. By default, Tracee only reports events related to the default policy deployed by Tracee after the installation.

The policy is a rule defining what we would like to track as Tracee events. If you were wondering if Tracee could apply enforcement rules by killing or blocking a specific event, the answer is no. Compared to the other agents, Tracee is more similar to Falco in detecting events.

Tracee events

As a runtime agent, Tracee will produce the time of the event, the details about the process, the user ID, the process ID, and information on the container, Kubernetes metadata, etc. Here is an example of an event generated by the Tracee agent:

As you can see, we have all the process details, the actual reference to the host process, the container details, and the k8S metadata. In addition, it includes details about the actual policy: event ID, event name, and policy name. If there were a syscall, it would include the syscall name and the actual args of the syscall. It’s like Tetragon but without complex kernel function mapping.

The event structure is almost similar to all the events that Tracee supports. Of course, the args would be different depending on the actual function or event.

You can configure the level of detail in your event. Tracee in Kubernetes has a config map that allows you to enable or disable details on the event. The configuration config map is tracee-config and will be in the same namespace as Tracee.

There are many options in Tracee's configuration. You can view them all in the Tracee docs.

It’s interesting that you can generate the logs in JSON, send the event directly using the fluent-forward-protocol, and that there’s a webhook option where we can post the events directly to an endpoint.

The section that defines the details that we would like to add or remove in our event is:

The default settings are good enough for us; enabling the details on the environment variable is too much.

Another aspect is enabling or disabling the Prometheus exporter, and we can configure the port used for the metrics.

Tracee allows you to configure the capabilities of the Tracee agent and the size of the cache used to receive the various eBPF events. It receives many events and checks if they match your policy before displaying them. So, the cache size would be very important if you don’t want to lose any events.

Tracee Policy

At first glance, I thought the Tracee policy was a bit light and couldn’t be extended from an observability perspective. On the contrary, the Tracee policy configuration is much easier than KubeArmor or Tetragon.

The policy has a scope and rules. Let’s look at what you can configure.

Policy Scope

Scope defines where the event will be coming from:

Once the scope is defined, you’ll create your rules made of one or several events.

Policy Rules

In the rules, you can define which type of event to filter for. In Tracee, there are 6 types of events:

You won’t have to define the type. You simply need to assign the right name corresponding to one of those categories.

The security category is interesting because it provides signatures that are predefined code filters that combine various syscall or network interactions to capture known security vulnerabilities.

When deploying Tracee, the default policy comprises a couple of those security signatures.

But Tracee is providing many other security signatures.

Let’s have a look at the default policy:

Out of this list, only container_create and container_remove are not in the security signature.

Of course, I won’t describe all of them because it would take forever, but I strongly recommend looking at those various signatures to enable the ones you need. Moreover, you should be able to react to dangerous security events when they occur.

Besides the security signature, you have the network events (including any network traffic; net_packt_http_request, or net_packet_http_response or specific DNS request), the syscall events, and extra events (like vfs_read or vfs_write).

Once this is done, you can apply various filters. First, we’re using a networking event, for example, in the actual data of the event. We want to drop the network traffic sent to localhost.

We can take advantage of the data exposed by the event. For example, in the case of HTTP, the event args field contains a data field and metadata. The data holds the details of the network communication, such as the src or destination. We can filter by data.dest=127.0.0.1 and data.src=127.0.0.1

Most network types will have similar signatures. Of course, you’ll find different details, such as the URL in the request event, the HTTP code, and the size of the response in the HTTP response. Instead of filtering the event data, we can filter on the event's global data, such as the process path, the pID, user ID, the pod namespace, etc. Building a rule is very easy.

Let’s say I want to produce events that report all the HTTP communication of the OpenTelemetry demo. I’ll simply create the following rule:

The simplicity of the configuration is mainly related to the predefined signatures. But one very exciting thing is that Tracee has an SDK allowing us to build your signature. You can define what to track and extract from the event. I think that is very smart.

Observability with Tracee

On the observability side, Tracee will extend your visibility by nature. Collecting Tracee events is quite simple: either collect the logs from Tracee with the log agent of your choice or enable fluent_forward to send the events directly to Fluent Bit or the collector with fluent forward support.

It would be great if Tracee could also offer the default option to send the events in OpenTelemetry log format using OTLP, HTTP, or grpcs. But again, getting the details we need with the current feature is very easy.

The other great thing is that most events have a standard structure. Parsing the events is relatively straightforward. The content of the event and the args fields that hold the data collected from the event would differ depending on the signature used. But for the rest, a common parsing rule would do the job.

Tracee exposes Prometheus metrics by default, but the documentation doesn’t describe the metrics produced and focuses on the events raised. Tracee uses a cache, so I would like to understand how it behaves. Unfortunately, we don’t have KPIs reporting Tracee’s or the operator's health, which is disappointing.

The only metrics that could help us in that direction are:

A view of how the cache behaves would help us tune Tracee to fit our needs.