KubeArmor deep dive: Securing Kubernetes with eBPF and LSM

As part of our Kubernetes security series, we recently explained what Falco is, what Tetragon is, how we configure them, and what type of observability data we can get from those agents.

Today, we will focus on another runtime agent relying on eBPF: KubeArmor.

Here’s a quick overview of the contents of this blog post:

KubeArmor overview

Like other runtime agents, KubeArmor relies on eBPF to detect suspicious events in your cluster. Depending on your policy, it may also leverage Linux Security Modules (LSMs), such as AppArmor, SELinux, and LSM BPF, to secure your cluster.

KubeArmor will report observability by pushing process, file, network, or capabilities usage events if enabled.

KubeArmor architecture

KubeArmor comprises various components, such as the daemonset, the operator, the controller, and the kubearmor-relay.

When you deploy KubeArmor, you’ll have the core component that all security agents rely on, the daemonset, which will deploy the BPF probe. This is the most critical component because the person responsible for reporting the events is responsible for applying the proper enforcement.

Then, you’ll have one of KubeArmor's essential elements: the operator. A Kubernetes deployment allows you to turn the KubeArmor events on or off based on annotations and configure our policies. The policies are configured with a new CRD introduced by KubeArmor: kubearmorpolicy.

The actual controller deployment spreads the configuration to the BPF probes running in our cluster.

Finally, there is the kubearmor-relay, where the KubeArmor agents expose events through the gRPC endpoint. The relay server will pull the agent's events using the gRPC protocol. The relay is then responsible for sharing the output in various protocols. If we want to collect events using log agents, we need to modify the relay's settings to generate the events and alerts more explicitly in stdout. I guess it has been turned off for performance reasons, but KubeArmor's “observability” journey is more complicated than the other agents.

Events in KubeArmor

KubeArmor can produce events from various cluster dimensions to report operations related to processes, files, networks, and last capabilities. Those events would only be produced if turned on. Add the correct annotation to the targeted Kubernetes component to achieve this visibility.

The annotation is Kubearmor-visibility, and the value is the list of dimensions you want to enable, such as file, network, process, and capabilities.

This annotation can be placed in various areas of your cluster, at the pod level, at the deployment level, or at the node or namespace level. Let’s examine the structure of those events.

Here is a process event:

It returns the process executed, the parent process, the Kubernetes metadata, and the system call type.

Here is a network event:

The operation, data, and resource field will provide different details based on the type of event. Operation is the field defining the source of the event, process, file, and network.

Here is an example of a file event:

That is the default structure of the events when building a policy. You’ll have similar information, but an Action field will be added to help you understand if it is an alert or a simple blocked event. The action could be either blocked or audited.

It also adds an Enforcer field to indicate the type of solution responsible for this detection: BPFLSM, eBPF monitor, Etc.

The other interesting part is to define policies' default behavior. For example, you could have a policy that allows specific events, but if your namespace has a default behavior defined to block them, then the event would be blocked.

Three annotations are available: kubearmor-file-posture, kubearmor-network-posture, and kubearmor-capabilities-posture, which could be equal to audit or block. These annotations could be placed at the namespace level. There is also a default posture for KubeArmor in general. This could be defined by configuring the operator using a specific CRD: KubeArmorConfig.

KubeArmor policies

If enabled, KubeArmor handles events related to files, processes, and networks. However, if you want to build alerting (to reduce the number of events) or more block-specific behavior, then you would need to configure a KubeArmorPolicy.

Various CRDs help us configure the policy. Technically, it’s the same configuration type but targets different types of objects.

Let’s look at the definition of this policy object.

All the policies will have a similar structure, such as a name, but you can assign details related to the alert, a message, a tag, and a severity. The object will have a Selector to limit or filter a specific workload in the case of a kubearmorpolicy or a namespace selector in the case of a cluster policy. Last, the policy has an action field to define what KubeArmor should do with this event: allow it, block it, or simply alert it with an audit.

The policy will define the type of events we’re targeting. Several targets exist: the process, file, network, capabilities, and last syscalls.

Each section will specify the right combinations of filters by defining a rule on a file path, process, networking actions, or syscalls that would like to be alerted (or blocked), and each has its filtering section. Here’s an example:

Here are the various possible filters. We can create a matching rule on the path, directory, or process details. For the file type, there are specific filtering rules:

Here are the network matching rules:

You can limit it to a specific protocol and communication pattern. It’s a bit tricky to set the proper configuration.

Here are the matching rules available for the capabilities:

And the following one for syscalls:

Even if it has a variety of settings, a good level of expertise is required before configuring those policies. I suggest turning off all the KubeArmor events and configuring your policies based on the observed events.

KubeArmor does not provide out-of-the-box policies for everyday vulnerable events. However, you can review the existing policy templates in their documentation.

KubeArmor also has a CLI to interact with the operator and use the command KubeArmor logs to look at the event produced if you have not turned on the stdout logging option. More interestingly, it has an action KubeArmor recommends that could recommend the policies you need to deploy on a given namespace.

Observability with KubeArmor

Like all security agents, the data produced is designed to help you extend the level of your observability, but it requires some settings to enable logging properly.

The OpenTelemetry collector receiver

The KubeArmor community has worked on an OpenTelemetry collector receiver to get the KubeArmor events. This is an excellent initiative because you won’t have to enable the stdout settings anymore. The only downside is that the receiver lives in the KubeArmor organization, not the opentelemetry-contrib repo. It would be great if it were submitted upstream so everyone can take advantage of the default contrib version. The major advantage is that we can collect logs from our collector directly by interacting with the agents' grpc interface. The other great advantage is that all the event files would be considered log attributes, simplifying the processing tasks.

The KubeArmor Sidekick

KubeArmor also worked on a project called Sidekick. Like the FalcoSidekick project, it is designed to collect the alerts retrieved by KubeArmor and push them to several market solutions, such as Slack, Teams, or other observability solutions.

Honestly, this project is not super helpful. Once you have configured the relay server to use a log agent or simply the KubeArmor receiver in our collector, the event produced is not difficult to parse because the structure of KubeArmor events is standard. Falco or Tetragon provide more fields depending on the event, so the way you need to parse depends on the type of event, but in KubeArmor, it is very structured. Any log agent can process all the events with a common rule.

The Prometheus exporter for KubeArmor

KubeArmor does not have a default Prometheus exporter available. However, when deploying KubeArmor, we have a service named kubearmor-controller-metrics-service, and I have tried to connect to it to scrape metrics from there. Still, it requires a specific TLS setting to work.

Otherwise, in the Kubearmor repo, a project provides a Prometheus exporter that will expose metrics. Again, this project connects to the Kubearmor agents, like the relay server, and produces metrics out of the alerts.

Because the relay server seems to do the same, it would make sense to connect it to the relay server to reduce the pressure we could put on the agents.

The metric exposed in this exporter does not add more details than the one already available in the logs. If you collect the logs, the metrics would have a limited value except for reducing the data we need to ingest.

I found no metrics or data to help us measure the health of the KubeArmor agents and operators, which is disappointing. This is missing crucial information. We want to report security events, but we also want to ensure that KubeArmor is healthy.