How to collect logs in Kubernetes with Loki and Promtail

This blog post is part of a Kubernetes series to help you initiate observability within your Kubernetes cluster.

This article also summarizes the content presented on the “Is it Observable” episode "how to collect logs in k8s using Loki and Promtail", briefly explaining:

Why logging is important

Logging has always been a good development practice because it gives us insights and information on what happens during the execution of our code. Logs are often used to diagnose issues and errors, and because of the information stored within them, logs are one of the main pillars of observability.

Why does logging need to be standardized?

Since there are no overarching logging standards for all projects, each developer can decide how and where to write application logs.

There are usually two options:

The first one is to write logs in files. This is a great solution, but you can quickly run into storage issues since all those files are stored on a disk. One way to solve this issue is using log collectors that extract logs and send them elsewhere. However, this adds further complexity to the pipeline.

The second option is to write your log collector within your application to send logs directly to a third-party endpoint. Here the disadvantage is that you rely on a third party, which means that if you change your login platform, you'll have to update your applications.

In conclusion, to take full advantage of the data stored in our logs, we need to implement solutions that store and index logs. The usage of cloud services, containers, commercial software, and more has made it increasingly difficult to capture our logs, search content, and store relevant information. We're dealing today with an inordinate amount of log formats and storage locations. Maintaining a solution built on Logstash, Kibana, and Elasticsearch (ELK stack) could become a nightmare.

To simplify our logging work, we need to implement a standard.

Standardizing Logging

We use standardized logging in a Linux environment to simply use “echo” in a bash script.

For example:

When you run it, you can see logs arriving in your terminal. The “echo” has sent those logs to STDOUT.

In a container or docker environment, it works the same way. Logging information is written using functions like system.out.println (in the java world). When we use the command: docker logs <ID of our container>, docker shows our logs in our terminal.

In the docker world, the docker runtime takes the logs in STDOUT and manages them for us. It will take it and write it into a log file, stored in var/lib/docker/containers/<ID of the container>. Each container will have its folder.

If we're working with containers, we know exactly where our logs will be stored!

Pushing the logs to STDOUT creates a standard. We can use this standardization to create a log stream pipeline to ingest our logs.

Centralized logging

Now we know where the logs are located, we can use a log collector/forwarder.

This tool is in charge of:

Once logs are stored centrally in our organization, we can then build a dashboard based on the content of our logs.

Now, let’s have a look at the two solutions that were presented during the YouTube tutorial this article is based on: Loki and Promtail.

Introduction to Grafana Loki

Loki is a horizontally-scalable, highly-available, multi-tenant log aggregation system built by Grafana Labs. This solution is often compared to Prometheus since they're very similar. To differentiate between them, we can say that Prometheus is for metrics what Loki is for logs.

Loki is made up of several components that get deployed to the Kubernetes cluster:

Loki server serves as storage, storing the logs in a time series database, but it won’t index them. To visualize the logs, you need to extend Loki with Grafana in combination with LogQL.

Loki agents will be deployed as a DaemonSet, and they're in charge of collecting logs from various pods/containers of our nodes. Loki supports various types of agents, but the default one is called Promtail.

Introduction to Promtail

Promtail does the following actions:

Promtail has a configuration file (config.yaml or promtail.yaml), which will be stored in the config map when deploying it with the help of the helm chart.

In the config file, you need to define several things:

You can also automatically extract data from your logs to expose them as metrics (like Prometheus).

Loki configuration

Loki’s configuration file is stored in a config map. Here you can specify where to store data and how to configure the query (timeout, max duration, etc.). W

When deploying Loki with the helm chart, all the expected configurations to collect logs for your pods will be done automatically. If you need to change the way you want to transform your log or want to filter to avoid collecting everything, then you will have to adapt the Promtail configuration and some settings in Loki.

Tutorial: collecting logs with Loki and Promtail

In this tutorial, we will use the standard configuration and settings of Promtail and Loki. We want to collect all the data and visualize it in Grafana.

The full tutorial can be found in video format on YouTube and as written step-by-step instructions on GitHub.

Here are the links: