Join us at KCD UK, October 21-22 in Edinburgh. Learn more
By Patrick Stephens•June 2025

Kubernetes logging with Fluent Bit

I will be showing you the “normal” (most wildly deployed) example of using Fluent Bit to collect container logs from a K8S cluster. We can then extend this to send to any output supported by Fluent Bit as well as include getting metrics and traces.

For the examples below I am using Kubernetes-in-docker (KIND) to run them but this is a vanilla K8S distribution so should be applicable to all others. We also use the helm tooling as this is the only officially supported approach: https://docs.fluentbit.io/manual/installation/kubernetes

Refer to our Github repo as well for the various examples: https://github.com/FluentDo/fluent-bit-examples

Mounting container logs

To read the logs we have to mount them from the host so typically we deploy Fluent Bit as a daemonset with a hostPath mount to the local log files. One important thing to watch out for is dangling symlinks being mounted: make sure you mount the links and their targets if required so they can be resolved. Using the official helm chart will automatically create a daemonset with these files all mounted for you: https://github.com/fluent/helm-charts/blob/main/charts/fluent-bit

The logs being ingested should follow the K8S standard and container runtime format, Fluent Bit provides two default parsers that let you handle this file format automatically and deal with the various edge cases when lines are split by the kubelet. The documentation shows the recommended usage for containers and in general I always say to follow this - do not define your own custom parsers for the logs unless you know what you’re doing.

pipeline:
  inputs:
    - name: tail
      path: /var/log/containers/*.log
      tag: kube.*
      multiline.parser: docker, cri

In the above example we are assuming there is a mounted set of container logs at /var/log/containers which is the default location used by the helm chart and most distributions (remember this may be symlinked as well). We then attempt to parse with the built-in multiline parsers for the docker and cri container runtime log formats.

Previously Fluent Bit also provided Multiline or Docker_Mode configuration options but these are deprecated now and only included for legacy usage - do not mix them with the new multiline.parser options but instead just use the new options.

Application-specific parsing

The parsers used above are mutually exclusive: the first one that matches will be used, they will not be applied in order so you cannot first do CRI format parsing then another application specific. If you want to first parse the default kubelet format then attempt some application specific parsing you should add a processor or filter to do it like so:

parsers:
  - name: my-custom-parser
    format: json

pipeline:
  inputs:
    - name: tail
      path: /var/log/containers/*.log
      tag: kube.*
      multiline.parser: docker, cri
      processors:
        logs:
          - name: parser
            parser: my-custom-parser
            key_name: log

We can see here that after we have finished processing the data in the input file, we then pass it to a custom parser that operates on the log key. You can use any of the other filters/processors in the same way and apply multiple as required.

If a parser does not apply then the data is left alone and unchanged - there is no data loss from an invalid parser. Potentially you can chain a series of different parsers and only those that apply will affect the data: for example with two wildly different log formats just try one parser then the other and it will apply whichever matches first.

Preventing duplicate or missing data

One other thing you may want to consider is the fact that your pods may be evicted or have delays in scheduling for various reasons so you want to ensure when a new pod starts it continues from wherever the last one left off. Otherwise you may miss data since the pod started or send duplicate data that another pod has already sent. This can also be true when running an agent outside of K8S, e.g. the Fluent Bit service starts up later than something you want to track logs from.

Fluent Bit provides a way to support this by persisting the offset in the input file it last read up to with a simple sqlite database you can optionally provide via the db parameter. The db file tracks which files have been read and how much of the file as well so that when Fluent Bit is restarted/rescheduled/etc. then it will continue from where it left off.

pipeline:
  inputs:
    - name: tail
      path: /var/log/containers/*.log
      tag: kube.*
      multiline.parser: docker, cri
      db: /var/log/fluent-bit.db

The simple example above shows how to use the same volume as the container logs to write the database file. If you want to use read only mounts for the log volume then you can use a separate volume with write access and set the path to it for the db option. This example would also work for an agent deployed directly on the node.

The sqlite database tracks files by inode value so it handles log file rotation automatically: when the file is rotated the old inode is read until completion then we carry on with the new inode for the next file. The database file can also be looked at via any sqlite tooling you may want to access it with.

The database file is not intended to be shared across processes or nodes - the inode values will not be unique for example on different nodes. As such ensure it is linked to only one Fluent Bit process at a time. You need a writable location for this database that automatically matches to the right pod each time it is started. A simple way to do this is to use a hostPath mount so the same config is shared across all pods but the actual filesystem is specific to each node then.

If your pod is running and persisting its file information to the database then is evicted and a new pod starts then the database must be linked to the new pod automatically which is why a hostPath can be a simple way to do this when running as a daemonset: it will always be for that specific node. Similarly only one pod should be writing to a specific database file at a time. For other deployment options (e.g. maybe you are running as a deployment instead of a daemonset) then you can figure out an alternative like using named directories or files in a persistent volume shared across all pods.

Kubernetes meta-data

Fluent Bit provides a simple kubernetes filter you can use to automatically query the K8S API to get pod meta-data (labels and annotations) to inject into the records you are sending to your outputs. This filter will also allow you to do some additional custom parsing and other behaviour (e.g. you can ignore logs by label) on the log records it receives.

pipeline:
  inputs:
    - name: tail
      path: /var/log/containers/*.log
      tag: kube.*
      multiline.parser: docker, cri
      processors:
        logs:
          - name: kubernetes
            kube_tag_prefix: kube.var.log.containers.

This relies on the K8S standard for kubelet log filenames which includes enough information to extract and then query the API server with. The log filename will include the namespace, pod and container names. From this we can then make a query to the K8S API to get the rest of the metadata for that specific container in that specific pod.

To ensure that the K8S filter in Fluent Bit has this information it must be provided the log filename in the tag. The tail input filter will do this if you provide a wildcard in the tag name, i.e. tag: kube.* will be automatically expanded to the full filename for the tag (with special characters replaced) so something like kube.var.log.containers.namespace_pod_container. The K8S filter has two configuration parameters relevant here: https://docs.fluentbit.io/manual/pipeline/filters/kubernetes#workflow-of-tail-and-kubernetes-filter

  • Kube_tag_prefix: defaults to kube.var.log.containers. and is stripped off the tag to just give you the filename. This must be correct otherwise you will get nonsense information which will then fail when queried. If you change the default tag to something other than kube.* or files are mounted to a different path then you will need to ensure this is correct.
  • Regex_Parser: this is the parser used to extract the information from the filename after it is stripped, i.e. it gets the namespace and other information. You likely do not need to change this.

The other thing to ensure you have configured correctly is RBAC to allow your Fluent Bit pods to query this information from the K8S API.

If you are seeing missing information from the kubernetes filter then a good first step is to set log_level debug which will provide you the HTTP requests you are making to the K8S API (check the pod information here is correct - usually down to mismatches in tag if not) and the HTTP responses (which can show you invalid RBAC configuration).

Helm chart deployment

We will use the official helm chart to deploy Fluent Bit with the following configuratIon:

      service:
        # Required for health checks in the chart
        http_server: on
      pipeline:
        inputs:
          - name: tail
            tag: kube.*
            path: /var/log/containers/*.log
            multiline.parser: docker, cri
            processors:
              logs:
                - name: kubernetes
                  kube_tag_prefix: kube.var.log.containers.
                  merge_log: on
        outputs:
           - name: stdout
             match: "*"

This is a very simple standalone configuration assuming a daemonset with a hostPath mount of /var/log, i.e. the helm chart defaults. We use the kubernetes filter as discussed above to retrieve additional information about each container log from the K8S API.

The merge_log parameter is a powerful tool to look at the log data and extract JSON key-value pairs or apply custom parsers you can specify by annotations on the pods: https://docs.fluentbit.io/manual/pipeline/filters/kubernetes#kubernetes-pod-annotations

We specify the http_server as the helm chart defaults to enabling K8S health checks on the pods which hit that endpoint so if it is not present the pods will never be marked healthy. You can disable them and then you do not need http_server set which will remove running an HTTP endpoint on port 2020: https://github.com/fluent/helm-charts/blob/54f30bd0c98d7ef7b7100c14d6cbd52236cb34e4/charts/fluent-bit/values.yaml#L201-L209

You can include files into an overall configuration as well. The include files are good ways to reuse common configuration or isolate specific configuration into separate files (e.g. for separate teams to control or to make it simpler for large configurations with well-named includes). Each configuration file is read independently and the data loaded into memory which means that you can also include “classic” configuration files into a top-level YAML configuration file or even mix-and-match:

includes:
  - yaml-include-1.yaml
  - classic-include-2.conf

This can be useful if you want to move things piecemeal a bit at a time for example.

One thing to note in YAML is to always quote wildcards as they can be treated as special characters: a good tip is to quote things if you start seeing configuration format errors just in case this is the problem.

YAML format with Helm chart

Currently the helm chart is defaulting to the old format configuration but we can use YAML configuration with a simple values file like so:

config:
  extraFiles:
    fluent-bit.yaml: |
<YAML config here>

args:
  - --workdir=/fluent-bit/etc
  - --config=/fluent-bit/etc/conf/fluent-bit.yaml

We override the default configuration file to the YAML configuration we have added to the configmap used by the helm chart. This is a slight workaround in that it leaves all the legacy configuration alone and just adds a new YAML one to use.

An example is provided here: https://github.com/FluentDo/fluent-bit-examples/tree/main/helm-yaml-config

We can run up a cluster with KIND then deploy the helm chart like so:

kind create cluster
helm repo add fluent https://fluent.github.io/helm-charts --force-update
helm repo update
helm upgrade --install fluent-bit fluent/fluent-bit --values ./values.yaml

Remember that with helm you can use helm template to generate you the actual YAML output (similar to what a lot of GitOps/IaC tools like Argo, etc. use to manage helm deployments) and verify it or use it directly.

Looking at the logs from the Fluent Bit pods should show you container logs with K8S metadata added:

$ kubectl logs -l "app.kubernetes.io/name=fluent-bit,app.kubernetes.io/instance=fluent-bit"
...
[0] kube.var.log.containers.kindnet-vdwzr_kube-system_kindnet-cni-6c3fd58a5ca253428cbc7de0c54cb107bfac4c5b8977f29107afab415d376a4c.log: [[1749731282.036662627, {}], {"time"=>"2025-06-12T12:28:02.036662627Z", "stream"=>"stderr", "_p"=>"F", "log"=>"I0612 12:28:02.036155       1 main.go:297] Handling node with IPs: map[172.18.0.2:{}]", "kubernetes"=>{"pod_name"=>"kindnet-vdwzr", "namespace_name"=>"kube-system", "pod_id"=>"4837efec-2287-4880-8e05-ed51cc678783", "labels"=>{"app"=>"kindnet", "controller-revision-hash"=>"6cd6f98bf8", "k8s-app"=>"kindnet", "pod-template-generation"=>"1", "tier"=>"node"}, "host"=>"kind-control-plane", "pod_ip"=>"172.18.0.2", "container_name"=>"kindnet-cni", "docker_id"=>"6c3fd58a5ca253428cbc7de0c54cb107bfac4c5b8977f29107afab415d376a4c", "container_hash"=>"sha256:409467f978b4a30fe717012736557d637f66371452c3b279c02b943b367a141c", "container_image"=>"docker.io/kindest/kindnetd:v20250512-df8de77b"}}]
[1] kube.var.log.containers.kindnet-vdwzr_kube-system_kindnet-cni-6c3fd58a5ca253428cbc7de0c54cb107bfac4c5b8977f29107afab415d376a4c.log: [[1749731282.036770275, {}], {"time"=>"2025-06-12T12:28:02.036770275Z", "stream"=>"stderr", "_p"=>"F", "log"=>"I0612 12:28:02.036253       1 main.go:301] handling current node", "kubernetes"=>{"pod_name"=>"kindnet-vdwzr", "namespace_name"=>"kube-system", "pod_id"=>"4837efec-2287-4880-8e05-ed51cc678783", "labels"=>{"app"=>"kindnet", "controller-revision-hash"=>"6cd6f98bf8", "k8s-app"=>"kindnet", "pod-template-generation"=>"1", "tier"=>"node"}, "host"=>"kind-control-plane", "pod_ip"=>"172.18.0.2", "container_name"=>"kindnet-cni", "docker_id"=>"6c3fd58a5ca253428cbc7de0c54cb107bfac4c5b8977f29107afab415d376a4c", "container_hash"=>"sha256:409467f978b4a30fe717012736557d637f66371452c3b279c02b943b367a141c", "container_image"=>"docker.io/kindest/kindnetd:v20250512-df8de77b"}}]
...

You can see from these example logs that various K8S metadata is nested under a kubernetes key.

Next: OTEL integration →