Fluent Bit tips and tricks

My tips for debugging boil down to the usual software engineering idioms:

1. Simplify your stack
2. Reproduce your issue minimally
3. Increase your log level
4. Treat warnings as errors

Simplify

Always attempt to simplify any issues - do not debug things via another massive observability stack with components all along the way mangling or affecting the final data you are looking at.

Do not attempt to debug by looking at the output in another tool but instead use the stdout output or filter to see what the raw data looks like to the Fluent Bit deployment: this will help as well to identify whether the problem is with Fluent Bit or with the component after it.

Once the data looks right on the Fluent Bit side then it may resolve the issue but if not we know it is either a problem with sending the data to the next component or something that component is doing.

Quite often it is easy to make assumptions about what you think your data looks like which may be incorrect, a good example of this is parsing kubelet logs as people may accidentally think their data looks like the output of kubectl logs … when actually you need to parse the raw file on disk. Typically in this case people will want to use the default cri or docker multiline parsers to handle the kubelet format first in the tail input then have a separate parser filter (or processor attached) to then parse the logs after reconstructing them from the kubelet format (which is what kubectl logs does first).

Local reproducer

Simplifying the problem also helps you to set up a simple local reproducer with no (or minimal) external dependencies: too often we get issues raised with “random” failures seen using multiple inputs, filters and outputs. If you can provide a simple reproducer then that means others can easily help you and use it as a regression test (either by yourself when accepting new versions and/or by the Fluent project) as well if it turns out to be an issue.

A local reproducer also allows you to iterate quickly to test possible changes/tweaks/updates. I like to do this with a simple container-based stack using the dummy output (or even tail if mounting sample data into the container). For example you can easily have a local fluent-bit.yaml and test it like so:

vi fluent-bit.yaml
…
docker run --rm -it -v $PWD/fluent-bit.yaml:/fluent-bit/etc/fluent-bit.yaml:ro fluent/fluent-bit -c /fluent-bit/etc/fluent-bit.yaml

This will mount the YAML file in and provide it as the configuration file to use (with the -c parameter). You can even just mount the whole local directory if you’re passing in various things like parser configuration, test files, etc.

More debugging

Once you have simplified and got a reproducible version then you can start adding more logging and investigating the issues still outstanding.

Increase your log_level to debug to check for problems and help debug things: this is particularly useful when using HTTP APIs where it will show you the request and response, e.g. for misconfigured K8S filters you may see an RBAC failure or incorrect queries showing why you are not getting any K8S metadata being added.

Always treat warnings as errors at least when debugging things. A good example here is warnings about unknown parsers which are then just ignored and your data is not parsed as expected. Quite often that is down to using relative paths with an incorrect working directory before - switch to absolute paths to try that.

Missing Kubernetes metadata is usually down to mismatched tag configuration or invalid RBAC configuration. Using log_level debug will give you the HTTP requests being made and the responses from the K8S API server which usually helps figure out the problem.

The tail input functions like the tail -f Linux command by default so will only ingest new log lines added to the file (with a newline) after Fluent Bit is started. You can use read_from_head: true (not recommended) to read data already in the file or using a state file via the db: xxx parameter which will read a new file completely and maintain where it is up to from then on.

Use the --help command to get information about every specific plugin and all its configuration options in the specific version you are using, e.g. to get it for the OpenTelemetry output:

docker run --rm -it fluent/fluent-bit -o opentelemetry --help

If you find any missing or broken documentation then please send any suggestions or updates to: https://github.com/fluent/fluent-bit-docs.

Please reach out to us on the Fluent Bit slack channel where I can be found quite often!