2.2 KiB
2.2 KiB
title | weight | tags | |
---|---|---|---|
Fluent Bit v3: Unified Layer for Logs, Metrics and Traces | 7 |
|
The last talk of the conference. Notes may be a bit unstructured due to tired note taker.
Background
- FluentD is already graduated
- FluentBit is a daughter-project of FluentD (also graduated)
Basics
- Fluentbit is compatible with
- prometheus (It can replace the prometheus scraper and node exporter)
- openmetrics
- opentelemetry (HTTPS input/output)
- FluentBit can export to Prometheus, Splunk, InfluxDB or others
- So pretty much it can be used to collect data from a bunch of sources and pipe it out to different backend destinations
- Fluent ecosystem: No vendor lock-in to observability
Arhitectures
- The fluent agent collects data and can send it to one or multiple locations
- FluentBit can be used for aggregation from other sources
In the kubernetes logging ecosystem
- Pods logs to console -> Streamed stdout/err gets piped to file
- The logs in the file get encoded as JSON with metadata (date, channel)
- Labels and annotations only live in the control plane -> You have to collect it additionally -> Expensive
New stuff
Limitations with classic architectures
- Problem: Multiple filters slow down the main loop
flowchart LR
subgraph main[Main Thread/Event loop]
buffer
schedule
retry
fitler1
filter2
filter3
end
in-->|pipe in data|main
main-->|filter and pipe out|out
Solution
- Solution: Processor - a seperate thread segmented by telemetry type
- Plugins can be written in your favourite language /c, rust, go, ...)
flowchart LR
subgraph in
reader
streamner1
processor2
processor3
end
in-->|pipe in data|main(Main Thread/Event loop)
main-->|filter and pipe out|out
General new features in v3
- Native HTTP/2 support in core
- Contetn modifier with multiple operations (insert, upsert, delete, rename, hash, extract, convert)
- Metrics selector (include or exclude metrics) with matcher (name, prefix, substring, regex)
- SQL processor -> Use SQL expression for selections (instead of filters)
- Better OpenTelemetry output