81 lines
2.2 KiB
Markdown
81 lines
2.2 KiB
Markdown
---
|
|
title: "Fluent Bit v3: Unified Layer for Logs, Metrics and Traces"
|
|
weight: 7
|
|
tags:
|
|
- monitoring
|
|
---
|
|
|
|
The last talk of the conference.
|
|
Notes may be a bit unstructured due to tired note taker.
|
|
|
|
## Background
|
|
|
|
* FluentD is already graduated
|
|
* FluentBit is a daughter-project of FluentD (also graduated)
|
|
|
|
## Basics
|
|
|
|
* Fluentbit is compatible with
|
|
* prometheus (It can replace the prometheus scraper and node exporter)
|
|
* openmetrics
|
|
* opentelemetry (HTTPS input/output)
|
|
* FluentBit can export to Prometheus, Splunk, InfluxDB or others
|
|
* So pretty much it can be used to collect data from a bunch of sources and pipe it out to different backend destinations
|
|
* Fluent ecosystem: No vendor lock-in to observability
|
|
|
|
### Arhitectures
|
|
|
|
* The fluent agent collects data and can send it to one or multiple locations
|
|
* FluentBit can be used for aggregation from other sources
|
|
|
|
### In the kubernetes logging ecosystem
|
|
|
|
* Pods logs to console -> Streamed stdout/err gets piped to file
|
|
* The logs in the file get encoded as JSON with metadata (date, channel)
|
|
* Labels and annotations only live in the control plane -> You have to collect it additionally -> Expensive
|
|
|
|
## New stuff
|
|
|
|
### Limitations with classic architectures
|
|
|
|
* Problem: Multiple filters slow down the main loop
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph main[Main Thread/Event loop]
|
|
buffer
|
|
schedule
|
|
retry
|
|
fitler1
|
|
filter2
|
|
filter3
|
|
end
|
|
in-->|pipe in data|main
|
|
main-->|filter and pipe out|out
|
|
```
|
|
|
|
### Solution
|
|
|
|
* Solution: Processor - a seperate thread segmented by telemetry type
|
|
* Plugins can be written in your favourite language /c, rust, go, ...)
|
|
|
|
```mermaid
|
|
flowchart LR
|
|
subgraph in
|
|
reader
|
|
streamner1
|
|
processor2
|
|
processor3
|
|
end
|
|
in-->|pipe in data|main(Main Thread/Event loop)
|
|
main-->|filter and pipe out|out
|
|
```
|
|
|
|
### General new features in v3
|
|
|
|
* Native HTTP/2 support in core
|
|
* Contetn modifier with multiple operations (insert, upsert, delete, rename, hash, extract, convert)
|
|
* Metrics selector (include or exclude metrics) with matcher (name, prefix, substring, regex)
|
|
* SQL processor -> Use SQL expression for selections (instead of filters)
|
|
* Better OpenTelemetry output
|