From a0d8562dad7487338b80fb29731eab6132bec637 Mon Sep 17 00:00:00 2001 From: Nicolai Ort Date: Mon, 21 Jul 2025 13:47:17 +0200 Subject: [PATCH] docs(day1): Added opensearch talk --- content/day1/06_opensearch.md | 62 +++++++++++++++++++++++++++++++++++ 1 file changed, 62 insertions(+) create mode 100644 content/day1/06_opensearch.md diff --git a/content/day1/06_opensearch.md b/content/day1/06_opensearch.md new file mode 100644 index 0000000..9262bd3 --- /dev/null +++ b/content/day1/06_opensearch.md @@ -0,0 +1,62 @@ +--- +title: OpenSearch - The Open source Path to Search and Observability +weight: 6 +tags: + - observability +--- + + + + +A introduction to opensearch and "look at the cool new features in 3.o" + +## History + +- Background: Was born out of the elasticsearch license change as a fork by AWS +- Since Late 2024: A part of the linux foundation + +## Platform + +### Elements + +- Core: Distributed Search Engine with Vector DB +- Dashboards: UI with Dashboards, Alerts, Reports, ... +- Data Preppers: Prepare Data for ingest and indexing + +```mermaid +graph LR +DataSource-->DataPrepper-->|Ingest into|Core +subgraph Core + LogIndex + TraceIndex + TimeseriesIndex +end +``` + +### Use-Cases + +- Search: Well - search (e.g. for Amazon's product search) + - Free text search & fuzzy search + - Faceting (Generate Attributes based on the content and search by them) + - Geospacial Search & Vector Search +- Observability: Log analytics + - Log analytics with specialized query language or natural language + - OTEL and Jaeger Support + - Query federation to prometheus for metrics +- AI/ML: It's a vector database + - Vector database that can be used for embeddings + - Multimodal search for text image and video with one model or one model per mode + - Neural sparse search and simmilarity search + - MCP and bring your own model support +- Security: Tracing, log detection and so on + +### Performance + +- Problem: Large Datasets are usually slow +- Solution: Specialized improvements + +## News: Openstack 3.0 + +- Baseupgrades for Lucene, JDK and Node (yay) +- Performance: Reader/Writer-Seperation, gRPC Support, Pull-based injection in addition to pushed-based +- Improvements: Cross cluster search for traces, better nested json support \ No newline at end of file