Nicolai Ort
3e4fbb616b
All checks were successful
Build latest image / build-container (push) Successful in 53s
88 lines
2.5 KiB
Markdown
88 lines
2.5 KiB
Markdown
---
|
|
title: CRD Data Architecture for Multi-Cluster Kubernetes
|
|
weight: 4
|
|
tags:
|
|
- rejekts
|
|
- multicluster
|
|
---
|
|
|
|
{{% button href="https://www.youtube.com/watch?v=e1BmT0jc_Fs" style="warning" icon="video" %}}Watch talk on YouTube{{% /button %}}
|
|
<!-- {{% button href="https://docs.google.com/presentation/d/1nEK0CVC_yQgIDqwsdh-PRihB6dc9RyT-" style="tip" icon="person-chalkboard" %}}Slides{{% /button %}} -->
|
|
|
|
## Background
|
|
|
|
CRDs:
|
|
- Platform: Apacke Spark, Argo, Jupyter Notebooks, ...
|
|
- Tier: Parts of a platform that include access policies, ressource usages and network controls -> e.g. beta or prod
|
|
- ClusterSet: Shards within a tier (aka availability zone)
|
|
- Cluster: Part of a Clusterset that can be destroyed/recreated
|
|
- ComputeNamespace = Namespace + ServiceAccount + LimitRange + ResourceQuota + RBAC
|
|
- Part of a clusterset, all clusters which are part of the set have the same ComputeNamespace
|
|
|
|
## Goals & Challenges
|
|
|
|
Scale: 9000 Software Engineers
|
|
|
|
Challenges:
|
|
- Scaling
|
|
- Decomposition
|
|
|
|
Goal: Manage software platforms on kubernetes via kubernetes utilizing opterators
|
|
|
|
## KEPs by the SIG-MultiCluster
|
|
|
|
### Cluster Profiles
|
|
|
|
- Name
|
|
- Manager
|
|
- Status
|
|
- K8S Version
|
|
- Conditions (Helath)
|
|
- Cluster Access, options
|
|
- Work API (another proposal)
|
|
- Push via OIDC
|
|
- Push with Secret
|
|
- Zertificate Auth
|
|
|
|
### ClusterSet
|
|
|
|
- Within a ClusterSet namespace sameness applies -> All namespaces are the same in all clusters of a set
|
|
- Mutation = Delete and recreate cluster
|
|
|
|
### Clusternames
|
|
|
|
- Unique Name
|
|
- Valid RFC 1123 DNS label
|
|
|
|
### Cluster Inventory
|
|
|
|
- All ClusterProfiles should reside in a dedicated hub clusters
|
|
|
|
TODO: Diagram
|
|
|
|
## HA
|
|
|
|
- They use Kine (by k3s) to shim etcd to postgres
|
|
|
|
## Referential Integrity
|
|
|
|
- The CRDs all refer to each other (e.g. Tier --> Platform)
|
|
- Solution: CEL Expressions combined with webhooks and operators for business logic validation
|
|
|
|
## Resource Creation
|
|
|
|
- They have a simple api that is just a kubeapi wrapper
|
|
|
|
TODO: Diagram
|
|
|
|
## Q&A
|
|
|
|
- Why does everyone build their own multicluster stuff instead of utilizing open soruce
|
|
- Threir solution predates SIG-Multicluster
|
|
- They are using some open source solutions like karmada
|
|
- Could you explain ClusterProvile<->ClusterInventory again: He did, see livestream
|
|
- Where does your postgres run (does it run on the same kubernetes it shims)?
|
|
- There are no cross-dependencies
|
|
- The managment-clusters are lightweight
|
|
- Are you running a real kubernetes cluster for the hub?
|
|
- Nope we just use the apiserver |