--- title: CRD Data Architecture for Multi-Cluster Kubernetes weight: 4 tags: - rejekts - multicluster --- ## Background CRDs: - Platform: Apacke Spark, Argo, Jupyter Notebooks, ... - Tier: Parts of a platform that include access policies, ressource usages and network controls -> e.g. beta or prod - ClusterSet: Shards within a tier (aka availability zone) - Cluster: Part of a Clusterset that can be destroyed/recreated - ComputeNamespace = Namespace + ServiceAccount + LimitRange + ResourceQuota + RBAC - Part of a clusterset, all clusters which are part of the set have the same ComputeNamespace ## Goals & Challenges Scale: 9000 Software Engineers Challenges: - Scaling - Decomposition Goal: Manage software platforms on kubernetes via kubernetes utilizing opterators ## KEPs by the SIG-MultiCluster ### Cluster Profiles - Name - Manager - Status - K8S Version - Conditions (Helath) - Cluster Access, options - Work API (another proposal) - Push via OIDC - Push with Secret - Zertificate Auth ### ClusterSet - Within a ClusterSet namespace sameness applies -> All namespaces are the same in all clusters of a set - Mutation = Delete and recreate cluster ### Clusternames - Unique Name - Valid RFC 1123 DNS label ### Cluster Inventory - All ClusterProfiles should reside in a dedicated hub clusters TODO: Diagram ## HA - They use Kine (by k3s) to shim etcd to postgres ## Referential Integrity - The CRDs all refer to each other (e.g. Tier --> Platform) - Solution: CEL Expressions combined with webhooks and operators for business logic validation ## Resource Creation - They have a simple api that is just a kubeapi wrapper TODO: Diagram ## Q&A - Why does everyone build their own multicluster stuff instead of utilizing open soruce - Threir solution predates SIG-Multicluster - They are using some open source solutions like karmada - Could you explain ClusterProvile<->ClusterInventory again: He did, see livestream - Where does your postgres run (does it run on the same kubernetes it shims)? - There are no cross-dependencies - The managment-clusters are lightweight - Are you running a real kubernetes cluster for the hub? - Nope we just use the apiserver