3.3 KiB
3.3 KiB
| title | weight |
|---|---|
| Building a large scale multi-cloud multi-region SaaS platform with kubernetes controllers | 8 |
Interchangeable wording in this talk: controller == operator
A talk by elastic.
About elastic
- Elestic cloud as a managed service
- Deployed across AWS/GCP/Azure in over 50 regions
- 600.000+ Containers
Elastic and Kube
- They offer elastic obervability
- They offer the ECK operator for simplified deployments
The baseline
- Goal: A large scale (1M+ containers resilient platform on k8s
- Architecture
- Global Control: The control plane (api) for users with controllers
- Regional Apps: The "shitload" of kubernetes clusters where the actual customer services live
Scalability
- Challenge: How large can our cluster be, how many clusters do we need
- Problem: Only basic guidelines exist for that
- Decision: Horizontaly scale the number of clusters (5ßß-1K nodes each)
- Decision: Disposable clusters
- Throw away without data loss
- Single source of throuth is not cluster etcd but external -> No etcd backups needed
- Everything can be recreated any time
Controllers
{{% notice style="note" %}} I won't copy the explanations of operators/controllers in this notes {{% /notice %}}
- Many different controllers, including (but not limited to)
- cluster controler: Register cluster to controller
- Project controller: Schedule user's project to cluster
- Product controllers (Elasticsearch, Kibana, etc.)
- Ingress/Certmanager
- Sometimes controllers depend on controllers -> potential complexity
- Pro:
- Resilient (Selfhealing)
- Level triggered (desired state vs procedure triggered)
- Simple reasoning when comparing desired state vs state machine
- Official controller runtime lib
- Workque: Automatic Dedup, Retry backoff and so on
Global Controllers
- Basic operation
- Uses project config from Elastic cloud as the desired state
- The actual state is a k9s ressource in another cluster
- Challenge: Where is the source of thruth if the data is not stored in etc
- Solution: External datastore (postgres)
- Challenge: How do we sync the db sources to kubernetes
- Potential solutions: Replace etcd with the external db
- Chosen solution:
- The controllers don't use CRDs for storage, but they expose a webapi
- Reconciliation still now interacts with the external db and go channels (que) instead
- Then the CRs for the operators get created by the global controller
Large scale
- Problem: Reconcile gets triggered for all objects on restart -> Make sure nothing gets missed and is used with the latest controller version
- Idea: Just create more workers for 100K+ Objects
- Problem: CPU go brrr and db gets overloaded
- Problem: If you create an item during restart, suddenly it is at the end of a 100Kü item work-queue
Reconcile
- User-driven events are processed asap
- reconcole of everything should happen, bus with low prio slowly in the background
- Solution: Status: LastReconciledRevision (timestamp) get's compare to revision, if larger -> User change
- Prioritization: Just a custom event handler with the normal queue and a low prio
- Low Prio Queue: Just a queue that adds items to the normal work-queue with a rate limit
flowchart LR
low-->rl(ratelimit)
rl-->wq(work queue)
wq-->controller
high-->wq
Related
- Argo for CI/CD
- Crossplane for cluster autoprovision