---
title: Building air-gapped control planes for a global pharma leader using crossplane and argo
weight: 8
tags:
 - rejekts
 - crossplane
---

<!-- {{% button href="https://youtu.be/rkteV6Mzjfs" style="warning" icon="video" %}}Watch talk on YouTube{{% /button %}} -->

Joint effort of novo-nordik and upbound.

## Background

- Ymir Platform: Foundational abstraction platform
- Goal: Faster time to market
- Usage in pharma: end-2-end compliance
- Airgap: Use gitopt and prevent human interaction with the control planes

## Setup

- Decision for crossplane was obvious
- Problem: Chicken and egg "we provision clusters via crossplane but crossplane needs a cluster"
- GitOps: Everything as code with automatic tests and argo
- Infra: Azure

### Public AKS

```mermaid
graph LR
    subgraph MC
        ProviderAzure
        ProviderKubernetes
    end
    ProviderAzure-->|Calls APU|AKS
    AKS-->|Provisions|Kubernetescluster
    ProviderKubernetes-->|Deploys service on|Kubernetescluster
```

### Bastion Bootstrap

- Options: Terraform/Opentofu
- Goal: Crossplane all the things
- Solution: Run Crossplane in a github action
  1. Kind Cluster
  2. Install Crossplane
  3. Propagete Credentials
  4. Create Cluster
- Tooling: Uptest - E2E Test automation Framework, can be used for bootstrapping since it creates kind cluster with crossplane

```mermaid
graph LR
    subgraph GitHubRunner
        Kubernetes
        Crossplane
    end
    subgraph Azure
        BastionVM
    end
    Crossplane-->|Create|BastionVM
```

### Next steps

- Problem: How to access bastion
- Solution: Auto-register bastion as github runner
- Create Bastion-Cluster via Uptest

```mermaid
graph LR
    subgraph Azure
        subgraph BastionVM
            GitHubRunner
            Kubernetes
            Crossplane
        end
        subgraph BastionCluster
            Kubernetes
            Argo
            CrossPlane
        end
    end
    Crossplane-->|Create|BastionCluster
```

TODO: Steal image from slides

## Challenges

- Argo sync waves:
    - Problem: Argo does not support eventual consistency
    - Example: Install a ProviderConfig before your Provider and sync fails without retry
    - Order stuff very carefully
- Delivering updates to private clusters
    - Difference between public and private: It's the same package
    - Upgrades/Downgrades: Change the package (Crossplane) and cluster (CRD)
- Testing: 
    - Static: Multiple stages and each stage has it's own bootstrap env that can be set to any branch
    - Ephemeral: Uptest

TODO: Steal images from slides


## Wrap-up

- Cloud native air-gapped ✅
- GitOps ✅
- Crossplane, no terraform ✅
- Extensible, reusable, API-first ✅