> ## Documentation Index
> Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Genesis Operations Runbook

> Day-2 operations for a Genesis Kubernetes deployment: upgrade, rollback, backup and restore, secrets, users, scaling, health, teardown, and incident triage.

Everything after the install. This runbook is written for the person who is paged
at 02:00 and has never read the Genesis source: each procedure states what you run,
what a good result looks like, and what to do when it is not.

It is **cloud-agnostic** — Genesis on Kubernetes is the same StatefulSet, the same
chart and the same data volume on GKE, EKS and AKS. Only the substrate underneath
differs, and every substrate difference is collected in
[Per-cloud specifics](#per-cloud-specifics) at the end rather than scattered through
the text.

<Note>
  **Installing, not operating?** Start with the install runbook for your cloud —
  [GKE](/docs/deployments/GCP/Genesis-On-GCP-GKE-Install),
  [EKS](/docs/deployments/AWS/Genesis-On-AWS),
  [AKS](/docs/deployments/Azure/Genesis-On-Azure) — and come back here once it is up.
</Note>

## Pin the identity before you touch anything

Every procedure below needs these facts, and every one of them is read from the
cluster rather than remembered. Write them at the top of your incident notes.

```bash theme={null}
# 1. Release name and namespace — the first column and the namespace column.
#    NOT the image tag and NOT the chart version, which are the other two facts.
helm list -A | grep -i genesis

# Set these once; the rest of this runbook uses them.
RELEASE=genesis
NS=genesis               # the HELM release namespace, for helm and kubectl -n
WS_NS=genesis-cli        # cliTerminal.namespace — read it back, do not assume

# 2. The INSTANCE namespace, which is the deployment's identity and is NOT always
#    $NS. The chart renders the StatefulSet into `values.namespace` where a release
#    sets one, and only otherwise into the Helm namespace — so the StatefulSet's own
#    live namespace IS the identity, whatever the values did.
INSTANCE_NS=$(kubectl get statefulset -A -l app.kubernetes.io/instance=$RELEASE \
  -o jsonpath='{.items[0].metadata.namespace}')
echo "$INSTANCE_NS"

# 3. Chart version and app version currently deployed — the CHART and
#    APP VERSION columns. Neither is the release name in column one.
helm list -n $NS

# 4. The pod. The StatefulSet is named from the Helm fullname, which is USUALLY the
#    release name — read it back rather than assuming "$RELEASE-0".
kubectl -n $NS get statefulset
#    app.kubernetes.io/instance is the release, so this stays correct on a cluster
#    running more than one Genesis release in one namespace.
POD=$(kubectl -n $NS get pod -l app.kubernetes.io/instance=$RELEASE -o name | head -1)

# 5. The image DIGEST actually running — not the tag you asked for.
kubectl -n $NS get $POD \
  -o jsonpath='{.status.containerStatuses[?(@.name=="genesis")].imageID}{"\n"}'

# 6. The data volume, and that it is Bound.
kubectl -n $NS get pvc
```

The rest of this runbook writes `$RELEASE-0` for brevity, which is right on a
default install. Where yours differs, substitute the pod name from step 3.

<Warning>
  **The release name is not a version.** `helm upgrade` with a *new* release name
  creates a new StatefulSet, which creates a **new, empty** PersistentVolumeClaim —
  the old data stays on an orphaned claim under the old name. This is the single
  most common way a Genesis deployment loses its database. Always upgrade the
  release name `helm list` reports, in the namespace `helm list` reports.
</Warning>

<Warning>
  **Read the workspace namespace back; never assume `genesis-cli`.** It is
  configurable (`cliTerminal.namespace`), and on a cluster running more than one
  Genesis instance the chart's default is somebody *else's* namespace, full of other
  people's attached sessions:

  ```bash theme={null}
  helm get values $RELEASE -n $NS -a -o json | jq -r '.cliTerminal.namespace, .cliTerminal.enabled'
  ```

  Note what that does **not** tell you. `helm get values -a` returns *computed*
  values, so every release installed from a chart that ships `genesis-cli` as its
  default reports `genesis-cli` whether the feature is on or off. The answer names a
  namespace; it does not establish that this release owns it. Ownership is the
  `genesiscomputing.ai/instance-namespace` label on the live namespace — see
  [Teardown without data loss](#teardown-without-data-loss).
</Warning>

## Upgrade

A Genesis upgrade is normally an **image** upgrade: same chart, same release name,
same namespace, same PVC. The pod is replaced; the volume reattaches.

### 1. Pull the new image into your mirror

If your nodes pull from a mirror rather than from the Genesis registry, the mirror
is the first thing to update — and the tag you are about to deploy must exist in it.
`crane` copies registry-to-registry without a local Docker daemon and preserves the
source digest and every platform in the manifest:

```bash theme={null}
SRC=us-central1-docker.pkg.dev/genesis-471800/genesis-dist
DST=<your-registry>/<your-repo>
crane copy $SRC/genesis:<new-tag> $DST/genesis:<new-tag>
crane copy $SRC/genesis-sandbox-runner:<new-tag> $DST/genesis-sandbox-runner:<new-tag>
```

Mirror the **workspace image too**, in the same pass. An operator who upgrades the
backend and forgets the sandbox runner gets a working console in which every
terminal fails to start — see
[Every workspace is `ImagePullBackOff`](#incident-triage).

<Warning>
  **On a laptop, pin the platform.** `docker pull` without `--platform linux/amd64`
  on an Apple-silicon machine mirrors an `arm64` image that your nodes cannot run,
  and the failure surfaces later as a crash-looping pod rather than as a bad copy.
  `crane copy` has no such trap — it copies the whole manifest list.
</Warning>

### 2. Run the upgrade

```bash theme={null}
helm upgrade $RELEASE <chart-ref> --version <chart-version> \
  -n $NS -f my-values.yaml \
  --set image.tag=<new-tag> \
  --wait --timeout 12m
```

<Warning>
  **Do not use `--reuse-values`.** It carries the previous release's values forward
  and ignores the new chart's defaults, so a values key introduced *after* your
  release was installed is simply missing at upgrade time. The chart has been broken
  this way for real: `secrets.existingSecretMintlifyJwtPrivateKey` was added with a
  correct default, which does nothing on a `--reuse-values` upgrade, and every lab
  environment failed to render at once. Always pass your complete values file with
  `-f`. (The chart now guards this case — `validate-reuse-values-upgrade.sh` renders
  with each optional secret map explicitly absent — but `-f` is still the supported
  path.)
</Warning>

### 3. Verify by content, not by "Ready"

A Ready pod proves a process started. It does not prove it is the build you shipped:
`main`, `latest` and any other moving tag resolve to whatever the registry holds at
pull time, and `IfNotPresent` will happily keep a cached older layer under the same
tag. Verify the **digest**, and then verify the **behaviour you upgraded for**.

```bash theme={null}
# a. The digest actually running. Compare it to the digest you copied in step 1.
kubectl -n $NS get pod $RELEASE-0 \
  -o jsonpath='{.status.containerStatuses[?(@.name=="genesis")].imageID}{"\n"}'
# expect: <registry>/genesis@sha256:<64 hex chars>

# b. The registry's digest for the tag you deployed.
crane digest $DST/genesis:<new-tag>
# expect: the same sha256 as (a)

# c. The server is serving, not merely running.
kubectl -n $NS exec $RELEASE-0 -c genesis -- \
  curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8080/api/healthcheck
# expect: 200

# d. The claim never changed identity.
kubectl -n $NS get pvc
# expect: the SAME claim name as before the upgrade, STATUS Bound
```

<Check>
  **Gate — do not declare the upgrade done until all four hold:** (a) equals (b);
  (c) is `200`; (d) names the same Bound claim as before; and the specific behaviour
  you upgraded for is exercised by hand. If (a) and (b) disagree, the node is running
  a cached layer — see
  [The upgrade "worked" but the old behaviour persists](#incident-triage).
</Check>

### What "version-gated" means

Some Genesis fixes change behaviour that has no switch: the code either does the old
thing or the new one. A fix described as **version-gated** is delivered only by
running an image that contains it, and no amount of configuration reproduces it on
an older image.

That is why step 3(a) reads the digest rather than the tag. When a support answer
says "fixed in current builds", the question it is answering is *which image is
running* — and the only answer that settles it is the digest, checked against the
digest of the build the fix landed in.

## Rollback

### Roll the release back

```bash theme={null}
helm history $RELEASE -n $NS            # find the revision you want
helm rollback $RELEASE <revision> -n $NS --wait --timeout 12m
```

`helm rollback` restores the *rendered manifest* of that revision — including the
image reference it carried. That reference is always `repository:tag`: **the chart
takes a tag, not a digest.** So if your values file pins a moving tag (`main`), the
rollback restores the tag and not the build, the node may pull the newest image again,
and you land back where you started with a green rollback to show for it.

The fix is to make the tag itself immutable. Mirror the known-good **digest** to a tag
of your own that nothing will ever move, then deploy that tag:

```bash theme={null}
crane copy $DST/genesis@sha256:<the-known-good-digest> $DST/genesis:known-good-2026-09-10

helm upgrade $RELEASE <chart-ref> --version <chart-version> -n $NS -f my-values.yaml \
  --set image.tag=known-good-2026-09-10
```

Then verify by digest exactly as in [step 3 of an upgrade](#3-verify-by-content-not-by-ready)
— the running `imageID` must equal the digest you copied. The point is the same in
both directions: deploy a build, not a label.

<Warning>
  **Never leave `image.tag` unset.** With no tag the chart falls back to the chart's
  `appVersion`, which is frequently a tag that does not exist in the registry your
  nodes pull from — so the rollback "succeeds" and the pod never starts.
</Warning>

### What survives a rollback, and what does not

| Survives | Does not survive |
| - | - |
| The data volume. The StatefulSet sets `persistentVolumeClaimRetentionPolicy.whenDeleted: Retain`, and a rollback does not delete the claim. | **Schema or data migrations the newer build performed.** Helm rolls back Kubernetes objects; it cannot un-migrate a SQLite database. |
| Workspace volumes and the workspace namespace — they carry `helm.sh/resource-policy: keep`. | Running workspace Pods, if the rollback restarts the backend: sessions are interrupted. The volumes are untouched, so users resume on their next attach. |
| Secrets you supplied through `existingSecret*` references, which live outside the release. | Secret *values* passed with `--set` on the newer revision, if the older revision's rendered manifest carried different ones. |

<Warning>
  **A rollback to a chart revision older than `0.3.70` strips the retention
  annotation from the live objects while leaving the objects in place.** The
  namespace and the ServiceAccounts stay, but they are no longer protected: the next
  operation that removes them from a rendered manifest will delete them, and a
  namespace delete cascades to every PVC inside it. Re-check after **any** rollback:

  ```bash theme={null}
  kubectl get namespace $NS \
    -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}{"\n"}'
  kubectl get namespace $WS_NS \
    -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}{"\n"}'
  # expect: keep      (an empty line is a FAIL — re-upgrade forward before proceeding)
  ```
</Warning>

<Check>
  **Gate — rollback complete when:** `helm history` shows the target revision
  `deployed`; the running digest is the one you intended; `/api/healthcheck` returns
  `200`; the PVC is the same Bound claim; and both namespaces still report
  `resource-policy: keep`.
</Check>

## Backup and restore

### What is on the volume

One PersistentVolumeClaim per deployment (`genesis-data-<statefulset-name>-0` by
default), mounted into the backend at four subpaths:

| Path in the container | What it holds |
| - | - |
| `/app/.genesis/db` | **The SQLite application database** — configuration, users and roles, bots, threads and conversations. The thing you cannot rebuild. |
| `/app/bot_storage` | Files bots have written or been given. |
| `/app/bot_git` | Bot git checkouts. |
| `/app/tmp` | Scratch. Safe to exclude from a backup. |

Workspace home volumes are **separate claims in the workspace namespace**, two per
user. They hold each person's saved Claude Code conversations and working tree. They
are not on the backend volume and are not covered by a backend snapshot.

### Snapshot the volume

The reliable mechanism is a **volume snapshot taken by the storage layer**, not a
`kubectl cp`. Find the underlying disk, then snapshot it with your cloud's mechanism
(commands per cloud are in [Per-cloud specifics](#per-cloud-specifics)):

```bash theme={null}
# The claim. Its name is <volumeClaimTemplate>-<statefulset>-0, and the StatefulSet is
# named from the Helm fullname — which is USUALLY but not always the release name, so
# read the name back rather than composing it.
kubectl -n $NS get pvc
PVC=$(kubectl -n $NS get pvc -o name \
  -l app.kubernetes.io/instance=$RELEASE 2>/dev/null | head -1)
# StatefulSet claims do not always inherit the release labels. If the line above came
# back empty, take the claim the live Pod is actually using — which is the definition:
PVC=${PVC:-persistentvolumeclaim/$(kubectl -n $NS get $POD \
  -o jsonpath='{.spec.volumes[?(@.name=="genesis-data")].persistentVolumeClaim.claimName}')}
echo "$PVC"

# The PersistentVolume behind it, then the cloud disk handle behind that
PV=$(kubectl -n $NS get $PVC -o jsonpath='{.spec.volumeName}')
kubectl get pv $PV -o jsonpath='{.spec.csi.volumeHandle}{"\n"}'
```

That last value is the disk your cloud's snapshot command takes as input. Snapshot
**that** handle — not a claim name you composed, which on a deployment using
`fullnameOverride` names a claim that does not exist, or worse, one that does and
belongs to something else.

<Warning>
  **The SQLite WAL must be captured with the database — checkpoint first.** Genesis
  runs SQLite in write-ahead-logging mode: recent committed transactions live in
  `genesis.db-wal` and `genesis.db-shm` beside `genesis.db` until a checkpoint folds
  them in, and a deployment holds **several** databases across `/app/.genesis/db` and
  `/app/bot_storage`. A raw snapshot taken mid-transaction, or a file copy that takes
  `genesis.db` alone, restores the database as of the **last checkpoint** rather than
  as of the snapshot — and it restores *cleanly*, with no error, which is what makes
  this dangerous.

  The image ships a pre-backup hook that folds every WAL in, in place, in seconds,
  without stopping the application. **Run it immediately before every snapshot:**

  ```bash theme={null}
  kubectl -n $NS exec $RELEASE-0 -c genesis -- /app/scripts/sqlite-checkpoint-hook.sh
  # expect a "checkpointed: ..." line per database, then:
  #   checkpoint complete: N succeeded, 0 failed
  ```

  It **exits non-zero if any database failed to checkpoint**, so wire it as a gating
  step rather than a best-effort one. For Velero, attach it as a pod annotation so
  every scheduled backup checkpoints first:

  ```yaml theme={null}
  podAnnotations:
    pre.hook.backup.velero.io/command: '["/app/scripts/sqlite-checkpoint-hook.sh"]'
    pre.hook.backup.velero.io/timeout: "60s"
    pre.hook.backup.velero.io/on-error: Fail
  ```
</Warning>

<Warning>
  **The master key is not on the volume.** Restoring the data volume without the
  matching `GENESIS_MASTER_KEY` gives you a database whose encrypted fields — every
  stored credential — cannot be read. A volume snapshot is therefore **half** a
  backup. The other half is the key, backed up separately, in a place that survives
  the loss of this cluster. See [Secrets](#secrets).
</Warning>

For a file-level copy — useful as a second, portable line of defence, not as a
replacement for volume snapshots — the chart ships one:

```bash theme={null}
# Streams /app/.genesis/db, bot_storage, bot_git (and tmp unless excluded)
# out of the running pod into a local .tgz.
GENESIS_LOCAL_NAMESPACE=$NS GENESIS_LOCAL_POD=$RELEASE-0 \
  bash local-k8s/backup-genesis.sh <kubectl-context> pre-upgrade
```

It refuses rather than writing an empty archive: it skips with exit 2 if the pod is
not `Running` or `/app/.genesis/db` is unreadable, and fails with exit 1 on an empty
result. Treat any exit other than 0 as "no backup was taken".

### The restore drill

**An untested backup is not a backup.** Run this on a schedule — quarterly at the
least, and once before the deployment carries anything that matters. Restore to a
*separate* release so the drill can never damage the live one.

<Steps>
  <Step title="Checkpoint, snapshot, and record the identity">
    Run the checkpoint hook, then snapshot the live volume. Write down the snapshot
    id, the claim name, the chart version, the image digest that produced it, **and
    which master key it was encrypted under**. A restore into a different build, or
    under a different key, is a different test.
  </Step>

  <Step title="Create a claim from the snapshot">
    Provision a new PVC from the snapshot using your cloud's restore mechanism, in a
    scratch namespace (`genesis-drill`). Confirm `STATUS Bound` before going on.
  </Step>

  <Step title="Install a second release onto that claim, with the same master key">
    Name the scratch release, and use that name for the rest of the drill:

    ```bash theme={null}
    DRILL=genesis-drill-restore
    ```

    Install the **same chart version and the same image digest** into the scratch
    namespace, pointed at the restored claim, and supply the **same
    `GENESIS_MASTER_KEY`** the snapshot was taken under. Supplying a different key is
    how a drill "passes" while proving nothing: the server starts, the threads are
    there, and every stored credential is unreadable.

    <Warning>
      **The restored volume carries the live deployment's configuration, including
      which workspace namespace to use — and `cliTerminal.enabled: false` pins
      nothing.** Leaving the chart flag off is *not* enough: the flag only decides
      whether the chart renders a platform, and the restored APP\_CONFIG can still say
      `cli_terminal.enabled: true` with `k8s_namespace: genesis-cli`. The drill's
      backend would then create and, worse, **reap** workspace Pods in the **production**
      workspace namespace, where a Pod and both of its volumes are named from a digest
      of the user id alone — so the drill and the live release collide on real people's
      home volumes.

      Force-lock both Pod-creating features off, by name. These are the double-underscore
      overrides, which outrank anything stored in the restored database:

      ```bash theme={null}
      helm upgrade --install $DRILL <chart-ref> --version <chart-version> \
        -n genesis-drill --create-namespace -f drill-values.yaml \
        --set config.extraEnv.GENESIS__CLI_TERMINAL__ENABLED=false \
        --set config.extraEnv.GENESIS__SANDBOX_RUNNER__ENABLED=false \
        --set secrets.existingSecretGenesisMasterKey.secretName=genesis-master-key \
        --set secrets.existingSecretGenesisMasterKey.secretKey=GENESIS_MASTER_KEY \
        --wait --timeout 12m
      ```

      **Never pass the key itself with `--set`.** It would land in your shell history,
      in the process arguments of a running command, and in the Helm release record —
      three places your escrow policy does not cover. Reference a Secret instead.
      Secrets are namespaced, so copy the live one into the scratch namespace first,
      stripping the fields that belong to the original object:

      ```bash theme={null}
      kubectl -n $NS get secret genesis-master-key -o json \
        | jq 'del(.metadata.namespace, .metadata.uid, .metadata.resourceVersion,
                  .metadata.creationTimestamp, .metadata.ownerReferences,
                  .metadata.managedFields)' \
        | kubectl -n genesis-drill create -f -
      ```

      Where the deployment resolves its key from a cloud secret manager instead, give
      the drill the same `config.extraEnv.GENESIS_MASTER_KEY_REF` and leave
      `secrets.genesisMasterKey` empty — the drill's service account needs read on that
      secret. Delete the copied Secret with the scratch namespace in Step 5.

      Then **prove the lock took**, before anyone touches the drill — a restored
      database that disagrees is exactly the case this guards against:

      ```bash theme={null}
      kubectl -n genesis-drill exec $DRILL-0 -c genesis -- \
        printenv GENESIS__CLI_TERMINAL__ENABLED GENESIS__SANDBOX_RUNNER__ENABLED
      # expect: false / false

      # And that the PRODUCTION workspace namespace is untouched. Take this BEFORE
      # installing the drill, and diff it afterwards. Names AND UIDs, not counts: a
      # count is equal again once a deleted volume has been replaced by a new one,
      # which is the loss this baseline exists to detect.
      kubectl -n $WS_NS get pod,pvc \
        -o custom-columns=KIND:.kind,NAME:.metadata.name,UID:.metadata.uid \
        --no-headers | sort > /tmp/ws-baseline.txt
      ```
    </Warning>

    **To exercise the terminal in the drill instead**, give it a workspace namespace of
    its own (`cliTerminal.enabled=true` with `cliTerminal.namespace=genesis-drill-cli`),
    never the production one — and own that namespace's whole lifecycle in Step 5, not
    just the release's. Two releases must never share one workspace namespace.
  </Step>

  <Step title="Prove the data came back, by content">
    ```bash theme={null}
    kubectl -n genesis-drill exec $DRILL-0 -c genesis -- \
      curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8080/api/healthcheck
    # expect 200
    ```

    Then sign in and check **three** things by eye, because a healthy process proves
    none of them: a named user still exists with their roles; a known chat thread
    still opens and shows its messages; a configured LLM provider is still configured.
    Record what you checked and the date.
  </Step>

  <Step title="Destroy the drill — all of it">
    A drill that leaves objects behind has created a second lifecycle beside the real
    one. Retained objects never go away on their own, so removal is explicit. Follow
    [Teardown without data loss](#teardown-without-data-loss) against the **drill**
    identity, in that order:

    ```bash theme={null}
    # Only if the drill ran a terminal-enabled release with its OWN workspace namespace:
    cluster/scripts/drain-workspaces.sh $DRILL -n genesis-drill

    helm uninstall $DRILL -n genesis-drill
    cluster/scripts/cleanup-retained-rbac.sh genesis-drill   # revokes the retained bindings

    kubectl -n genesis-drill delete pvc --all
    kubectl delete namespace genesis-drill
    kubectl delete namespace genesis-drill-cli   # only if the drill created one
    ```

    Then confirm nothing of the drill's remains. Deletion was *requested* above;
    these check it happened — and cluster-scoped bindings in particular survive a
    namespace delete:

    ```bash theme={null}
    helm list -A | grep "$DRILL"                       # expect: no output
    kubectl get namespace genesis-drill genesis-drill-cli   # expect: NotFound for both
    kubectl get clusterrolebinding -l genesiscomputing.ai/instance-namespace=genesis-drill
    kubectl get rolebinding -A -l genesiscomputing.ai/instance-namespace=genesis-drill
    kubectl get pvc -A | grep genesis-drill            # expect: no output
    ```

    A namespace stuck `Terminating` is not deleted — find what is holding it
    (`kubectl get namespace genesis-drill -o jsonpath='{.status.conditions}'`) rather
    than recording the drill as clean.

    Finally, prove the production workspace namespace came through untouched:

    ```bash theme={null}
    kubectl -n $WS_NS get pod,pvc \
      -o custom-columns=KIND:.kind,NAME:.metadata.name,UID:.metadata.uid \
      --no-headers | sort > /tmp/ws-after.txt
    diff /tmp/ws-baseline.txt /tmp/ws-after.txt
    ```

    Lines added or removed for **Pods** are normal — workspaces are created and reaped
    while you work. Any change to a **PVC** line is not: a missing or re-UID'd claim is
    someone's home volume, and it is an incident, not a drill result.
  </Step>
</Steps>

<Check>
  **Gate — the backup is proven only when all five hold:**

  1. A claim provisioned from the snapshot reached `Bound`
  2. The drill release returned `200`, with the **same** master key
  3. A human confirmed a user, a thread and a provider configuration by looking at them
  4. **`diff /tmp/ws-baseline.txt /tmp/ws-after.txt` shows no PVC line changed** in the
     production workspace namespace — same names, same UIDs. The drill never touched
     live users' volumes. (Pod lines may differ; workspaces come and go.)
  5. **Nothing drill-owned remains, checked rather than assumed**: `helm list` does not
     show the drill release, both scratch namespaces are `NotFound` (not `Terminating`),
     and no Role/ClusterRoleBinding carries the drill's instance label

  Anything less has tested the storage layer, not the backup — and 4 and 5 are what
  make it safe to run the drill on a schedule rather than once.
</Check>

## Secrets

### The master key

`GENESIS_MASTER_KEY` encrypts every credential Genesis stores. It is required — the
chart refuses to render without it — and it is **not recoverable**: lose it and every
stored connection credential in the database becomes undecryptable ciphertext, which
is a rebuild, not a recovery.

Three supported ways to supply it, in increasing order of operational maturity:

| How | Values | Use when |
| - | - | - |
| Inline at install | `--set secrets.genesisMasterKey=<hex>` | A trial. The value is in your shell history and in the Helm release record. |
| An existing Kubernetes Secret | `secrets.existingSecretGenesisMasterKey.secretName` / `.secretKey` | The normal production path. The key is created and rotated outside the chart. |
| Resolved from a cloud secret manager at startup | `config.extraEnv.GENESIS_MASTER_KEY_REF` set to an `aws_*` / `gcp_*` reference, `secrets.genesisMasterKey` left empty | The pod holds no key at rest; it fetches it with its cloud identity (IRSA / Workload Identity). |

Generate one with `python3 -c "import secrets; print(secrets.token_hex(32))"`.

<Warning>
  **Name an escrow owner before go-live, in writing.** The key must exist in a
  corporate secret store that survives the departure of whoever installed Genesis,
  with a named owner and a named deputy. This is an operator responsibility and there
  is no product feature that substitutes for it: Genesis cannot re-derive the key, and
  the deployment's credentials are only as recoverable as the escrow record.
</Warning>

<Warning>
  **There is no master-key rotation procedure, and swapping the key is not a values
  change.** Genesis documents none, and the reason is structural: every credential in
  the database is ciphertext under the current key, so changing the key makes all of
  it unreadable — the server starts, and every connection fails. Treat the key as
  **fixed for the life of the deployment**, and protect it by escrow rather than by
  rotation.

  If a rotation is forced on you by policy or by a suspected compromise, it is a
  migration, not an upgrade: plan a maintenance window, take and **verify** a backup
  first, expect to re-enter every stored credential under the new key by hand, and
  confirm each connection afterwards. Do not begin one on the strength of this
  paragraph — raise it with Genesis first.

  The corollary for backups: **rotate the key and your existing backups stop being
  restorable.** If the key ever changes, take a fresh verified backup the same day.
</Warning>

### LLM and database credentials

| Credential | Supplied as | Rotation |
| - | - | - |
| Databricks LLM token | `secrets.databricksApiKey`, or `secrets.existingSecretDatabricksApiKey`. The chart sets both `DATABRICKS_API_KEY` and `DATABRICKS_TOKEN` from the one value. | Update the Secret, then restart the pod so the container picks up the new value. |
| Databricks SQL warehouse PAT (a *separate* credential from the LLM one) | `secrets.databricksDbPat` / `existingSecretDatabricksDbPat` | Same. |
| OpenAI / Azure OpenAI key | `secrets.openaiApiKey` / `existingSecretOpenaiApiKey` | Same. |

<Note>
  **Service principal or PAT?** A personal access token is issued to a person and dies
  with their account — the classic Genesis outage is an LLM that stops answering
  because the engineer who installed it left the company. For anything beyond a trial,
  authenticate as a **service principal / machine identity** owned by a team, and
  record its expiry in the same calendar that holds your certificate renewals. Where
  the platform supports it, prefer no stored credential at all: the chart's
  `genesisConfig` path accepts `auth_mode: "iam_role"` for Bedrock, which authenticates
  through the pod's cloud identity and stores nothing.
</Note>

Two further secrets the chart accepts, both with a documented rotation:

* **The Mintlify documentation signing key** (Ed25519, PKCS#8) is accepted **only** as
  an existing Kubernetes Secret — never in `genesisConfig.content`, `config.extraEnv`,
  a ConfigMap, or a plaintext Helm value. Rotating it is update-the-Secret **and**
  roll the workload, because the pod's environment is what carries it. Give it a named
  security or platform owner, the same way the master key gets one.
* **Custom CA certificates** are fetched once at pod startup, so rotating one needs a
  StatefulSet restart. The same is true of any value resolved from a cloud secret
  manager: after you rotate it there, restart so the pod re-resolves it.

An `existingSecret*` reference is the rotation-friendly shape for all of these,
because the Secret is updated without a `helm upgrade` and the credential never
appears in the release record. The pod reads these into environment variables at
start, so a rotation takes effect on the next pod restart:

```bash theme={null}
kubectl -n $NS rollout restart statefulset/$RELEASE
kubectl -n $NS rollout status statefulset/$RELEASE --timeout=12m
```

<Check>
  **Gate — a rotation is complete when** the pod has restarted onto the new Secret,
  `/api/healthcheck` returns `200`, a chat message gets a reply, **and** the embedding
  endpoint is verified separately (see
  [Health and verification](#health-and-verification)). Chat and embeddings can use
  different credentials and different endpoints; proving one says nothing about the
  other.
</Check>

## Configuration precedence

Genesis reads one setting from up to five places. When a change "does not take
effect", it is almost always because a higher tier is holding the old value. Highest
wins, and the **Config Sources** page names the tier that actually won for any field —
which is the fastest way to end an argument about it:

| Tier | Where it comes from | Config Sources calls it | Changed by |
| - | - | - | - |
| 1 | `GENESIS__<SECTION>__<FIELD>` environment variable, usually written by the chart | `env_override` | `helm upgrade` |
| 2 | A stored APP\_CONFIG value | `database` | An admin in **Advanced Settings**, the config API, or a `genesisConfig.content` apply |
| 3 | A stored APP\_CONFIG value that is itself an environment reference (`env_…`) | `db_env_var` | The same, plus whatever the referenced variable holds |
| 4 | A plain environment variable bound to the field as its default (`EMBEDDING_MODEL`, `GENESIS_CLI_TERMINAL_ENABLED`, …), typically from `config.extraEnv` | `default_env_var` | `helm upgrade` |
| 5 | The schema default in the image | `default_plain` | The image |

Read the two environment-variable spellings carefully, because they look alike and
behave oppositely:

* **`GENESIS__CLI_TERMINAL__ENABLED`** — prefix `GENESIS__`, then the config path
  uppercased with each `.` replaced by `__`. This is the **force-lock**: it is applied
  when the configuration snapshot is built, so it outranks everything below it. It is
  never persisted to the database.
* **`GENESIS_CLI_TERMINAL_ENABLED`** — single underscores. This is only a **default**.
  For a field declared `env_leaf`, the config-to-env bridge *overwrites this variable*
  with the admin's stored value, live and with no restart — so on its own it does not
  hold against the panel.

<Warning>
  **A force-lock does more than pin the value — through the panel it refuses the
  write.** An admin editing a `GENESIS__`-locked field in Advanced Settings gets an
  error, not a stored-but-overridden value:

  ```text theme={null}
  Cannot update '<path>': field is locked by environment variable GENESIS__...
  ```

  A **reset** on a locked field is refused the same way, and nothing is deleted:

  ```text theme={null}
  Cannot reset fields locked by environment variables: ...
  ```

  A batch update containing one locked path is refused in its entirety. The lock test
  is the *presence* of the variable, whatever its value.

  Two consequences worth knowing before you debug one: on a chart-managed install with
  `cliTerminal.enabled=true`, the three pinned rows below **cannot be changed or reset
  from the panel at all** — the failure is a configuration error in the UI, not a 403
  at attach time. And the other write paths do **not** perform this check: a
  `genesisConfig.content` apply, the config CLI, and the AI Providers / Embeddings
  pages — which own `llm.active_provider`, `embedding.active_provider` and
  `embedding.active_model` — store the row, and the override simply keeps winning. So
  a stored value can exist under a lock even though the panel would have refused to
  create it, and for those three fields a save under a pin is accepted and takes
  effect only once the pin is removed.
</Warning>

<Warning>
  **Editing `genesisConfig.content` can wipe Advanced Settings.** The file is not a
  precedence tier; it is a loader that writes into tier 2 at startup, and it behaves
  in two very different ways depending on whether its content changed:

  * **Content unchanged** — it fills in only fields that have no stored value, and
    never overwrites an admin's panel setting.
  * **Content changed** (its hash differs, or the file path changed) — it performs a
    **full replacement**, overwriting and deleting stored rows.

  So editing one line of that file and running `helm upgrade` discards configuration
  your admins set in the UI. This is the most destructive precedence interaction in
  the system. Decide up front whether the values file or the panel is the source of
  truth for a given field, and if it is the file, tell your admins the panel is
  advisory for it.
</Warning>

### Which fields the chart pins

Setting `cliTerminal.enabled=true` writes, as container `env`, **both** spellings of
three fields:

| Field | Pinned to | Why it must agree with the cluster |
| - | - | - |
| `cli_terminal.enabled` | `true` | An admin who once switched the feature off in the panel would otherwise keep it off through the chart flag — every attach refused with *"CLI terminal access is disabled on this deployment"* while the cluster looks fully prepared. |
| `cli_terminal.runtime` | `kubernetes` | The default backend shells out to a Docker daemon that does not exist in a Pod. |
| `cli_terminal.k8s_namespace` | `cliTerminal.namespace` | The backend must create Pods where the chart put the RBAC. Disagreeing silently means a 403 on every attach — or worse, live workspaces in a namespace the drain scripts do not look in. |

`cliTerminal.maxWorkspaces` is deliberately **not** pinned: lowering the cap in the
panel is safe and a lock would block it to prevent the unsafe direction. See
[Scaling](#scaling) for the consequence of raising it there.

Two asymmetries that surprise people:

* **`cliTerminal.enabled=false` pins nothing.** The chart emits no lock at all when
  the feature is off, because deployments that provision the workspace namespace and
  RBAC outside the chart enable the feature through APP\_CONFIG and must keep working.
  So `false` means "the chart provides no platform", not "the feature is off
  everywhere".
* **Off ECR, the workspace image is yours and is not locked.** With `ecr.enabled`, the
  chart owns the image and pins it, because the pull secret it places authenticates to
  that registry and nowhere else. Off ECR, repointing the workspace image stays an
  admin's to make.

### What applies live and what needs a restart

| Change | Effect |
| - | - |
| `cli_terminal` sizing — image, CPU, memory, volume sizes | Applies to workspace Pods created **after** the change. A user with a running workspace keeps it until it is reaped and recreated, so nothing interrupts an agent mid-task. |
| `cli_terminal.runtime`, `cli_terminal.k8s_namespace` | **Fixed for the life of the server process.** A live change is logged and ignored, because workspaces already created live in the old namespace and nothing else could find them to reap. Change these from the values file, and retire the existing workspaces first. |
| Anything supplied as a plain environment variable (`config.extraEnv`, secrets) | Read at container start. Needs `rollout restart`. |

### Reset to defaults

"Set default" on a configuration row deletes the stored value and falls back to the
tier below. On a current build, for an **unlocked** `cli_terminal.*` or
`sandbox_runner.*` row, that restores **the deployment's own pre-panel value** — the
value the environment held before an admin ever touched the panel — rather than the
schema default. The response says so explicitly when it happens, naming the paths that
landed on an environment default rather than a schema one.

For a **locked** row the reset is refused outright and nothing is deleted, as above.
On a chart-managed install that covers `cli_terminal.enabled`, `.runtime` and
`.k8s_namespace`: they cannot be reset from the panel, by design.

<Warning>
  **On an older image, "reset to defaults" reads as off while the feature stays
  live.** Before the fix, using **Set default** on a `cli_terminal.*` or
  `sandbox_runner.*` row deleted the stored value and reported success — the panel
  then showed the feature off — while the environment variable the runtime actually
  reads kept the value the panel had bridged into it. The terminal stayed reachable
  until the pod restarted: a live shell behind an "off" switch.

  This is [version-gated](#what-version-gated-means). If your image predates the fix,
  either upgrade it, or verify the real state after a reset — attempt an attach, or
  restart the pod — rather than trusting the panel.
</Warning>

## Users: onboarding and offboarding

### Onboarding

1. The user signs in through your identity provider. Genesis maps their IdP groups to
   Genesis roles on **each login**, from the `groupRoleMapping` file supplied to the
   chart, so group membership is the thing to change — not anything inside Genesis.
2. Terminal access is **permission-gated**, separately from the feature being enabled:
   a user needs the `use_cli_terminal` permission. Without it the WebSocket closes with
   *"Access denied"*, which is a different failure from the feature being off.
3. The admin **CLI Workspaces** tab at `/pulse/cli-workspaces` lists running workspace
   Pods with restart and cancel, and is gated by the `view_sandboxes` permission.

The first terminal a user opens creates their workspace Pod and pulls the \~700 MB
workspace image, which can take up to a minute. Subsequent attaches are immediate.

### Offboarding: the step everyone misses

Revoking the IdP group removes access. It does **not** reclaim disk.

<Warning>
  **Workspace volumes are never auto-reclaimed, and lifetime users — not concurrent
  users — is what grows your disk.** Each person who has *ever* attached holds two
  durable PVCs (10 GiB each by default) for their home and work directories. Reaping
  an idle workspace returns CPU and memory; it returns no disk, because the volume is
  what makes a user's saved conversations survive the reap. A deployment with 8 people
  using it and 60 people who have tried it is carrying 120 volumes.

  Nothing reclaims them automatically and that is deliberate: an "oldest unused" policy
  would delete the work of whoever is on holiday. Retirement is an operator action tied
  to a person leaving.
</Warning>

Both of a user's claims carry the ownership metadata needed to find them —
`genesis.computing/owner-id` as a readable annotation, and
`genesis.computing/workspace-owner` as the selectable digest, because a label value
cannot hold an email address:

```bash theme={null}
# who owns what
kubectl -n $WS_NS get pvc \
  -o custom-columns=NAME:.metadata.name,OWNER:'.metadata.annotations.genesis\.computing/owner-id'

# find one person's digest, then retire both of their volumes -- AFTER their Pod is gone
kubectl -n $WS_NS get pvc -o json \
  | jq -r '.items[] | select(.metadata.annotations["genesis.computing/owner-id"]=="<email>")
           | .metadata.labels["genesis.computing/workspace-owner"]' | sort -u
kubectl -n $WS_NS delete pvc -l genesis.computing/workspace-owner=<digest>
```

Cancel the user's workspace from the admin **CLI Workspaces** tab (or delete the Pod)
first: deleting a claim while its Pod still exists leaves the claim `Terminating`
until the Pod goes.

<Check>
  **Gate — offboarding is complete when:** the user's IdP group membership is removed;
  `kubectl -n $WS_NS get pod` shows no workspace for them; and
  `kubectl -n $WS_NS get pvc` returns **zero** claims annotated with their address.
  This is destructive and unrecoverable — it is that person's saved conversations — so
  confirm the address in the annotation before deleting, not the digest.
</Check>

## Scaling

### The capacity formula

The backend is fixed; workspaces scale with people.

```text theme={null}
vCPU  = 4  + 0.5 × Nconcurrent
RAM   = 16 + 4   × Nconcurrent   (Gi reserved; each may grow to 8Gi)
disk  = 50 + 20  × Nlifetime     (Gi; 2 × 10Gi PVCs per person who has EVER attached)
```

`Nconcurrent` is bounded by `cliTerminal.maxWorkspaces` (default 25) and
`Nlifetime` by `cliTerminal.maxLifetimeUsers` (default 50). At those defaults the
feature asks for **12.5 CPU and 100 GiB reserved**, bursting to 200 GiB, on top of the
backend's own 4 CPU / 16 GiB — and up to 1000 GiB of durable disk at the claim
ceiling. A count quota reserves nothing, so the disk ceiling costs only what people
actually claim.

Three facts that make real deployments size differently than expected:

* **A workspace outlives the connection.** Disconnecting does not delete the Pod; the
  idle reaper does, after `cli_terminal.idle_reap_seconds` (default 1 hour). Size for
  the peak number of workspaces that **exist**, not for the number attached right now.
  Shortening the reap interval returns capacity sooner and never costs a user work.
* **One user is one Pod, however many agents they run.** Up to
  `cli_terminal.max_sessions_per_workspace` (default 8) sessions share one container's
  CPU and memory. More sessions per user is cheaper than more users — and more likely
  to hit the memory ceiling, which OOM-kills the container and takes every session that
  user has open.
* **Reserved is not used.** A cluster sitting at 5% observed CPU can correctly refuse
  to schedule a workspace. Compare **requests** against **allocatable minus committed
  requests**, never against `kubectl top`.

### Raising the cap

```bash theme={null}
helm upgrade $RELEASE <chart-ref> --version <chart-version> -n $NS -f my-values.yaml \
  --set cliTerminal.maxWorkspaces=40 \
  --set cliTerminal.maxLifetimeUsers=100      # must exceed maxWorkspaces or the render fails
```

<Warning>
  **Raise it in the values file, never in Advanced Settings.** `maxWorkspaces` renders
  the namespace's `count/pods` ResourceQuota **and** is written to the backend as
  `GENESIS_CLI_TERMINAL_MAX_WORKSPACES`. That env var is only a default, and the field
  is `env_leaf`, so an admin raising the cap in the panel outranks it immediately while
  the quota stays where it was rendered. The application then admits a workspace the API
  server refuses with a bare 403 — **and the quota is the authority**. Lowering it in
  the panel is harmless; raising it is what bites.
</Warning>

Then confirm the cluster can actually supply the new number, which is a different
question from whether the quota permits it:

```bash theme={null}
kubectl -n $WS_NS get resourcequota -o yaml | grep -A6 'hard:'
kubectl describe nodes | grep -A6 'Allocated resources'
```

### Node auto-upgrade will interrupt sessions

A managed node pool upgrade drains and replaces nodes. Workspace Pods are evicted
with them: **running agent sessions are interrupted**, and every volume survives, so
users resume on their next attach on a new Pod.

Treat node auto-upgrade as a **maintenance window**, not as a transparent operation.
Set the maintenance window to your users' off-hours, announce it, and prefer draining
workspaces yourself beforehand so the interruption happens when you chose it:

```bash theme={null}
cluster/scripts/drain-workspaces.sh $RELEASE      # stops agents, keeps every volume
```

<Note>
  **A workspace Pod is pinned to its volume's zone.** The home PVC is `ReadWriteOnce`
  and, on zonal storage, bound to one zone — so a regional cluster with plenty of free
  capacity **in other zones** leaves a workspace `Pending` indefinitely with
  `didn't match PersistentVolume's node affinity`. Shrinking the workspace size does not
  fix it; the constraint is the zone. Capacity has to exist where that user's volume
  already lives.
</Note>

## Health and verification

### Endpoints

| Endpoint | Returns | Token needed | Use it for |
| - | - | - | - |
| `/api/health/live` | `200` whenever the process is alive, regardless of readiness | No | Liveness probe. Never use it as "Genesis is working". |
| `/api/health` | `503` with progress detail while booting, `200` once ready. Carries `status`, `ready`, `degraded`, `degraded_reason`, `llm_configured`, `llm_validation_error`, `uptime_seconds` and the version | No | Readiness and startup probes, and the **first thing to read in an incident** — `degraded_reason` and `llm_validation_error` frequently answer the question outright. |
| `/api/health/startup` | Granular boot progress — phase, percentage, current step. Always `200` | No | "It has been eight minutes, is it stuck or still working?" |
| `/api/healthcheck` | `200` when ready, `503` while starting | No | The simple external check. |
| `/api/health/slots` | Agent slot usage, PII-free | **Yes** | Diagnosing "Genesis is slow / not answering" — whether slots are saturated. |
| `/api/health/detailed` | System metrics and error tracking | **Yes**, plus `ACCESS_CONFIG_PANEL` | Deeper triage. There is no redacted variant; without the permission it is `403`. |

<Note>
  **First boot legitimately takes up to 10 minutes.** The startup probe allows
  `failureThreshold: 60` × `periodSeconds: 10`, because the first boot builds the
  database and warms sessions. `/api/health/startup` tells you whether that time is
  being used or wasted — check it before restarting a pod that is simply still working.
</Note>

The token-free endpoints are matched **exactly** — no prefix matching, and `GET`/`HEAD`
only — so `/api/health/detailed` authenticates even though `/api/health` does not. That
matters behind an `auth_request`-mode ingress: an ingress controller's auth annotation
applies to the whole Ingress, so an external probe gets a login redirect rather than a
health answer. The chart's `ingress.authExemptHealth` block renders a second Ingress
carrying only those exact probe paths, listed with and without a trailing slash, so
health checks read server liveness instead of the proxy's login page.

### Chat working does not mean embeddings working

This is the failure that costs the most trust, because nothing looks broken. Chat and
embeddings are separate routes, frequently separate endpoints, and sometimes separate
credentials. When embeddings fail, Genesis keeps answering: semantic search silently
falls back to graph-only lexical matching, so queries still return results — worse
ones — and the data-context tool reports `index_status: ready_with_warnings` rather
than an error.

<Warning>
  **`ready_with_warnings` is not an embedding verdict.** An install whose embeddings
  are completely dead still answers every query as `ready_with_warnings` from the
  lexical fallback. It tells you a query had warnings; it does not tell you semantic
  search is working. Use the two surfaces below instead.
</Warning>

Both are **present-tense**: they report whether embedding works *now*, not whether it
ever did.

**1. The AI Providers page**, at `/config/ai-providers/` (embeddings tab
`?tab=embeddings`), shows an embedding status chip with its reason. It is fed by
`GET /api/config/ai-providers/embedding-health`, deliberately as a separate request
from the page overview, so a slow or absent indexing subprocess delays the status alone
rather than the whole page. The chip **does not poll** — it is a one-shot read with a
30-second cache, so **reload the page** to re-check rather than watching it.

**2. `GET_INDEX_STATS`**, which is an *action on the lineage tool*, not a REST
endpoint. Over HTTP:

```bash theme={null}
curl -sS -X POST https://<host>/api/tool_call \
  -H "Authorization: Bearer <token>" -H 'Content-Type: application/json' \
  -d '{"func":"lineage_tools_lineage_action","func_args":{"action":"GET_INDEX_STATS"}}'
```

It returns `embedding_provider`, `embedding_model`, `embedding_status` and
`embedding_status_reason` from one snapshot, so the verdict always describes the
identity printed beside it — alongside index counts. It costs nothing: it reads
recorded outcomes and makes **no** provider call.

All four embedding fields are present on every successful response, including on an
install with no embedding client at all — that case reports `unconfigured`. Alert on the
**value**, and treat an absent `embedding_status` as a fault in the check itself rather
than as a pass: a response without one came from a Twin too old to carry the field, and
says nothing about whether embedding works.

| Value | Means | What to do |
| - | - | - |
| `ok` | This identity has embedded successfully and there is not yet conclusive evidence it has stopped. | Read the reason anyway: `ok` can be reporting failures since the last success, and it names the count and the last error. |
| `unverified` | Nothing attempted yet, or too few failures to conclude anything. | Not an alarm. Force a real call with the connection test below. |
| `systemic_failure` | This identity cannot currently embed — semantic search is dead. | Page-worthy. See [the catalogue entry](#incident-triage). |
| `unconfigured` | **There is no embedding client**, so there is no identity to judge and `embedding_provider` / `embedding_model` are empty. Semantic search cannot work: every result is graph-only lexical matching. | Configure an embedding provider and model. Expected on a fresh install, an alarm anywhere it is not — and never to be read as a milder `unverified`, which means something *is* configured but unproven. |
| `unavailable` *(the REST route only)* | **No verdict was obtained** — the indexing subprocess is unreachable or disabled, or the index is not initialised. | An explicit unknown, neither green nor an alarm. Distinct from `unconfigured`: that one is an answer, this one is the absence of one. A surface that reads as working because the check failed is the same defect as one that reads as working while broken. |

The reason is shown for every value, `ok` included.

**3. Force a real embedding call.** The two surfaces above read recorded evidence; when
the verdict is `unverified` or you have just rotated a credential, make the deployment
actually embed something:

```bash theme={null}
curl -sS -X POST https://<host>/api/config/llm-providers/embedding-config/test \
  -H "Authorization: Bearer <token>" -H 'Content-Type: application/json' \
  -d '{}'
# expect: {"success": true, ..., "dimensions": <n>}
```

It issues a real embedding request against the configured provider and model and
returns the vector dimension it got back. `"success": false` carries the provider's own
error text, which is usually the whole diagnosis. This is the only surface that makes a
provider call on demand, and it needs `ACCESS_CONFIG_PANEL`.

**4. The boot-time smoke test** leaves a greppable line in the logs, so an incident can
be dated to a restart:

```bash theme={null}
kubectl -n $NS logs $RELEASE-0 -c genesis | grep -i 'failed startup smoke test'
# a line here means: "Document indexing will fail until this is fixed."
```

<Check>
  **Gate — the deployment is healthy when all five hold.** Anything less is a partial
  answer:

  1. `curl -sS -o /dev/null -w '%{http_code}\n' https://<host>/api/healthcheck` → `200`
  2. `kubectl -n $NS get pods` and `kubectl -n $WS_NS get pods` → nothing in
     `ImagePullBackOff`, `CrashLoopBackOff` or `Pending`
  3. A signed-in user sends a chat message and gets a reply
  4. The embedding connection test returns `"success": true` with a dimension count —
     **not** merely a chip that is not red
  5. A user opens a terminal and gets a shell
</Check>

### The chart's own validators

The chart ships render-time validators, and they are packaged **inside the chart
tarball**, so an operator who never sees the source can still run them:

```bash theme={null}
helm pull <chart-ref> --version <chart-version> --untar --untardir /tmp/genesis-chart
bash /tmp/genesis-chart/genesis/validate-cli-terminal.sh
bash /tmp/genesis-chart/genesis/validate-retention.sh
```

Expected final lines, and both exit `0`:

```text theme={null}
In-pod coding agent mode renders correctly.
Retention contract holds in the rendered manifest.
```

<Note>
  These render the chart locally and **never contact a cluster** — `KUBECONFIG` is
  pinned to `/dev/null` inside them. They answer "does this chart version still produce
  a correct, data-safe manifest", which is exactly the question to ask before an upgrade
  and after a rollback. They cannot tell you anything about your running deployment; the
  gate above does that. Invoke them with `bash` — files inside a Helm tarball are not
  extracted executable.
</Note>

## Teardown without data loss

The order is not a style preference. Each step removes something the next step needs.

<Note>
  **`cluster/scripts/…` is not in the chart tarball.** Unlike the validators, these
  scripts live in the Genesis repository, so a deployment installed only from the OCI
  chart does not have them — ask Genesis for them **before** you need them, not during
  a teardown.

  Do not improvise a `kubectl delete pod` equivalent. The ownership checks described
  below are the entire point of these scripts: on a cluster running more than one
  Genesis instance, acting on the chart's default namespace means stopping another
  instance's users mid-turn and, with the wrong flag, destroying their saved
  conversations.
</Note>

<Steps>
  <Step title="Drain the workspaces first — while the release is still up">
    ```bash theme={null}
    cluster/scripts/drain-workspaces.sh $RELEASE
    ```

    This deletes the workspace **Pods** and nothing else: every PVC and the workspace
    namespace stay, so each user resumes on their next attach.

    It must run first because afterwards nothing can do it: the reaper goes with the
    backend, the RoleBinding that let it act goes with the release, and the
    release-to-namespace mapping goes with `helm get values`. It is a separate script
    from `teardown.sh` precisely so it is runnable on a release that is **staying up**
    — which is also what you run before setting `cliTerminal.enabled=false`.
  </Step>

  <Step title="Uninstall the release">
    ```bash theme={null}
    helm uninstall $RELEASE -n $NS
    ```

    This leaves behind, by design: the data PVC, the Genesis namespace, the workspace
    namespace and every workspace PVC, the ServiceAccounts, and the per-instance sandbox
    RoleBinding and ClusterRoleBinding. All carry `helm.sh/resource-policy: keep`.
  </Step>

  <Step title="Revoke the retained RBAC">
    ```bash theme={null}
    cluster/scripts/cleanup-retained-rbac.sh $INSTANCE_NS
    ```

    <Warning>
      **`$INSTANCE_NS`, not `$NS`.** The argument is the **instance** namespace — the
      identity the chart wrote into the label — and on a release that sets
      `values.namespace` that is a different value from the Helm release namespace.
      Passing the Helm namespace there searches for an owner that does not exist: the
      script reports nothing to delete, the bindings stay, and the final gate below
      inspects the wrong identity and agrees. By this point `helm uninstall` has already
      removed the release metadata, so `$INSTANCE_NS` must have been captured **before**
      the uninstall — which is why it is read in
      [Pin the identity](#pin-the-identity-before-you-touch-anything) and not here.
    </Warning>

    Until this runs, those two bindings keep granting sandbox pod management and
    namespace read to the instance's ServiceAccount. They live **outside** the Genesis
    namespace — the ClusterRoleBinding is cluster-scoped and the RoleBinding is in the
    sandbox namespace — so deleting the Genesis namespace does not reach them. An
    operator who tore the instance down has not revoked what they believe they revoked.

    Its argument is the **Genesis instance namespace**, and it selects on the
    `genesiscomputing.ai/instance-namespace` label rather than on a Helm release name,
    so it still works after the release record is gone. It deliberately leaves the
    shared, fixed-name sandbox objects alone: other instances bind to them.
  </Step>

  <Step title="Retire the data, deliberately and last">
    ```bash theme={null}
    kubectl -n $NS get pvc                     # look at what you are about to destroy
    kubectl -n $NS delete pvc <the-claim-name-you-just-read>
    kubectl delete namespace $NS
    ```

    Whether the underlying disk is destroyed depends on the StorageClass reclaim policy
    (`Delete` or `Retain`) — check it before assuming either.
  </Step>

  <Step title="Retire the workspace namespace, separately and explicitly">
    The workspace namespace holds every user's home volume and therefore their saved
    conversations, so nothing removes it implicitly — not `helm uninstall`, and not
    setting `cliTerminal.enabled=false`. Removal is its own command, and it asks first
    unless you pass `--force`:

    ```bash theme={null}
    cluster/scripts/teardown.sh <name> --delete-workspaces
    ```

    To retire **one** departed user instead of everyone, use the PVC commands in
    [Offboarding](#offboarding-the-step-everyone-misses).
  </Step>
</Steps>

### Ownership is by instance label, never by release name

Both cleanup scripts refuse to act on a workspace namespace they cannot show belongs
to this instance, and the reason is worth understanding before you reach for a flag to
override it.

| The live namespace says | What happens | This is |
| - | - | - |
| `genesiscomputing.ai/instance-namespace` = **this** instance | Drained, and deleted when asked — **even if `cliTerminal.enabled` is now `false`**, because disabling leaves the Pods running and teardown is the last thing that can stop them | **Ownership-verified** |
| **No** such label | Acted on only when the release currently enables the feature, or when you name the namespace with `--workspace-ns` | **Operator-trusted**, not verified. A namespace older than the chart version that writes the label, or one you brought with `createNamespace: false` |
| The label names a **different** instance, or ownership cannot be read at all | Refused, with the reason | Never automatic, and `--workspace-ns` does not override it |

<Warning>
  **The identity is the instance namespace, never the Helm release namespace.** On an
  installation that sets `values.namespace`, those are two different values, and
  comparing a workspace namespace's owner against the *release* namespace refuses that
  instance its **own** workspaces. Both scripts resolve the identity from the release
  and carry it into every command they print, because after the uninstall nothing can
  resolve it again.

  `drain-workspaces.sh --instance-ns <ns>` supplies it by hand once no release is left
  to read. While the release **is** readable, that flag may only confirm what the release
  says — a different value is refused, because an identity accepted from the command line
  would let a live teardown drain another instance's workspaces by naming that instance.
  `teardown.sh` takes no such flag at all.
</Warning>

<Check>
  **Gate — teardown is complete when all four hold.** Note the selector uses
  `$INSTANCE_NS`: on a release that sets `values.namespace`, checking `$NS` here asks
  about an identity that never existed and passes while every binding is still standing.

  ```bash theme={null}
  helm list -A | grep -i "$RELEASE"
  # expect: no output

  kubectl get clusterrolebinding -l genesiscomputing.ai/instance-namespace=$INSTANCE_NS
  kubectl get rolebinding -A -l genesiscomputing.ai/instance-namespace=$INSTANCE_NS
  # expect: no resources found, from both

  kubectl get pvc -A | grep genesis
  # expect: only claims you have decided to keep
  ```

  A PVC you meant to destroy that is still listed is a disk still being billed; a
  binding still listed is access you believe you revoked and did not.
</Check>

## Incident triage

**This catalogue is the single home for Genesis failure modes.** The install runbooks
link here rather than restating it, so there is one description to keep current. Each
row is symptom → cause → fix; read the symptom column first and do not skip to the fix,
because several of these present identically and are cured by opposite actions.

### The cluster never comes up

<AccordionGroup>
  <Accordion title="Symptom: `0 nodes out of 1 have registered`, or kube-system pods stuck ContainerCreating with CSINode at 0 drivers and PVCs unbound">
    **Cause.** The nodes cannot reach the container registry, so they cannot pull
    Kubernetes' **own** system images (`pause`, `netd`, `pdcsi`, `kube-proxy`). The node
    then never registers. Nothing in the Genesis chart or your values file is at fault —
    and the symptom wears different costumes on different node modes, which is why it
    reads as a storage, chart or quota problem and is none of them.

    **Verify — the first minute after the cluster object exists.** This is the highest
    value minute in the whole install:

    ```bash theme={null}
    kubectl get pods -n kube-system
    ```

    Every system pod should reach `Running` within roughly a minute. A cluster creation
    that timed out still leaves a usable kubeconfig behind, so this works on that wreck
    too — you do not have to delete and retry to diagnose it.

    **Fix.** Grant node egress to the registry before creating the cluster: a NAT
    gateway, a private-access path to the registry, or a tag-scoped internet route. The
    exact mechanism is per-cloud — see [Per-cloud specifics](#per-cloud-specifics) and,
    for GCP, the detailed remedies in the
    [GKE install runbook](/docs/deployments/GCP/Genesis-On-GCP-GKE-Install).

    **And a route is not a permission.** A higher-priority egress deny — a VPC firewall
    rule, or a policy inherited from a folder or organisation — blocks the pull with a
    perfect route in place, and the symptom is identical. Read the *effective* rules, not
    the network's own.
  </Accordion>

  <Accordion title="Symptom: the backend pod sits Pending on a volume that never attaches, mid-install">
    **Cause.** A preemptible / Spot node was reclaimed during the install, churning the
    node the backend was scheduled on and wedging PVC binding. It reads as a storage
    fault and is a node one.

    **Fix.** Bring the cluster up on on-demand nodes and finish the install. Spot is fine
    for read-only workloads afterwards.
  </Accordion>
</AccordionGroup>

### The deployment is up but something is wrong

<AccordionGroup>
  <Accordion title="Symptom: NetworkPolicy applied cleanly and enforces nothing">
    **Cause.** Kubernetes accepts a NetworkPolicy object whether or not your CNI
    implements it. On a non-enforcing cluster the policy is inert, workspace egress stays
    wide open, and there is **no error on either side** — the failure looks exactly like
    working containment.

    This is measured, not assumed. From the chart's own `values.yaml`:

    > It is only as good as your CNI. Kubernetes accepts the object whether or not the
    > CNI enforces it. Measured on stock EKS 1.34 with the default VPC CNI: applied
    > cleanly, enforced nothing, workspace reached the public internet. GKE blocked it.
    > Verify.

    **Verify on the cluster, never from the setting — and run the control first.**
    A blocked connection on its own proves nothing: in a locked-down enterprise VPC the
    cluster may have no egress to that address at all, so a probe that simply fails tells
    you the destination was unreachable, not that the policy did the blocking. Two Pods,
    identical except for the one label the policy selects on:

    ```bash theme={null}
    # 1. CONTROL — no workspace label, so the policy does not select it.
    #    This measures whether the cluster can reach the destination at all.
    kubectl -n $WS_NS run probe-control --rm --attach --restart=Never \
      --image=busybox --command -- \
      sh -c 'if nc -z -w 5 1.1.1.1 443; then echo CONTROL-REACHED; else echo CONTROL-BLOCKED; fi'

    # 2. SUBJECT — identical, but carrying the label the policy selects.
    kubectl -n $WS_NS run probe-subject --rm --attach --restart=Never \
      --image=busybox --labels=app=genesis-cli-terminal --command -- \
      sh -c 'if nc -z -w 5 1.1.1.1 443; then echo SUBJECT-REACHED; else echo SUBJECT-BLOCKED; fi'
    ```

    `--attach` blocks until each container exits and `--rm` removes the Pod, so there is
    nothing to clean up and no half-started Pod to read at the wrong moment.

    | Control | Subject | Verdict |
    | - | - | - |
    | `CONTROL-REACHED` | `SUBJECT-BLOCKED` | **Enforced.** The label is the only difference between the two Pods, so the policy is what blocked it. |
    | `CONTROL-REACHED` | `SUBJECT-REACHED` | **Not enforced.** The policy is decorative — your CNI is not implementing it. |
    | `CONTROL-BLOCKED` | anything | **Inconclusive.** The cluster cannot reach that destination regardless of the policy, so this probe cannot see the policy at all. Re-run against a destination the control *can* reach, and do not record a result until it does. |

    <Warning>
      **Two readings that look like a pass and are not.** `CONTROL-BLOCKED` with
      `SUBJECT-BLOCKED` is the trap: it looks exactly like containment and is the normal
      state of a cluster with no internet egress, where the policy may be entirely inert.
      And **silence is never a pass** — a Pod that was not scheduled or could not pull
      `busybox` prints nothing. Insist on one of the four words from each command; if one
      is missing, check `kubectl -n $WS_NS get events --sort-by=.lastTimestamp | tail`
      and run it again.
    </Warning>

    **Fix.** Turn on your CNI's policy engine — the mechanism is per-cloud, and on GKE it
    is **creation-time only** for Dataplane V2, with a Calico retrofit available on an
    existing cluster. See [Per-cloud specifics](#per-cloud-specifics).

    **Before you treat this as an outage**, check whether you needed the policy at all:
    `cliTerminal.networkPolicy.enabled` is **off by default**, and that is the intended
    posture. See [What the network posture actually is](#what-the-network-posture-actually-is).
  </Accordion>

  <Accordion title="Symptom: the terminal returns 403 or 'disabled' although the chart rendered everything">
    Three different failures share this description. The WebSocket close reason tells
    you which, so read it before changing anything — they are cured by opposite actions.

    **Where to read it.** The server closes the socket *before* accepting it, with close
    code `1008` and the reason as text. Take it from the browser's devtools console or
    Network → WS on the failed attach, or from the backend log, which records the same
    string:

    ```bash theme={null}
    kubectl -n $NS logs $RELEASE-0 -c genesis | grep 'CLI terminal: refusing connection'
    ```

    | Close reason | Cause | Fix |
    | - | - | - |
    | *"CLI terminal access is disabled on this deployment"* | The **effective** `cli_terminal.enabled` is false. Either the chart flag is off, or an admin stored `false` in Advanced Settings on a build or a path where the chart does not force-lock it. | Set `cliTerminal.enabled=true` in the values file and upgrade. That writes **both** `GENESIS_CLI_TERMINAL_ENABLED` and the `GENESIS__CLI_TERMINAL__ENABLED` force-lock, so a stored panel value can no longer contradict it. Confirm in **Config Sources**, which attributes the effective value to `env_override`. |
    | *"Access denied"* | The user lacks the `use_cli_terminal` permission. This is Genesis RBAC, not Kubernetes RBAC — nothing about the cluster is wrong. | Grant the permission, or the role carrying it, to that user's IdP group. |
    | The socket attaches, then workspace creation fails with a Kubernetes 403 | The namespace or RBAC is missing, or the backend is asking in a **different** namespace from the one the chart granted. | Check that `cli_terminal.k8s_namespace` equals the namespace the RoleBinding is in. This is the disagreement `cliTerminal.enabled=true` exists to prevent: it pins the namespace to the objects it just rendered. |

    ```bash theme={null}
    kubectl auth can-i create pods -n $WS_NS \
      --as=system:serviceaccount:$NS:<the-backend-service-account>
    # expect: yes
    kubectl -n $WS_NS get role,rolebinding,resourcequota
    ```

    **The historical trap.** Earlier deployments applied the terminal platform with a
    separate `kubectl apply` of a manifest. That is no longer the path: the chart renders
    the namespace, RBAC and quota itself. A hand-applied copy now diverges silently from
    the chart's — only one of the two ever gets updated.
  </Accordion>

  <Accordion title="Symptom: a Kubernetes 403 on workspace creation for a user who could attach yesterday">
    **Cause.** The **quota**, not RBAC. `cliTerminal.maxWorkspaces` renders the
    namespace's `count/pods` ResourceQuota *and* seeds the backend's own cap — but the
    backend's cap is only a default, and an admin who raised it in Advanced Settings
    outranks it immediately while the quota stays where it was rendered. The application
    then admits a workspace the API server refuses.

    ```bash theme={null}
    kubectl -n $WS_NS get resourcequota -o yaml | grep -A6 'hard:'
    kubectl -n $WS_NS get pods | wc -l
    ```

    **Fix.** Raise it in the values file and re-render — not in the panel. See
    [Raising the cap](#raising-the-cap). The quota is the authority.
  </Accordion>

  <Accordion title="Symptom: every workspace is ImagePullBackOff, for a correctly configured operator">
    **Cause.** One of two, and they are distinguishable:

    1. **The tag is not in the mirror.** `image.tag` must be a tag that actually exists
       in the registry your nodes pull from. The chart's default falls back to the chart's
       `appVersion`, which is frequently **not** a tag present in the distribution registry
       — only a named channel tag is. An upgrade that mirrors the backend image and forgets
       the workspace image produces exactly this: a working console, no terminals.
    2. **The node's identity cannot pull.** A Kubernetes node pulls as **its own**
       service account or instance identity, not as the human who ran `helm`. A grant made
       to your corporate directory does not cover the node.

    ```bash theme={null}
    kubectl -n $WS_NS describe pod <workspace-pod> | tail -25
    # read the Events: "manifest unknown" is cause 1; "denied"/"unauthorized" is cause 2
    crane manifest <registry>/<repo>/genesis-sandbox-runner:<tag> > /dev/null && echo TAG-EXISTS
    ```

    **Fix.** For (1), mirror the tag, or point `GENESIS_CLI_TERMINAL_IMAGE` at a tag that
    exists. For (2), grant the node identity read on the registry — the mechanism is
    per-cloud, see [Per-cloud specifics](#per-cloud-specifics).
  </Accordion>

  <Accordion title="Symptom: chat works, but search results got worse and nobody can say when">
    **Cause.** The embedding endpoint is unroutable or not served. Genesis does not fail
    the query: semantic search falls back to graph-only lexical matching and the index
    reports `ready_with_warnings`, so the install keeps answering and the degradation is
    invisible from chat.

    **Verify the embedding path independently of chat** — they are separate routes and
    frequently separate endpoints and credentials. In order:

    1. `POST /api/config/llm-providers/embedding-config/test` — forces a real embedding
       call and returns the dimension, or the provider's own error text.
    2. Read `embedding_status` and `embedding_status_reason` from the AI Providers page
       or `GET_INDEX_STATS` for the recorded history behind that one call.
    3. `kubectl -n $NS logs $RELEASE-0 -c genesis | grep -i 'failed startup smoke test'`
       to date the failure to a restart.

    All three are in [Health and verification](#chat-working-does-not-mean-embeddings-working).

    **Fix**, by what the reason says:

    | Reason names | Cause | Fix |
    | - | - | - |
    | An unroutable model id | The model id has no provider prefix, so nothing can dispatch it. Conclusive before any call is made. | Configure a provider-qualified model id, or set the embedding provider explicitly. |
    | Repeated failures, never a success | The configuration is wrong — wrong endpoint name, or a credential that was never valid for it. | Confirm the serving endpoint exists under exactly that name on your LLM host, and that the credential is scoped to it. |
    | Worked, then stopped | An expired credential, a withdrawn serving endpoint, or a network path that changed. The bar for this verdict is deliberately higher, because a proven identity is known routable. | Rotate or restore the credential; confirm the endpoint is still served. |

    Recovery is observable: a single success clears the failure record and the status
    returns to `ok` without a restart.

    <Note>
      **This is a Professional Services responsibility at install time, not a
      customer-infrastructure one.** The embedding serving endpoint must exist and be
      reachable, and it must be verified **separately from chat**. An install signed off
      on "the bot replies" ships with semantic search dead and nobody knowing for weeks.
    </Note>
  </Accordion>

  <Accordion title="Symptom: 'reset to defaults' reported success and the feature is still live">
    **Cause.** A version-gated defect. On older images, **Set default** on a
    `cli_terminal.*` or `sandbox_runner.*` row deleted the stored value and reported
    success, while the environment variable the runtime actually reads kept the value the
    panel had bridged into it. The panel showed the feature off; the terminal stayed
    reachable until the pod restarted.

    **Fix.** Upgrade the image — current builds restore the deployment's own pre-panel
    value on reset, and leave a `GENESIS__…` force-lock winning where one is set. Until
    then, never trust the panel after a reset: attempt an attach, or restart the pod, and
    read the real state.
  </Accordion>

  <Accordion title="Symptom: workspaces sit Pending and every attach fails — the feature looks broken">
    **Cause.** Starved, not broken. Either the cluster cannot supply the requests, or the
    user's volume pins them to a zone with no room, or the node pool cannot grow.

    ```bash theme={null}
    kubectl -n $WS_NS describe pod <pending-pod> | tail -20
    ```

    | Message | Cause | Fix |
    | - | - | - |
    | `Insufficient cpu` / `Insufficient memory` | Genuinely no schedulable headroom. Compare requests against allocatable minus committed, never against observed usage. | Add capacity, or lower `cliTerminal.maxWorkspaces` to what the cluster holds. |
    | `didn't match PersistentVolume's node affinity` | The user's `ReadWriteOnce` volume is bound to one zone and that zone is full. Shrinking the workspace does not help. | Free capacity in that zone, or accept losing that user's volume. |
    | `FailedScaleUp: ... quota exceeded` | The cloud account's quota refuses a **new node** — frequently the **disk** quota, not CPU. Observed at 11/200 CPU while disk sat at 418/500 and a new node needed \~100 GB. | Raise the quota that is actually exhausted. Check disk, not just CPU. |
  </Accordion>

  <Accordion title="Symptom: the upgrade 'worked' but the old behaviour persists">
    **Cause.** The pod is Ready on a cached layer under the same moving tag, or the
    rollout never actually replaced the pod.

    ```bash theme={null}
    kubectl -n $NS get pod $RELEASE-0 \
      -o jsonpath='{.status.containerStatuses[?(@.name=="genesis")].imageID}{"\n"}'
    crane digest <registry>/<repo>/genesis:<tag>
    # these two must be the same sha256
    kubectl -n $NS get pod $RELEASE-0 -o jsonpath='{.status.startTime}{"\n"}'
    ```

    **Fix.** Deploy by digest, or mirror the build to an immutable tag of your own and
    deploy that. A moving tag plus `IfNotPresent` is the combination that produces this.
  </Accordion>

  <Accordion title="Symptom: helm upgrade fails with 'invalid ownership metadata'">
    **Cause.** Helm refusing to adopt an object it did not create — typically the
    workspace namespace, applied by hand on an earlier deployment. Not a chart fault.

    ```text theme={null}
    Error: rendered manifests contain a resource that already exists.
    Unable to continue with update: Namespace "genesis-cli" ... exists and cannot be
    imported into the current release: invalid ownership metadata
    ```

    **Fix.** Set `cliTerminal.createNamespace=false` and leave the namespace where it is,
    then delete only the Role, RoleBinding, ResourceQuota and NetworkPolicy so the chart
    can render its own. **Do not delete the namespace** — that cascades to every user's
    home volume.

    A namespace this chart created re-adopts cleanly, because retention stops the delete
    without stripping the ownership annotations Helm wrote. Only a hand-created one is
    rejected.
  </Accordion>
</AccordionGroup>

### What the network posture actually is

State this accurately when asked, because the reassuring version is wrong and the
difference is a security finding:

* **By default, a workspace Pod has exactly the network access any other Pod in the
  cluster has** — the same access the Genesis backend has, because the Genesis namespace
  carries no NetworkPolicy either. That is deliberate: these are development workspaces,
  and installing packages and cloning repositories is the point of them.
* **The workspace egress lockdown is opt-in** (`cliTerminal.networkPolicy.enabled=true`),
  confining workspaces to DNS plus the Genesis backend — at the cost of `git`, `gh`, npm
  and pip timing out.
* **It is only as good as the CNI**, and an unenforcing CNI reports success. Verify it on
  the cluster before describing it to a security reviewer.

Do **not** describe agents as locked to DNS and the backend by default. They are not.

## Per-cloud specifics

The Genesis release is identical everywhere; the substrate underneath is not. Everything
above is cloud-agnostic — these are the rows that differ.

<Note>
  **How to read this table.** A cell marked **✓** is established by Genesis
  documentation or measured on a real cluster, and the linked page carries the detail.
  A cell marked **◦** names the standard mechanism for that platform, which Genesis
  does not document and you must confirm against your cloud's own documentation and
  with your platform team. Do not carry a **◦** cell into a design review as a Genesis
  requirement.
</Note>

| Concern | GCP / GKE | AWS / EKS | Azure / AKS |
| - | - | - | - |
| **Node egress to the registry** | **✓** Cloud NAT, **or** Private Google Access reaching `*.googleapis.com`, `*.pkg.dev` and `*.gcr.io`, **or** a tag-scoped `0.0.0.0/0` route on the node tag. A Cloud NAT still needs a `default-internet-gateway` route **and** must cover the node subnet's ranges — a region match alone proves nothing. And a route is not a permission: a higher-priority egress deny produces an identical symptom. | **◦** NAT gateway on the node subnets, or VPC endpoints for the registry and its backing object store. | **◦** NAT gateway or the cluster's outbound type. The Genesis doc states only the requirement — "cluster nodes must pull container images from the registry URL provided by Genesis" — not a mechanism. |
| **NetworkPolicy enforcement** | **✓** **Dataplane V2** (`--enable-dataplane-v2`), **creation-time only**. An existing Standard cluster retrofits the Calico engine with `gcloud container clusters update … --enable-network-policy` (rolling node recreation). Autopilot always enforces. Verify with `--format='value(networkConfig.datapathProvider)'`, want `ADVANCED_DATAPATH`. | **✓ (measured negative)** The default VPC CNI **enforces nothing** — measured on stock EKS 1.34: the chart's policy applied cleanly and a workspace-labelled pod reached the public internet. **◦** Enabling enforcement means turning it on in the CNI addon, or running Calico or Cilium. | **◦** A policy engine must be chosen **at cluster creation** — the chart's own AKS recipe uses `az aks create --network-plugin azure --network-policy azure`. It cannot be added to a cluster created without one. |
| **Pod identity for cloud secrets** | **◦** Workload Identity on the cluster, with `secretmanager.secretAccessor`, for `gcp_*` secret references. The GKE install runbook does not cover setting it up. | **✓** **IRSA.** `serviceAccount.create: true` plus `serviceAccount.annotations` carrying `eks.amazonaws.com/role-arn` — and the same annotation shape on `sandbox.podServiceAccount.annotations` for sandbox pods. `create: true` is required so the sandbox RBAC has an account to bind. Used for Secrets Manager resolution, `GENESIS_MASTER_KEY_REF` and custom CA certs, with no init container or CSI driver. | **◦** Workload Identity / managed identity. The Genesis doc describes it in two sentences of prose and gives no annotation key or values example. |
| **Registry and node pull grant** | **✓** Artifact Registry. The **node's own service account** needs `roles/artifactregistry.reader` on the repository — a grant to your corporate directory does not cover the node. No pull secret. | **✓** ECR, first-class in the chart: `ecr.enabled: true` with `ecr.registry` and an `ecr.token` passed at install time creates the pull secret and overrides `image.repository`. The token is short-lived, so deploy scripts refresh it every run. | **✓** Also **ECR**, not ACR — Azure deployments pull from the Genesis registry through a `kubectl create secret docker-registry` secret named in `imagePullSecrets`. |
| **Default StorageClass** | **✓** `standard-rwo` (pd-balanced — counts against **SSD** quota, which is the quota that usually runs out first) | **✓** cluster default, typically `gp2` or `gp3` | **✓** cluster default, typically `managed-csi` or `managed-premium` |
| **Snapshot mechanism** | **◦** Persistent Disk snapshots (`gcloud compute disks snapshot <disk> --zone <zone>`). **There is no GCP backup doc** — this row is generic, and the restore drill above is the part that matters. | **✓** EBS snapshots, three worked paths — Velero, AWS Backup for EKS, or a manual `aws ec2 create-snapshot`. See [AWS backup and recovery](/docs/deployments/AWS/Backup-And-Recovery). | **✓** Kubernetes `VolumeSnapshot` with the Azure Disk CSI driver, or Azure Backup for AKS. See [Azure backup and snapshot options](/docs/deployments/Azure/Backup-And-Snapshot-Options) — options and trade-offs rather than commands. |
| **Quota names to check before scaling** | **✓** `CPUS`, `SSD_TOTAL_GB`, and `IN_USE_ADDRESSES` if you use a LoadBalancer or Ingress. **Check disk, not just CPU** — node scale-up has been observed failing on disk quota (418/500) with CPU at 11/200. | **◦** vCPU limits per instance family, EBS volume and snapshot limits, Elastic IP limits. Not named in the Genesis docs. | **◦** Regional vCPU quota per family, managed disk limits, public IP limits. Not named in the Genesis docs. |

Whatever the cloud, the **`VolumeSnapshot`** API is the portable option where your
cluster has a CSI driver and a snapshot controller installed: one procedure everywhere,
at the cost of a cluster prerequisite. The cloud-native command always works.

<Note>
  **Two facts hold on every cloud, and neither is in the snapshot commands**: run the
  [checkpoint hook](#snapshot-the-volume) before the snapshot, and back the
  [master key](#the-master-key) up separately — it is not on the volume. Use the
  per-cloud pages for the snapshot mechanics, and the
  [restore drill](#the-restore-drill) for the part none of them covers: proving the
  backup actually restores.
</Note>

***

Related: [Installing Genesis on GKE](/docs/deployments/GCP/Genesis-On-GCP-GKE-Install) ·
[Genesis on AWS](/docs/deployments/AWS/Genesis-On-AWS) ·
[Genesis on Azure](/docs/deployments/Azure/Genesis-On-Azure) ·
[Environment strategy and governance](/docs/deployments/Environment-Strategy-And-Governance)