Genesis Operations Runbook - Genesis Computing

Everything after the install. This runbook is written for the person who is paged at 02:00 and has never read the Genesis source: each procedure states what you run, what a good result looks like, and what to do when it is not. It is cloud-agnostic — Genesis on Kubernetes is the same StatefulSet, the same chart and the same data volume on GKE, EKS and AKS. Only the substrate underneath differs, and every substrate difference is collected in Per-cloud specifics at the end rather than scattered through the text.

Installing, not operating? Start with the install runbook for your cloud — GKE, EKS, AKS — and come back here once it is up.

Pin the identity before you touch anything

Every procedure below needs these facts, and every one of them is read from the cluster rather than remembered. Write them at the top of your incident notes.

# 1. Release name and namespace — the first column and the namespace column.
#    NOT the image tag and NOT the chart version, which are the other two facts.
helm list -A | grep -i genesis

# Set these once; the rest of this runbook uses them.
RELEASE=genesis
NS=genesis               # the HELM release namespace, for helm and kubectl -n
WS_NS=genesis-cli        # cliTerminal.namespace — read it back, do not assume

# 2. The INSTANCE namespace, which is the deployment's identity and is NOT always
#    $NS. The chart renders the StatefulSet into `values.namespace` where a release
#    sets one, and only otherwise into the Helm namespace — so the StatefulSet's own
#    live namespace IS the identity, whatever the values did.
INSTANCE_NS=$(kubectl get statefulset -A -l app.kubernetes.io/instance=$RELEASE \
  -o jsonpath='{.items[0].metadata.namespace}')
echo "$INSTANCE_NS"

# 3. Chart version and app version currently deployed — the CHART and
#    APP VERSION columns. Neither is the release name in column one.
helm list -n $NS

# 4. The pod. The StatefulSet is named from the Helm fullname, which is USUALLY the
#    release name — read it back rather than assuming "$RELEASE-0".
kubectl -n $NS get statefulset
#    app.kubernetes.io/instance is the release, so this stays correct on a cluster
#    running more than one Genesis release in one namespace.
POD=$(kubectl -n $NS get pod -l app.kubernetes.io/instance=$RELEASE -o name | head -1)

# 5. The image DIGEST actually running — not the tag you asked for.
kubectl -n $NS get $POD \
  -o jsonpath='{.status.containerStatuses[?(@.name=="genesis")].imageID}{"\n"}'

# 6. The data volume, and that it is Bound.
kubectl -n $NS get pvc

The rest of this runbook writes $RELEASE-0 for brevity, which is right on a default install. Where yours differs, substitute the pod name from step 3.

The release name is not a version. helm upgrade with a new release name creates a new StatefulSet, which creates a new, empty PersistentVolumeClaim — the old data stays on an orphaned claim under the old name. This is the single most common way a Genesis deployment loses its database. Always upgrade the release name helm list reports, in the namespace helm list reports.

Read the workspace namespace back; never assume genesis-cli. It is configurable (cliTerminal.namespace), and on a cluster running more than one Genesis instance the chart’s default is somebody else’s namespace, full of other people’s attached sessions:

helm get values $RELEASE -n $NS -a -o json | jq -r '.cliTerminal.namespace, .cliTerminal.enabled'

Note what that does not tell you. helm get values -a returns computed values, so every release installed from a chart that ships genesis-cli as its default reports genesis-cli whether the feature is on or off. The answer names a namespace; it does not establish that this release owns it. Ownership is the genesiscomputing.ai/instance-namespace label on the live namespace — see Teardown without data loss.

Upgrade

A Genesis upgrade is normally an image upgrade: same chart, same release name, same namespace, same PVC. The pod is replaced; the volume reattaches.

1. Pull the new image into your mirror

If your nodes pull from a mirror rather than from the Genesis registry, the mirror is the first thing to update — and the tag you are about to deploy must exist in it. crane copies registry-to-registry without a local Docker daemon and preserves the source digest and every platform in the manifest:

SRC=us-central1-docker.pkg.dev/genesis-471800/genesis-dist
DST=<your-registry>/<your-repo>
crane copy $SRC/genesis:<new-tag> $DST/genesis:<new-tag>
crane copy $SRC/genesis-sandbox-runner:<new-tag> $DST/genesis-sandbox-runner:<new-tag>

Mirror the workspace image too, in the same pass. An operator who upgrades the backend and forgets the sandbox runner gets a working console in which every terminal fails to start — see Every workspace is ImagePullBackOff.

On a laptop, pin the platform. docker pull without --platform linux/amd64 on an Apple-silicon machine mirrors an arm64 image that your nodes cannot run, and the failure surfaces later as a crash-looping pod rather than as a bad copy. crane copy has no such trap — it copies the whole manifest list.

2. Run the upgrade

helm upgrade $RELEASE <chart-ref> --version <chart-version> \
  -n $NS -f my-values.yaml \
  --set image.tag=<new-tag> \
  --wait --timeout 12m

Do not use --reuse-values. It carries the previous release’s values forward and ignores the new chart’s defaults, so a values key introduced after your release was installed is simply missing at upgrade time. The chart has been broken this way for real: secrets.existingSecretMintlifyJwtPrivateKey was added with a correct default, which does nothing on a --reuse-values upgrade, and every lab environment failed to render at once. Always pass your complete values file with -f. (The chart now guards this case — validate-reuse-values-upgrade.sh renders with each optional secret map explicitly absent — but -f is still the supported path.)

3. Verify by content, not by “Ready”

A Ready pod proves a process started. It does not prove it is the build you shipped: main, latest and any other moving tag resolve to whatever the registry holds at pull time, and IfNotPresent will happily keep a cached older layer under the same tag. Verify the digest, and then verify the behaviour you upgraded for.

# a. The digest actually running. Compare it to the digest you copied in step 1.
kubectl -n $NS get pod $RELEASE-0 \
  -o jsonpath='{.status.containerStatuses[?(@.name=="genesis")].imageID}{"\n"}'
# expect: <registry>/genesis@sha256:<64 hex chars>

# b. The registry's digest for the tag you deployed.
crane digest $DST/genesis:<new-tag>
# expect: the same sha256 as (a)

# c. The server is serving, not merely running.
kubectl -n $NS exec $RELEASE-0 -c genesis -- \
  curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8080/api/healthcheck
# expect: 200

# d. The claim never changed identity.
kubectl -n $NS get pvc
# expect: the SAME claim name as before the upgrade, STATUS Bound

Gate — do not declare the upgrade done until all four hold: (a) equals (b); (c) is 200; (d) names the same Bound claim as before; and the specific behaviour you upgraded for is exercised by hand. If (a) and (b) disagree, the node is running a cached layer — see The upgrade “worked” but the old behaviour persists.

What “version-gated” means

Some Genesis fixes change behaviour that has no switch: the code either does the old thing or the new one. A fix described as version-gated is delivered only by running an image that contains it, and no amount of configuration reproduces it on an older image. That is why step 3(a) reads the digest rather than the tag. When a support answer says “fixed in current builds”, the question it is answering is which image is running — and the only answer that settles it is the digest, checked against the digest of the build the fix landed in.

Rollback

Roll the release back

helm history $RELEASE -n $NS            # find the revision you want
helm rollback $RELEASE <revision> -n $NS --wait --timeout 12m

helm rollback restores the rendered manifest of that revision — including the image reference it carried. That reference is always repository:tag: the chart takes a tag, not a digest. So if your values file pins a moving tag (main), the rollback restores the tag and not the build, the node may pull the newest image again, and you land back where you started with a green rollback to show for it. The fix is to make the tag itself immutable. Mirror the known-good digest to a tag of your own that nothing will ever move, then deploy that tag:

crane copy $DST/genesis@sha256:<the-known-good-digest> $DST/genesis:known-good-2026-09-10

helm upgrade $RELEASE <chart-ref> --version <chart-version> -n $NS -f my-values.yaml \
  --set image.tag=known-good-2026-09-10

Then verify by digest exactly as in step 3 of an upgrade — the running imageID must equal the digest you copied. The point is the same in both directions: deploy a build, not a label.

Never leave image.tag unset. With no tag the chart falls back to the chart’s appVersion, which is frequently a tag that does not exist in the registry your nodes pull from — so the rollback “succeeds” and the pod never starts.

What survives a rollback, and what does not

Survives	Does not survive
The data volume. The StatefulSet sets `persistentVolumeClaimRetentionPolicy.whenDeleted: Retain`, and a rollback does not delete the claim.	Schema or data migrations the newer build performed. Helm rolls back Kubernetes objects; it cannot un-migrate a SQLite database.
Workspace volumes and the workspace namespace — they carry `helm.sh/resource-policy: keep`.	Running workspace Pods, if the rollback restarts the backend: sessions are interrupted. The volumes are untouched, so users resume on their next attach.
Secrets you supplied through `existingSecret*` references, which live outside the release.	Secret values passed with `--set` on the newer revision, if the older revision’s rendered manifest carried different ones.

A rollback to a chart revision older than 0.3.70 strips the retention annotation from the live objects while leaving the objects in place. The namespace and the ServiceAccounts stay, but they are no longer protected: the next operation that removes them from a rendered manifest will delete them, and a namespace delete cascades to every PVC inside it. Re-check after any rollback:

kubectl get namespace $NS \
  -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}{"\n"}'
kubectl get namespace $WS_NS \
  -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}{"\n"}'
# expect: keep      (an empty line is a FAIL — re-upgrade forward before proceeding)

Gate — rollback complete when: helm history shows the target revision deployed; the running digest is the one you intended; /api/healthcheck returns 200; the PVC is the same Bound claim; and both namespaces still report resource-policy: keep.

Backup and restore

What is on the volume

One PersistentVolumeClaim per deployment (genesis-data-<statefulset-name>-0 by default), mounted into the backend at four subpaths:

Path in the container	What it holds
`/app/.genesis/db`	The SQLite application database — configuration, users and roles, bots, threads and conversations. The thing you cannot rebuild.
`/app/bot_storage`	Files bots have written or been given.
`/app/bot_git`	Bot git checkouts.
`/app/tmp`	Scratch. Safe to exclude from a backup.

Workspace home volumes are separate claims in the workspace namespace, two per user. They hold each person’s saved Claude Code conversations and working tree. They are not on the backend volume and are not covered by a backend snapshot.

Snapshot the volume

The reliable mechanism is a volume snapshot taken by the storage layer, not a kubectl cp. Find the underlying disk, then snapshot it with your cloud’s mechanism (commands per cloud are in Per-cloud specifics):

# The claim. Its name is <volumeClaimTemplate>-<statefulset>-0, and the StatefulSet is
# named from the Helm fullname — which is USUALLY but not always the release name, so
# read the name back rather than composing it.
kubectl -n $NS get pvc
PVC=$(kubectl -n $NS get pvc -o name \
  -l app.kubernetes.io/instance=$RELEASE 2>/dev/null | head -1)
# StatefulSet claims do not always inherit the release labels. If the line above came
# back empty, take the claim the live Pod is actually using — which is the definition:
PVC=${PVC:-persistentvolumeclaim/$(kubectl -n $NS get $POD \
  -o jsonpath='{.spec.volumes[?(@.name=="genesis-data")].persistentVolumeClaim.claimName}')}
echo "$PVC"

# The PersistentVolume behind it, then the cloud disk handle behind that
PV=$(kubectl -n $NS get $PVC -o jsonpath='{.spec.volumeName}')
kubectl get pv $PV -o jsonpath='{.spec.csi.volumeHandle}{"\n"}'

That last value is the disk your cloud’s snapshot command takes as input. Snapshot that handle — not a claim name you composed, which on a deployment using fullnameOverride names a claim that does not exist, or worse, one that does and belongs to something else.

The SQLite WAL must be captured with the database — checkpoint first. Genesis runs SQLite in write-ahead-logging mode: recent committed transactions live in genesis.db-wal and genesis.db-shm beside genesis.db until a checkpoint folds them in, and a deployment holds several databases across /app/.genesis/db and /app/bot_storage. A raw snapshot taken mid-transaction, or a file copy that takes genesis.db alone, restores the database as of the last checkpoint rather than as of the snapshot — and it restores cleanly, with no error, which is what makes this dangerous.The image ships a pre-backup hook that folds every WAL in, in place, in seconds, without stopping the application. Run it immediately before every snapshot:

kubectl -n $NS exec $RELEASE-0 -c genesis -- /app/scripts/sqlite-checkpoint-hook.sh
# expect a "checkpointed: ..." line per database, then:
#   checkpoint complete: N succeeded, 0 failed

It exits non-zero if any database failed to checkpoint, so wire it as a gating step rather than a best-effort one. For Velero, attach it as a pod annotation so every scheduled backup checkpoints first:

podAnnotations:
  pre.hook.backup.velero.io/command: '["/app/scripts/sqlite-checkpoint-hook.sh"]'
  pre.hook.backup.velero.io/timeout: "60s"
  pre.hook.backup.velero.io/on-error: Fail

The master key is not on the volume. Restoring the data volume without the matching GENESIS_MASTER_KEY gives you a database whose encrypted fields — every stored credential — cannot be read. A volume snapshot is therefore half a backup. The other half is the key, backed up separately, in a place that survives the loss of this cluster. See Secrets.

For a file-level copy — useful as a second, portable line of defence, not as a replacement for volume snapshots — the chart ships one:

# Streams /app/.genesis/db, bot_storage, bot_git (and tmp unless excluded)
# out of the running pod into a local .tgz.
GENESIS_LOCAL_NAMESPACE=$NS GENESIS_LOCAL_POD=$RELEASE-0 \
  bash local-k8s/backup-genesis.sh <kubectl-context> pre-upgrade

It refuses rather than writing an empty archive: it skips with exit 2 if the pod is not Running or /app/.genesis/db is unreadable, and fails with exit 1 on an empty result. Treat any exit other than 0 as “no backup was taken”.

The restore drill

An untested backup is not a backup. Run this on a schedule — quarterly at the least, and once before the deployment carries anything that matters. Restore to a separate release so the drill can never damage the live one.

Checkpoint, snapshot, and record the identity

Run the checkpoint hook, then snapshot the live volume. Write down the snapshot id, the claim name, the chart version, the image digest that produced it, and which master key it was encrypted under. A restore into a different build, or under a different key, is a different test.

Create a claim from the snapshot

Provision a new PVC from the snapshot using your cloud’s restore mechanism, in a scratch namespace (genesis-drill). Confirm STATUS Bound before going on.

Install a second release onto that claim, with the same master key

Name the scratch release, and use that name for the rest of the drill:

DRILL=genesis-drill-restore

Install the same chart version and the same image digest into the scratch namespace, pointed at the restored claim, and supply the same GENESIS_MASTER_KEY the snapshot was taken under. Supplying a different key is how a drill “passes” while proving nothing: the server starts, the threads are there, and every stored credential is unreadable.

The restored volume carries the live deployment’s configuration, including which workspace namespace to use — and cliTerminal.enabled: false pins nothing. Leaving the chart flag off is not enough: the flag only decides whether the chart renders a platform, and the restored APP_CONFIG can still say cli_terminal.enabled: true with k8s_namespace: genesis-cli. The drill’s backend would then create and, worse, reap workspace Pods in the production workspace namespace, where a Pod and both of its volumes are named from a digest of the user id alone — so the drill and the live release collide on real people’s home volumes.Force-lock both Pod-creating features off, by name. These are the double-underscore overrides, which outrank anything stored in the restored database:

helm upgrade --install $DRILL <chart-ref> --version <chart-version> \
  -n genesis-drill --create-namespace -f drill-values.yaml \
  --set config.extraEnv.GENESIS__CLI_TERMINAL__ENABLED=false \
  --set config.extraEnv.GENESIS__SANDBOX_RUNNER__ENABLED=false \
  --set secrets.existingSecretGenesisMasterKey.secretName=genesis-master-key \
  --set secrets.existingSecretGenesisMasterKey.secretKey=GENESIS_MASTER_KEY \
  --wait --timeout 12m

Never pass the key itself with --set. It would land in your shell history, in the process arguments of a running command, and in the Helm release record — three places your escrow policy does not cover. Reference a Secret instead. Secrets are namespaced, so copy the live one into the scratch namespace first, stripping the fields that belong to the original object:

kubectl -n $NS get secret genesis-master-key -o json \
  | jq 'del(.metadata.namespace, .metadata.uid, .metadata.resourceVersion,
            .metadata.creationTimestamp, .metadata.ownerReferences,
            .metadata.managedFields)' \
  | kubectl -n genesis-drill create -f -

Where the deployment resolves its key from a cloud secret manager instead, give the drill the same config.extraEnv.GENESIS_MASTER_KEY_REF and leave secrets.genesisMasterKey empty — the drill’s service account needs read on that secret. Delete the copied Secret with the scratch namespace in Step 5.Then prove the lock took, before anyone touches the drill — a restored database that disagrees is exactly the case this guards against:

kubectl -n genesis-drill exec $DRILL-0 -c genesis -- \
  printenv GENESIS__CLI_TERMINAL__ENABLED GENESIS__SANDBOX_RUNNER__ENABLED
# expect: false / false

# And that the PRODUCTION workspace namespace is untouched. Take this BEFORE
# installing the drill, and diff it afterwards. Names AND UIDs, not counts: a
# count is equal again once a deleted volume has been replaced by a new one,
# which is the loss this baseline exists to detect.
kubectl -n $WS_NS get pod,pvc \
  -o custom-columns=KIND:.kind,NAME:.metadata.name,UID:.metadata.uid \
  --no-headers | sort > /tmp/ws-baseline.txt

To exercise the terminal in the drill instead, give it a workspace namespace of its own (cliTerminal.enabled=true with cliTerminal.namespace=genesis-drill-cli), never the production one — and own that namespace’s whole lifecycle in Step 5, not just the release’s. Two releases must never share one workspace namespace.

Prove the data came back, by content

kubectl -n genesis-drill exec $DRILL-0 -c genesis -- \
  curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8080/api/healthcheck
# expect 200

Then sign in and check three things by eye, because a healthy process proves none of them: a named user still exists with their roles; a known chat thread still opens and shows its messages; a configured LLM provider is still configured. Record what you checked and the date.

Destroy the drill — all of it

A drill that leaves objects behind has created a second lifecycle beside the real one. Retained objects never go away on their own, so removal is explicit. Follow Teardown without data loss against the drill identity, in that order:

# Only if the drill ran a terminal-enabled release with its OWN workspace namespace:
cluster/scripts/drain-workspaces.sh $DRILL -n genesis-drill

helm uninstall $DRILL -n genesis-drill
cluster/scripts/cleanup-retained-rbac.sh genesis-drill   # revokes the retained bindings

kubectl -n genesis-drill delete pvc --all
kubectl delete namespace genesis-drill
kubectl delete namespace genesis-drill-cli   # only if the drill created one

Then confirm nothing of the drill’s remains. Deletion was requested above; these check it happened — and cluster-scoped bindings in particular survive a namespace delete:

helm list -A | grep "$DRILL"                       # expect: no output
kubectl get namespace genesis-drill genesis-drill-cli   # expect: NotFound for both
kubectl get clusterrolebinding -l genesiscomputing.ai/instance-namespace=genesis-drill
kubectl get rolebinding -A -l genesiscomputing.ai/instance-namespace=genesis-drill
kubectl get pvc -A | grep genesis-drill            # expect: no output

A namespace stuck Terminating is not deleted — find what is holding it (kubectl get namespace genesis-drill -o jsonpath='{.status.conditions}') rather than recording the drill as clean.Finally, prove the production workspace namespace came through untouched:

kubectl -n $WS_NS get pod,pvc \
  -o custom-columns=KIND:.kind,NAME:.metadata.name,UID:.metadata.uid \
  --no-headers | sort > /tmp/ws-after.txt
diff /tmp/ws-baseline.txt /tmp/ws-after.txt

Lines added or removed for Pods are normal — workspaces are created and reaped while you work. Any change to a PVC line is not: a missing or re-UID’d claim is someone’s home volume, and it is an incident, not a drill result.

Gate — the backup is proven only when all five hold:

A claim provisioned from the snapshot reached Bound
The drill release returned 200, with the same master key
A human confirmed a user, a thread and a provider configuration by looking at them
diff /tmp/ws-baseline.txt /tmp/ws-after.txt shows no PVC line changed in the production workspace namespace — same names, same UIDs. The drill never touched live users’ volumes. (Pod lines may differ; workspaces come and go.)
Nothing drill-owned remains, checked rather than assumed: helm list does not show the drill release, both scratch namespaces are NotFound (not Terminating), and no Role/ClusterRoleBinding carries the drill’s instance label

Anything less has tested the storage layer, not the backup — and 4 and 5 are what make it safe to run the drill on a schedule rather than once.

Secrets

The master key

GENESIS_MASTER_KEY encrypts every credential Genesis stores. It is required — the chart refuses to render without it — and it is not recoverable: lose it and every stored connection credential in the database becomes undecryptable ciphertext, which is a rebuild, not a recovery. Three supported ways to supply it, in increasing order of operational maturity:

How	Values	Use when
Inline at install	`--set secrets.genesisMasterKey=<hex>`	A trial. The value is in your shell history and in the Helm release record.
An existing Kubernetes Secret	`secrets.existingSecretGenesisMasterKey.secretName` / `.secretKey`	The normal production path. The key is created and rotated outside the chart.
Resolved from a cloud secret manager at startup	`config.extraEnv.GENESIS_MASTER_KEY_REF` set to an `aws_` / `gcp_` reference, `secrets.genesisMasterKey` left empty	The pod holds no key at rest; it fetches it with its cloud identity (IRSA / Workload Identity).

Generate one with python3 -c "import secrets; print(secrets.token_hex(32))".

Name an escrow owner before go-live, in writing. The key must exist in a corporate secret store that survives the departure of whoever installed Genesis, with a named owner and a named deputy. This is an operator responsibility and there is no product feature that substitutes for it: Genesis cannot re-derive the key, and the deployment’s credentials are only as recoverable as the escrow record.

There is no master-key rotation procedure, and swapping the key is not a values change. Genesis documents none, and the reason is structural: every credential in the database is ciphertext under the current key, so changing the key makes all of it unreadable — the server starts, and every connection fails. Treat the key as fixed for the life of the deployment, and protect it by escrow rather than by rotation.If a rotation is forced on you by policy or by a suspected compromise, it is a migration, not an upgrade: plan a maintenance window, take and verify a backup first, expect to re-enter every stored credential under the new key by hand, and confirm each connection afterwards. Do not begin one on the strength of this paragraph — raise it with Genesis first.The corollary for backups: rotate the key and your existing backups stop being restorable. If the key ever changes, take a fresh verified backup the same day.

LLM and database credentials

Credential	Supplied as	Rotation
Databricks LLM token	`secrets.databricksApiKey`, or `secrets.existingSecretDatabricksApiKey`. The chart sets both `DATABRICKS_API_KEY` and `DATABRICKS_TOKEN` from the one value.	Update the Secret, then restart the pod so the container picks up the new value.
Databricks SQL warehouse PAT (a separate credential from the LLM one)	`secrets.databricksDbPat` / `existingSecretDatabricksDbPat`	Same.
OpenAI / Azure OpenAI key	`secrets.openaiApiKey` / `existingSecretOpenaiApiKey`	Same.

Service principal or PAT? A personal access token is issued to a person and dies with their account — the classic Genesis outage is an LLM that stops answering because the engineer who installed it left the company. For anything beyond a trial, authenticate as a service principal / machine identity owned by a team, and record its expiry in the same calendar that holds your certificate renewals. Where the platform supports it, prefer no stored credential at all: the chart’s genesisConfig path accepts auth_mode: "iam_role" for Bedrock, which authenticates through the pod’s cloud identity and stores nothing.

Two further secrets the chart accepts, both with a documented rotation:

The Mintlify documentation signing key (Ed25519, PKCS#8) is accepted only as an existing Kubernetes Secret — never in genesisConfig.content, config.extraEnv, a ConfigMap, or a plaintext Helm value. Rotating it is update-the-Secret and roll the workload, because the pod’s environment is what carries it. Give it a named security or platform owner, the same way the master key gets one.
Custom CA certificates are fetched once at pod startup, so rotating one needs a StatefulSet restart. The same is true of any value resolved from a cloud secret manager: after you rotate it there, restart so the pod re-resolves it.

An existingSecret* reference is the rotation-friendly shape for all of these, because the Secret is updated without a helm upgrade and the credential never appears in the release record. The pod reads these into environment variables at start, so a rotation takes effect on the next pod restart:

kubectl -n $NS rollout restart statefulset/$RELEASE
kubectl -n $NS rollout status statefulset/$RELEASE --timeout=12m

Gate — a rotation is complete when the pod has restarted onto the new Secret, /api/healthcheck returns 200, a chat message gets a reply, and the embedding endpoint is verified separately (see Health and verification). Chat and embeddings can use different credentials and different endpoints; proving one says nothing about the other.

Configuration precedence

Genesis reads one setting from up to five places. When a change “does not take effect”, it is almost always because a higher tier is holding the old value. Highest wins, and the Config Sources page names the tier that actually won for any field — which is the fastest way to end an argument about it:

Tier	Where it comes from	Config Sources calls it	Changed by
1	`GENESIS__<SECTION>__<FIELD>` environment variable, usually written by the chart	`env_override`	`helm upgrade`
2	A stored APP_CONFIG value	`database`	An admin in Advanced Settings, the config API, or a `genesisConfig.content` apply
3	A stored APP_CONFIG value that is itself an environment reference (`env_…`)	`db_env_var`	The same, plus whatever the referenced variable holds
4	A plain environment variable bound to the field as its default (`EMBEDDING_MODEL`, `GENESIS_CLI_TERMINAL_ENABLED`, …), typically from `config.extraEnv`	`default_env_var`	`helm upgrade`
5	The schema default in the image	`default_plain`	The image

Read the two environment-variable spellings carefully, because they look alike and behave oppositely:

GENESIS__CLI_TERMINAL__ENABLED — prefix GENESIS__, then the config path uppercased with each . replaced by __. This is the force-lock: it is applied when the configuration snapshot is built, so it outranks everything below it. It is never persisted to the database.
GENESIS_CLI_TERMINAL_ENABLED — single underscores. This is only a default. For a field declared env_leaf, the config-to-env bridge overwrites this variable with the admin’s stored value, live and with no restart — so on its own it does not hold against the panel.

A force-lock does more than pin the value — through the panel it refuses the write. An admin editing a GENESIS__-locked field in Advanced Settings gets an error, not a stored-but-overridden value:

Cannot update '<path>': field is locked by environment variable GENESIS__...

A reset on a locked field is refused the same way, and nothing is deleted:

Cannot reset fields locked by environment variables: ...

A batch update containing one locked path is refused in its entirety. The lock test is the presence of the variable, whatever its value.Two consequences worth knowing before you debug one: on a chart-managed install with cliTerminal.enabled=true, the three pinned rows below cannot be changed or reset from the panel at all — the failure is a configuration error in the UI, not a 403 at attach time. And the other write paths do not perform this check: a genesisConfig.content apply, the config CLI, and the AI Providers / Embeddings pages — which own llm.active_provider, embedding.active_provider and embedding.active_model — store the row, and the override simply keeps winning. So a stored value can exist under a lock even though the panel would have refused to create it, and for those three fields a save under a pin is accepted and takes effect only once the pin is removed.

Editing genesisConfig.content can wipe Advanced Settings. The file is not a precedence tier; it is a loader that writes into tier 2 at startup, and it behaves in two very different ways depending on whether its content changed:

Content unchanged — it fills in only fields that have no stored value, and never overwrites an admin’s panel setting.
Content changed (its hash differs, or the file path changed) — it performs a full replacement, overwriting and deleting stored rows.

So editing one line of that file and running helm upgrade discards configuration your admins set in the UI. This is the most destructive precedence interaction in the system. Decide up front whether the values file or the panel is the source of truth for a given field, and if it is the file, tell your admins the panel is advisory for it.

Which fields the chart pins

Setting cliTerminal.enabled=true writes, as container env, both spellings of three fields:

Field	Pinned to	Why it must agree with the cluster
`cli_terminal.enabled`	`true`	An admin who once switched the feature off in the panel would otherwise keep it off through the chart flag — every attach refused with “CLI terminal access is disabled on this deployment” while the cluster looks fully prepared.
`cli_terminal.runtime`	`kubernetes`	The default backend shells out to a Docker daemon that does not exist in a Pod.
`cli_terminal.k8s_namespace`	`cliTerminal.namespace`	The backend must create Pods where the chart put the RBAC. Disagreeing silently means a 403 on every attach — or worse, live workspaces in a namespace the drain scripts do not look in.

cliTerminal.maxWorkspaces is deliberately not pinned: lowering the cap in the panel is safe and a lock would block it to prevent the unsafe direction. See Scaling for the consequence of raising it there. Two asymmetries that surprise people:

cliTerminal.enabled=false pins nothing. The chart emits no lock at all when the feature is off, because deployments that provision the workspace namespace and RBAC outside the chart enable the feature through APP_CONFIG and must keep working. So false means “the chart provides no platform”, not “the feature is off everywhere”.
Off ECR, the workspace image is yours and is not locked. With ecr.enabled, the chart owns the image and pins it, because the pull secret it places authenticates to that registry and nowhere else. Off ECR, repointing the workspace image stays an admin’s to make.

What applies live and what needs a restart

Change	Effect
`cli_terminal` sizing — image, CPU, memory, volume sizes	Applies to workspace Pods created after the change. A user with a running workspace keeps it until it is reaped and recreated, so nothing interrupts an agent mid-task.
`cli_terminal.runtime`, `cli_terminal.k8s_namespace`	Fixed for the life of the server process. A live change is logged and ignored, because workspaces already created live in the old namespace and nothing else could find them to reap. Change these from the values file, and retire the existing workspaces first.
Anything supplied as a plain environment variable (`config.extraEnv`, secrets)	Read at container start. Needs `rollout restart`.

Reset to defaults

“Set default” on a configuration row deletes the stored value and falls back to the tier below. On a current build, for an unlocked cli_terminal.* or sandbox_runner.* row, that restores the deployment’s own pre-panel value — the value the environment held before an admin ever touched the panel — rather than the schema default. The response says so explicitly when it happens, naming the paths that landed on an environment default rather than a schema one. For a locked row the reset is refused outright and nothing is deleted, as above. On a chart-managed install that covers cli_terminal.enabled, .runtime and .k8s_namespace: they cannot be reset from the panel, by design.

On an older image, “reset to defaults” reads as off while the feature stays live. Before the fix, using Set default on a cli_terminal.* or sandbox_runner.* row deleted the stored value and reported success — the panel then showed the feature off — while the environment variable the runtime actually reads kept the value the panel had bridged into it. The terminal stayed reachable until the pod restarted: a live shell behind an “off” switch.This is version-gated. If your image predates the fix, either upgrade it, or verify the real state after a reset — attempt an attach, or restart the pod — rather than trusting the panel.

Users: onboarding and offboarding

Onboarding

The user signs in through your identity provider. Genesis maps their IdP groups to Genesis roles on each login, from the groupRoleMapping file supplied to the chart, so group membership is the thing to change — not anything inside Genesis.
Terminal access is permission-gated, separately from the feature being enabled: a user needs the use_cli_terminal permission. Without it the WebSocket closes with “Access denied”, which is a different failure from the feature being off.
The admin CLI Workspaces tab at /pulse/cli-workspaces lists running workspace Pods with restart and cancel, and is gated by the view_sandboxes permission.

The first terminal a user opens creates their workspace Pod and pulls the ~700 MB workspace image, which can take up to a minute. Subsequent attaches are immediate.

Offboarding: the step everyone misses

Revoking the IdP group removes access. It does not reclaim disk.

Workspace volumes are never auto-reclaimed, and lifetime users — not concurrent users — is what grows your disk. Each person who has ever attached holds two durable PVCs (10 GiB each by default) for their home and work directories. Reaping an idle workspace returns CPU and memory; it returns no disk, because the volume is what makes a user’s saved conversations survive the reap. A deployment with 8 people using it and 60 people who have tried it is carrying 120 volumes.Nothing reclaims them automatically and that is deliberate: an “oldest unused” policy would delete the work of whoever is on holiday. Retirement is an operator action tied to a person leaving.

Both of a user’s claims carry the ownership metadata needed to find them — genesis.computing/owner-id as a readable annotation, and genesis.computing/workspace-owner as the selectable digest, because a label value cannot hold an email address:

# who owns what
kubectl -n $WS_NS get pvc \
  -o custom-columns=NAME:.metadata.name,OWNER:'.metadata.annotations.genesis\.computing/owner-id'

# find one person's digest, then retire both of their volumes -- AFTER their Pod is gone
kubectl -n $WS_NS get pvc -o json \
  | jq -r '.items[] | select(.metadata.annotations["genesis.computing/owner-id"]=="<email>")
           | .metadata.labels["genesis.computing/workspace-owner"]' | sort -u
kubectl -n $WS_NS delete pvc -l genesis.computing/workspace-owner=<digest>

Cancel the user’s workspace from the admin CLI Workspaces tab (or delete the Pod) first: deleting a claim while its Pod still exists leaves the claim Terminating until the Pod goes.

Gate — offboarding is complete when: the user’s IdP group membership is removed; kubectl -n $WS_NS get pod shows no workspace for them; and kubectl -n $WS_NS get pvc returns zero claims annotated with their address. This is destructive and unrecoverable — it is that person’s saved conversations — so confirm the address in the annotation before deleting, not the digest.

Scaling

The capacity formula

The backend is fixed; workspaces scale with people.

vCPU  = 4  + 0.5 × Nconcurrent
RAM   = 16 + 4   × Nconcurrent   (Gi reserved; each may grow to 8Gi)
disk  = 50 + 20  × Nlifetime     (Gi; 2 × 10Gi PVCs per person who has EVER attached)

Nconcurrent is bounded by cliTerminal.maxWorkspaces (default 25) and Nlifetime by cliTerminal.maxLifetimeUsers (default 50). At those defaults the feature asks for 12.5 CPU and 100 GiB reserved, bursting to 200 GiB, on top of the backend’s own 4 CPU / 16 GiB — and up to 1000 GiB of durable disk at the claim ceiling. A count quota reserves nothing, so the disk ceiling costs only what people actually claim. Three facts that make real deployments size differently than expected:

A workspace outlives the connection. Disconnecting does not delete the Pod; the idle reaper does, after cli_terminal.idle_reap_seconds (default 1 hour). Size for the peak number of workspaces that exist, not for the number attached right now. Shortening the reap interval returns capacity sooner and never costs a user work.
One user is one Pod, however many agents they run. Up to cli_terminal.max_sessions_per_workspace (default 8) sessions share one container’s CPU and memory. More sessions per user is cheaper than more users — and more likely to hit the memory ceiling, which OOM-kills the container and takes every session that user has open.
Reserved is not used. A cluster sitting at 5% observed CPU can correctly refuse to schedule a workspace. Compare requests against allocatable minus committed requests, never against kubectl top.

Raising the cap

helm upgrade $RELEASE <chart-ref> --version <chart-version> -n $NS -f my-values.yaml \
  --set cliTerminal.maxWorkspaces=40 \
  --set cliTerminal.maxLifetimeUsers=100      # must exceed maxWorkspaces or the render fails

Raise it in the values file, never in Advanced Settings. maxWorkspaces renders the namespace’s count/pods ResourceQuota and is written to the backend as GENESIS_CLI_TERMINAL_MAX_WORKSPACES. That env var is only a default, and the field is env_leaf, so an admin raising the cap in the panel outranks it immediately while the quota stays where it was rendered. The application then admits a workspace the API server refuses with a bare 403 — and the quota is the authority. Lowering it in the panel is harmless; raising it is what bites.

Then confirm the cluster can actually supply the new number, which is a different question from whether the quota permits it:

kubectl -n $WS_NS get resourcequota -o yaml | grep -A6 'hard:'
kubectl describe nodes | grep -A6 'Allocated resources'

Node auto-upgrade will interrupt sessions

A managed node pool upgrade drains and replaces nodes. Workspace Pods are evicted with them: running agent sessions are interrupted, and every volume survives, so users resume on their next attach on a new Pod. Treat node auto-upgrade as a maintenance window, not as a transparent operation. Set the maintenance window to your users’ off-hours, announce it, and prefer draining workspaces yourself beforehand so the interruption happens when you chose it:

cluster/scripts/drain-workspaces.sh $RELEASE      # stops agents, keeps every volume

A workspace Pod is pinned to its volume’s zone. The home PVC is ReadWriteOnce and, on zonal storage, bound to one zone — so a regional cluster with plenty of free capacity in other zones leaves a workspace Pending indefinitely with didn't match PersistentVolume's node affinity. Shrinking the workspace size does not fix it; the constraint is the zone. Capacity has to exist where that user’s volume already lives.

Health and verification

Endpoints

Endpoint	Returns	Token needed	Use it for
`/api/health/live`	`200` whenever the process is alive, regardless of readiness	No	Liveness probe. Never use it as “Genesis is working”.
`/api/health`	`503` with progress detail while booting, `200` once ready. Carries `status`, `ready`, `degraded`, `degraded_reason`, `llm_configured`, `llm_validation_error`, `uptime_seconds` and the version	No	Readiness and startup probes, and the first thing to read in an incident — `degraded_reason` and `llm_validation_error` frequently answer the question outright.
`/api/health/startup`	Granular boot progress — phase, percentage, current step. Always `200`	No	”It has been eight minutes, is it stuck or still working?”
`/api/healthcheck`	`200` when ready, `503` while starting	No	The simple external check.
`/api/health/slots`	Agent slot usage, PII-free	Yes	Diagnosing “Genesis is slow / not answering” — whether slots are saturated.
`/api/health/detailed`	System metrics and error tracking	Yes, plus `ACCESS_CONFIG_PANEL`	Deeper triage. There is no redacted variant; without the permission it is `403`.

First boot legitimately takes up to 10 minutes. The startup probe allows failureThreshold: 60 × periodSeconds: 10, because the first boot builds the database and warms sessions. /api/health/startup tells you whether that time is being used or wasted — check it before restarting a pod that is simply still working.

The token-free endpoints are matched exactly — no prefix matching, and GET/HEAD only — so /api/health/detailed authenticates even though /api/health does not. That matters behind an auth_request-mode ingress: an ingress controller’s auth annotation applies to the whole Ingress, so an external probe gets a login redirect rather than a health answer. The chart’s ingress.authExemptHealth block renders a second Ingress carrying only those exact probe paths, listed with and without a trailing slash, so health checks read server liveness instead of the proxy’s login page.

Chat working does not mean embeddings working

This is the failure that costs the most trust, because nothing looks broken. Chat and embeddings are separate routes, frequently separate endpoints, and sometimes separate credentials. When embeddings fail, Genesis keeps answering: semantic search silently falls back to graph-only lexical matching, so queries still return results — worse ones — and the data-context tool reports index_status: ready_with_warnings rather than an error.

ready_with_warnings is not an embedding verdict. An install whose embeddings are completely dead still answers every query as ready_with_warnings from the lexical fallback. It tells you a query had warnings; it does not tell you semantic search is working. Use the two surfaces below instead.

Both are present-tense: they report whether embedding works now, not whether it ever did. 1. The AI Providers page, at /config/ai-providers/ (embeddings tab ?tab=embeddings), shows an embedding status chip with its reason. It is fed by GET /api/config/ai-providers/embedding-health, deliberately as a separate request from the page overview, so a slow or absent indexing subprocess delays the status alone rather than the whole page. The chip does not poll — it is a one-shot read with a 30-second cache, so reload the page to re-check rather than watching it. 2. GET_INDEX_STATS, which is an action on the lineage tool, not a REST endpoint. Over HTTP:

curl -sS -X POST https://<host>/api/tool_call \
  -H "Authorization: Bearer <token>" -H 'Content-Type: application/json' \
  -d '{"func":"lineage_tools_lineage_action","func_args":{"action":"GET_INDEX_STATS"}}'

It returns embedding_provider, embedding_model, embedding_status and embedding_status_reason from one snapshot, so the verdict always describes the identity printed beside it — alongside index counts. It costs nothing: it reads recorded outcomes and makes no provider call. All four embedding fields are present on every successful response, including on an install with no embedding client at all — that case reports unconfigured. Alert on the value, and treat an absent embedding_status as a fault in the check itself rather than as a pass: a response without one came from a Twin too old to carry the field, and says nothing about whether embedding works.

Value	Means	What to do
`ok`	This identity has embedded successfully and there is not yet conclusive evidence it has stopped.	Read the reason anyway: `ok` can be reporting failures since the last success, and it names the count and the last error.
`unverified`	Nothing attempted yet, or too few failures to conclude anything.	Not an alarm. Force a real call with the connection test below.
`systemic_failure`	This identity cannot currently embed — semantic search is dead.	Page-worthy. See the catalogue entry.
`unconfigured`	There is no embedding client, so there is no identity to judge and `embedding_provider` / `embedding_model` are empty. Semantic search cannot work: every result is graph-only lexical matching.	Configure an embedding provider and model. Expected on a fresh install, an alarm anywhere it is not — and never to be read as a milder `unverified`, which means something is configured but unproven.
`unavailable` (the REST route only)	No verdict was obtained — the indexing subprocess is unreachable or disabled, or the index is not initialised.	An explicit unknown, neither green nor an alarm. Distinct from `unconfigured`: that one is an answer, this one is the absence of one. A surface that reads as working because the check failed is the same defect as one that reads as working while broken.

The reason is shown for every value, ok included. 3. Force a real embedding call. The two surfaces above read recorded evidence; when the verdict is unverified or you have just rotated a credential, make the deployment actually embed something:

curl -sS -X POST https://<host>/api/config/llm-providers/embedding-config/test \
  -H "Authorization: Bearer <token>" -H 'Content-Type: application/json' \
  -d '{}'
# expect: {"success": true, ..., "dimensions": <n>}

It issues a real embedding request against the configured provider and model and returns the vector dimension it got back. "success": false carries the provider’s own error text, which is usually the whole diagnosis. This is the only surface that makes a provider call on demand, and it needs ACCESS_CONFIG_PANEL. 4. The boot-time smoke test leaves a greppable line in the logs, so an incident can be dated to a restart:

kubectl -n $NS logs $RELEASE-0 -c genesis | grep -i 'failed startup smoke test'
# a line here means: "Document indexing will fail until this is fixed."

Gate — the deployment is healthy when all five hold. Anything less is a partial answer:

curl -sS -o /dev/null -w '%{http_code}\n' https://<host>/api/healthcheck → 200
kubectl -n $NS get pods and kubectl -n $WS_NS get pods → nothing in ImagePullBackOff, CrashLoopBackOff or Pending
A signed-in user sends a chat message and gets a reply
The embedding connection test returns "success": true with a dimension count — not merely a chip that is not red
A user opens a terminal and gets a shell

The chart’s own validators

The chart ships render-time validators, and they are packaged inside the chart tarball, so an operator who never sees the source can still run them:

helm pull <chart-ref> --version <chart-version> --untar --untardir /tmp/genesis-chart
bash /tmp/genesis-chart/genesis/validate-cli-terminal.sh
bash /tmp/genesis-chart/genesis/validate-retention.sh

Expected final lines, and both exit 0:

In-pod coding agent mode renders correctly.
Retention contract holds in the rendered manifest.

These render the chart locally and never contact a cluster — KUBECONFIG is pinned to /dev/null inside them. They answer “does this chart version still produce a correct, data-safe manifest”, which is exactly the question to ask before an upgrade and after a rollback. They cannot tell you anything about your running deployment; the gate above does that. Invoke them with bash — files inside a Helm tarball are not extracted executable.

Teardown without data loss

The order is not a style preference. Each step removes something the next step needs.

cluster/scripts/… is not in the chart tarball. Unlike the validators, these scripts live in the Genesis repository, so a deployment installed only from the OCI chart does not have them — ask Genesis for them before you need them, not during a teardown.Do not improvise a kubectl delete pod equivalent. The ownership checks described below are the entire point of these scripts: on a cluster running more than one Genesis instance, acting on the chart’s default namespace means stopping another instance’s users mid-turn and, with the wrong flag, destroying their saved conversations.

Drain the workspaces first — while the release is still up

cluster/scripts/drain-workspaces.sh $RELEASE

This deletes the workspace Pods and nothing else: every PVC and the workspace namespace stay, so each user resumes on their next attach.It must run first because afterwards nothing can do it: the reaper goes with the backend, the RoleBinding that let it act goes with the release, and the release-to-namespace mapping goes with helm get values. It is a separate script from teardown.sh precisely so it is runnable on a release that is staying up — which is also what you run before setting cliTerminal.enabled=false.

Uninstall the release

helm uninstall $RELEASE -n $NS

This leaves behind, by design: the data PVC, the Genesis namespace, the workspace namespace and every workspace PVC, the ServiceAccounts, and the per-instance sandbox RoleBinding and ClusterRoleBinding. All carry helm.sh/resource-policy: keep.

Revoke the retained RBAC

cluster/scripts/cleanup-retained-rbac.sh $INSTANCE_NS

$INSTANCE_NS, not $NS. The argument is the instance namespace — the identity the chart wrote into the label — and on a release that sets values.namespace that is a different value from the Helm release namespace. Passing the Helm namespace there searches for an owner that does not exist: the script reports nothing to delete, the bindings stay, and the final gate below inspects the wrong identity and agrees. By this point helm uninstall has already removed the release metadata, so $INSTANCE_NS must have been captured before the uninstall — which is why it is read in Pin the identity and not here.

Until this runs, those two bindings keep granting sandbox pod management and namespace read to the instance’s ServiceAccount. They live outside the Genesis namespace — the ClusterRoleBinding is cluster-scoped and the RoleBinding is in the sandbox namespace — so deleting the Genesis namespace does not reach them. An operator who tore the instance down has not revoked what they believe they revoked.Its argument is the Genesis instance namespace, and it selects on the genesiscomputing.ai/instance-namespace label rather than on a Helm release name, so it still works after the release record is gone. It deliberately leaves the shared, fixed-name sandbox objects alone: other instances bind to them.

Retire the data, deliberately and last

kubectl -n $NS get pvc                     # look at what you are about to destroy
kubectl -n $NS delete pvc <the-claim-name-you-just-read>
kubectl delete namespace $NS

Whether the underlying disk is destroyed depends on the StorageClass reclaim policy (Delete or Retain) — check it before assuming either.

Retire the workspace namespace, separately and explicitly

The workspace namespace holds every user’s home volume and therefore their saved conversations, so nothing removes it implicitly — not helm uninstall, and not setting cliTerminal.enabled=false. Removal is its own command, and it asks first unless you pass --force:

cluster/scripts/teardown.sh <name> --delete-workspaces

To retire one departed user instead of everyone, use the PVC commands in Offboarding.

Ownership is by instance label, never by release name

Both cleanup scripts refuse to act on a workspace namespace they cannot show belongs to this instance, and the reason is worth understanding before you reach for a flag to override it.

The live namespace says	What happens	This is
`genesiscomputing.ai/instance-namespace` = this instance	Drained, and deleted when asked — even if `cliTerminal.enabled` is now `false`, because disabling leaves the Pods running and teardown is the last thing that can stop them	Ownership-verified
No such label	Acted on only when the release currently enables the feature, or when you name the namespace with `--workspace-ns`	Operator-trusted, not verified. A namespace older than the chart version that writes the label, or one you brought with `createNamespace: false`
The label names a different instance, or ownership cannot be read at all	Refused, with the reason	Never automatic, and `--workspace-ns` does not override it

The identity is the instance namespace, never the Helm release namespace. On an installation that sets values.namespace, those are two different values, and comparing a workspace namespace’s owner against the release namespace refuses that instance its own workspaces. Both scripts resolve the identity from the release and carry it into every command they print, because after the uninstall nothing can resolve it again.drain-workspaces.sh --instance-ns <ns> supplies it by hand once no release is left to read. While the release is readable, that flag may only confirm what the release says — a different value is refused, because an identity accepted from the command line would let a live teardown drain another instance’s workspaces by naming that instance. teardown.sh takes no such flag at all.

Gate — teardown is complete when all four hold. Note the selector uses $INSTANCE_NS: on a release that sets values.namespace, checking $NS here asks about an identity that never existed and passes while every binding is still standing.

helm list -A | grep -i "$RELEASE"
# expect: no output

kubectl get clusterrolebinding -l genesiscomputing.ai/instance-namespace=$INSTANCE_NS
kubectl get rolebinding -A -l genesiscomputing.ai/instance-namespace=$INSTANCE_NS
# expect: no resources found, from both

kubectl get pvc -A | grep genesis
# expect: only claims you have decided to keep

A PVC you meant to destroy that is still listed is a disk still being billed; a binding still listed is access you believe you revoked and did not.

Incident triage

This catalogue is the single home for Genesis failure modes. The install runbooks link here rather than restating it, so there is one description to keep current. Each row is symptom → cause → fix; read the symptom column first and do not skip to the fix, because several of these present identically and are cured by opposite actions.

The cluster never comes up

Symptom: 0 nodes out of 1 have registered, or kube-system pods stuck ContainerCreating with CSINode at 0 drivers and PVCs unbound

Cause. The nodes cannot reach the container registry, so they cannot pull Kubernetes’ own system images (pause, netd, pdcsi, kube-proxy). The node then never registers. Nothing in the Genesis chart or your values file is at fault — and the symptom wears different costumes on different node modes, which is why it reads as a storage, chart or quota problem and is none of them.Verify — the first minute after the cluster object exists. This is the highest value minute in the whole install:

kubectl get pods -n kube-system

Every system pod should reach Running within roughly a minute. A cluster creation that timed out still leaves a usable kubeconfig behind, so this works on that wreck too — you do not have to delete and retry to diagnose it.Fix. Grant node egress to the registry before creating the cluster: a NAT gateway, a private-access path to the registry, or a tag-scoped internet route. The exact mechanism is per-cloud — see Per-cloud specifics and, for GCP, the detailed remedies in the GKE install runbook.And a route is not a permission. A higher-priority egress deny — a VPC firewall rule, or a policy inherited from a folder or organisation — blocks the pull with a perfect route in place, and the symptom is identical. Read the effective rules, not the network’s own.

Symptom: the backend pod sits Pending on a volume that never attaches, mid-install

Cause. A preemptible / Spot node was reclaimed during the install, churning the node the backend was scheduled on and wedging PVC binding. It reads as a storage fault and is a node one.Fix. Bring the cluster up on on-demand nodes and finish the install. Spot is fine for read-only workloads afterwards.

The deployment is up but something is wrong

Symptom: NetworkPolicy applied cleanly and enforces nothing

Cause. Kubernetes accepts a NetworkPolicy object whether or not your CNI implements it. On a non-enforcing cluster the policy is inert, workspace egress stays wide open, and there is no error on either side — the failure looks exactly like working containment.This is measured, not assumed. From the chart’s own values.yaml:

It is only as good as your CNI. Kubernetes accepts the object whether or not the CNI enforces it. Measured on stock EKS 1.34 with the default VPC CNI: applied cleanly, enforced nothing, workspace reached the public internet. GKE blocked it. Verify.

Verify on the cluster, never from the setting — and run the control first. A blocked connection on its own proves nothing: in a locked-down enterprise VPC the cluster may have no egress to that address at all, so a probe that simply fails tells you the destination was unreachable, not that the policy did the blocking. Two Pods, identical except for the one label the policy selects on:

# 1. CONTROL — no workspace label, so the policy does not select it.
#    This measures whether the cluster can reach the destination at all.
kubectl -n $WS_NS run probe-control --rm --attach --restart=Never \
  --image=busybox --command -- \
  sh -c 'if nc -z -w 5 1.1.1.1 443; then echo CONTROL-REACHED; else echo CONTROL-BLOCKED; fi'

# 2. SUBJECT — identical, but carrying the label the policy selects.
kubectl -n $WS_NS run probe-subject --rm --attach --restart=Never \
  --image=busybox --labels=app=genesis-cli-terminal --command -- \
  sh -c 'if nc -z -w 5 1.1.1.1 443; then echo SUBJECT-REACHED; else echo SUBJECT-BLOCKED; fi'

--attach blocks until each container exits and --rm removes the Pod, so there is nothing to clean up and no half-started Pod to read at the wrong moment.

Control	Subject	Verdict
`CONTROL-REACHED`	`SUBJECT-BLOCKED`	Enforced. The label is the only difference between the two Pods, so the policy is what blocked it.
`CONTROL-REACHED`	`SUBJECT-REACHED`	Not enforced. The policy is decorative — your CNI is not implementing it.
`CONTROL-BLOCKED`	anything	Inconclusive. The cluster cannot reach that destination regardless of the policy, so this probe cannot see the policy at all. Re-run against a destination the control can reach, and do not record a result until it does.

Two readings that look like a pass and are not. CONTROL-BLOCKED with SUBJECT-BLOCKED is the trap: it looks exactly like containment and is the normal state of a cluster with no internet egress, where the policy may be entirely inert. And silence is never a pass — a Pod that was not scheduled or could not pull busybox prints nothing. Insist on one of the four words from each command; if one is missing, check kubectl -n $WS_NS get events --sort-by=.lastTimestamp | tail and run it again.

Fix. Turn on your CNI’s policy engine — the mechanism is per-cloud, and on GKE it is creation-time only for Dataplane V2, with a Calico retrofit available on an existing cluster. See Per-cloud specifics.Before you treat this as an outage, check whether you needed the policy at all: cliTerminal.networkPolicy.enabled is off by default, and that is the intended posture. See What the network posture actually is.

Symptom: the terminal returns 403 or 'disabled' although the chart rendered everything

Three different failures share this description. The WebSocket close reason tells you which, so read it before changing anything — they are cured by opposite actions.Where to read it. The server closes the socket before accepting it, with close code 1008 and the reason as text. Take it from the browser’s devtools console or Network → WS on the failed attach, or from the backend log, which records the same string:

kubectl -n $NS logs $RELEASE-0 -c genesis | grep 'CLI terminal: refusing connection'

Close reason	Cause	Fix
”CLI terminal access is disabled on this deployment”	The effective `cli_terminal.enabled` is false. Either the chart flag is off, or an admin stored `false` in Advanced Settings on a build or a path where the chart does not force-lock it.	Set `cliTerminal.enabled=true` in the values file and upgrade. That writes both `GENESIS_CLI_TERMINAL_ENABLED` and the `GENESIS__CLI_TERMINAL__ENABLED` force-lock, so a stored panel value can no longer contradict it. Confirm in Config Sources, which attributes the effective value to `env_override`.
”Access denied”	The user lacks the `use_cli_terminal` permission. This is Genesis RBAC, not Kubernetes RBAC — nothing about the cluster is wrong.	Grant the permission, or the role carrying it, to that user’s IdP group.
The socket attaches, then workspace creation fails with a Kubernetes 403	The namespace or RBAC is missing, or the backend is asking in a different namespace from the one the chart granted.	Check that `cli_terminal.k8s_namespace` equals the namespace the RoleBinding is in. This is the disagreement `cliTerminal.enabled=true` exists to prevent: it pins the namespace to the objects it just rendered.

kubectl auth can-i create pods -n $WS_NS \
  --as=system:serviceaccount:$NS:<the-backend-service-account>
# expect: yes
kubectl -n $WS_NS get role,rolebinding,resourcequota

The historical trap. Earlier deployments applied the terminal platform with a separate kubectl apply of a manifest. That is no longer the path: the chart renders the namespace, RBAC and quota itself. A hand-applied copy now diverges silently from the chart’s — only one of the two ever gets updated.

Symptom: a Kubernetes 403 on workspace creation for a user who could attach yesterday

Cause. The quota, not RBAC. cliTerminal.maxWorkspaces renders the namespace’s count/pods ResourceQuota and seeds the backend’s own cap — but the backend’s cap is only a default, and an admin who raised it in Advanced Settings outranks it immediately while the quota stays where it was rendered. The application then admits a workspace the API server refuses.

kubectl -n $WS_NS get resourcequota -o yaml | grep -A6 'hard:'
kubectl -n $WS_NS get pods | wc -l

Fix. Raise it in the values file and re-render — not in the panel. See Raising the cap. The quota is the authority.

Symptom: every workspace is ImagePullBackOff, for a correctly configured operator

Cause. One of two, and they are distinguishable:

The tag is not in the mirror. image.tag must be a tag that actually exists in the registry your nodes pull from. The chart’s default falls back to the chart’s appVersion, which is frequently not a tag present in the distribution registry — only a named channel tag is. An upgrade that mirrors the backend image and forgets the workspace image produces exactly this: a working console, no terminals.
The node’s identity cannot pull. A Kubernetes node pulls as its own service account or instance identity, not as the human who ran helm. A grant made to your corporate directory does not cover the node.

kubectl -n $WS_NS describe pod <workspace-pod> | tail -25
# read the Events: "manifest unknown" is cause 1; "denied"/"unauthorized" is cause 2
crane manifest <registry>/<repo>/genesis-sandbox-runner:<tag> > /dev/null && echo TAG-EXISTS

Fix. For (1), mirror the tag, or point GENESIS_CLI_TERMINAL_IMAGE at a tag that exists. For (2), grant the node identity read on the registry — the mechanism is per-cloud, see Per-cloud specifics.

Symptom: chat works, but search results got worse and nobody can say when

Cause. The embedding endpoint is unroutable or not served. Genesis does not fail the query: semantic search falls back to graph-only lexical matching and the index reports ready_with_warnings, so the install keeps answering and the degradation is invisible from chat.Verify the embedding path independently of chat — they are separate routes and frequently separate endpoints and credentials. In order:

POST /api/config/llm-providers/embedding-config/test — forces a real embedding call and returns the dimension, or the provider’s own error text.
Read embedding_status and embedding_status_reason from the AI Providers page or GET_INDEX_STATS for the recorded history behind that one call.
kubectl -n $NS logs $RELEASE-0 -c genesis | grep -i 'failed startup smoke test' to date the failure to a restart.

All three are in Health and verification.Fix, by what the reason says:

Reason names	Cause	Fix
An unroutable model id	The model id has no provider prefix, so nothing can dispatch it. Conclusive before any call is made.	Configure a provider-qualified model id, or set the embedding provider explicitly.
Repeated failures, never a success	The configuration is wrong — wrong endpoint name, or a credential that was never valid for it.	Confirm the serving endpoint exists under exactly that name on your LLM host, and that the credential is scoped to it.
Worked, then stopped	An expired credential, a withdrawn serving endpoint, or a network path that changed. The bar for this verdict is deliberately higher, because a proven identity is known routable.	Rotate or restore the credential; confirm the endpoint is still served.

Recovery is observable: a single success clears the failure record and the status returns to ok without a restart.

This is a Professional Services responsibility at install time, not a customer-infrastructure one. The embedding serving endpoint must exist and be reachable, and it must be verified separately from chat. An install signed off on “the bot replies” ships with semantic search dead and nobody knowing for weeks.

Symptom: 'reset to defaults' reported success and the feature is still live

Cause. A version-gated defect. On older images, Set default on a cli_terminal.* or sandbox_runner.* row deleted the stored value and reported success, while the environment variable the runtime actually reads kept the value the panel had bridged into it. The panel showed the feature off; the terminal stayed reachable until the pod restarted.Fix. Upgrade the image — current builds restore the deployment’s own pre-panel value on reset, and leave a GENESIS__… force-lock winning where one is set. Until then, never trust the panel after a reset: attempt an attach, or restart the pod, and read the real state.

Symptom: workspaces sit Pending and every attach fails — the feature looks broken

Cause. Starved, not broken. Either the cluster cannot supply the requests, or the user’s volume pins them to a zone with no room, or the node pool cannot grow.

kubectl -n $WS_NS describe pod <pending-pod> | tail -20

Message	Cause	Fix
`Insufficient cpu` / `Insufficient memory`	Genuinely no schedulable headroom. Compare requests against allocatable minus committed, never against observed usage.	Add capacity, or lower `cliTerminal.maxWorkspaces` to what the cluster holds.
`didn't match PersistentVolume's node affinity`	The user’s `ReadWriteOnce` volume is bound to one zone and that zone is full. Shrinking the workspace does not help.	Free capacity in that zone, or accept losing that user’s volume.
`FailedScaleUp: ... quota exceeded`	The cloud account’s quota refuses a new node — frequently the disk quota, not CPU. Observed at 11/200 CPU while disk sat at 418/500 and a new node needed ~100 GB.	Raise the quota that is actually exhausted. Check disk, not just CPU.

Symptom: the upgrade 'worked' but the old behaviour persists

Cause. The pod is Ready on a cached layer under the same moving tag, or the rollout never actually replaced the pod.

kubectl -n $NS get pod $RELEASE-0 \
  -o jsonpath='{.status.containerStatuses[?(@.name=="genesis")].imageID}{"\n"}'
crane digest <registry>/<repo>/genesis:<tag>
# these two must be the same sha256
kubectl -n $NS get pod $RELEASE-0 -o jsonpath='{.status.startTime}{"\n"}'

Fix. Deploy by digest, or mirror the build to an immutable tag of your own and deploy that. A moving tag plus IfNotPresent is the combination that produces this.

Symptom: helm upgrade fails with 'invalid ownership metadata'

Cause. Helm refusing to adopt an object it did not create — typically the workspace namespace, applied by hand on an earlier deployment. Not a chart fault.

Error: rendered manifests contain a resource that already exists.
Unable to continue with update: Namespace "genesis-cli" ... exists and cannot be
imported into the current release: invalid ownership metadata

Fix. Set cliTerminal.createNamespace=false and leave the namespace where it is, then delete only the Role, RoleBinding, ResourceQuota and NetworkPolicy so the chart can render its own. Do not delete the namespace — that cascades to every user’s home volume.A namespace this chart created re-adopts cleanly, because retention stops the delete without stripping the ownership annotations Helm wrote. Only a hand-created one is rejected.

What the network posture actually is

State this accurately when asked, because the reassuring version is wrong and the difference is a security finding:

By default, a workspace Pod has exactly the network access any other Pod in the cluster has — the same access the Genesis backend has, because the Genesis namespace carries no NetworkPolicy either. That is deliberate: these are development workspaces, and installing packages and cloning repositories is the point of them.
The workspace egress lockdown is opt-in (cliTerminal.networkPolicy.enabled=true), confining workspaces to DNS plus the Genesis backend — at the cost of git, gh, npm and pip timing out.
It is only as good as the CNI, and an unenforcing CNI reports success. Verify it on the cluster before describing it to a security reviewer.

Do not describe agents as locked to DNS and the backend by default. They are not.

Per-cloud specifics

The Genesis release is identical everywhere; the substrate underneath is not. Everything above is cloud-agnostic — these are the rows that differ.

How to read this table. A cell marked ✓ is established by Genesis documentation or measured on a real cluster, and the linked page carries the detail. A cell marked ◦ names the standard mechanism for that platform, which Genesis does not document and you must confirm against your cloud’s own documentation and with your platform team. Do not carry a ◦ cell into a design review as a Genesis requirement.

Concern	GCP / GKE	AWS / EKS	Azure / AKS
Node egress to the registry	✓ Cloud NAT, or Private Google Access reaching `.googleapis.com`, `.pkg.dev` and `.gcr.io`, or a tag-scoped `0.0.0.0/0` route on the node tag. A Cloud NAT still needs a `default-internet-gateway` route and* must cover the node subnet’s ranges — a region match alone proves nothing. And a route is not a permission: a higher-priority egress deny produces an identical symptom.	◦ NAT gateway on the node subnets, or VPC endpoints for the registry and its backing object store.	◦ NAT gateway or the cluster’s outbound type. The Genesis doc states only the requirement — “cluster nodes must pull container images from the registry URL provided by Genesis” — not a mechanism.
NetworkPolicy enforcement	✓ Dataplane V2 (`--enable-dataplane-v2`), creation-time only. An existing Standard cluster retrofits the Calico engine with `gcloud container clusters update … --enable-network-policy` (rolling node recreation). Autopilot always enforces. Verify with `--format='value(networkConfig.datapathProvider)'`, want `ADVANCED_DATAPATH`.	✓ (measured negative) The default VPC CNI enforces nothing — measured on stock EKS 1.34: the chart’s policy applied cleanly and a workspace-labelled pod reached the public internet. ◦ Enabling enforcement means turning it on in the CNI addon, or running Calico or Cilium.	◦ A policy engine must be chosen at cluster creation — the chart’s own AKS recipe uses `az aks create --network-plugin azure --network-policy azure`. It cannot be added to a cluster created without one.
Pod identity for cloud secrets	◦ Workload Identity on the cluster, with `secretmanager.secretAccessor`, for `gcp_*` secret references. The GKE install runbook does not cover setting it up.	✓ IRSA. `serviceAccount.create: true` plus `serviceAccount.annotations` carrying `eks.amazonaws.com/role-arn` — and the same annotation shape on `sandbox.podServiceAccount.annotations` for sandbox pods. `create: true` is required so the sandbox RBAC has an account to bind. Used for Secrets Manager resolution, `GENESIS_MASTER_KEY_REF` and custom CA certs, with no init container or CSI driver.	◦ Workload Identity / managed identity. The Genesis doc describes it in two sentences of prose and gives no annotation key or values example.
Registry and node pull grant	✓ Artifact Registry. The node’s own service account needs `roles/artifactregistry.reader` on the repository — a grant to your corporate directory does not cover the node. No pull secret.	✓ ECR, first-class in the chart: `ecr.enabled: true` with `ecr.registry` and an `ecr.token` passed at install time creates the pull secret and overrides `image.repository`. The token is short-lived, so deploy scripts refresh it every run.	✓ Also ECR, not ACR — Azure deployments pull from the Genesis registry through a `kubectl create secret docker-registry` secret named in `imagePullSecrets`.
Default StorageClass	✓ `standard-rwo` (pd-balanced — counts against SSD quota, which is the quota that usually runs out first)	✓ cluster default, typically `gp2` or `gp3`	✓ cluster default, typically `managed-csi` or `managed-premium`
Snapshot mechanism	◦ Persistent Disk snapshots (`gcloud compute disks snapshot <disk> --zone <zone>`). There is no GCP backup doc — this row is generic, and the restore drill above is the part that matters.	✓ EBS snapshots, three worked paths — Velero, AWS Backup for EKS, or a manual `aws ec2 create-snapshot`. See AWS backup and recovery.	✓ Kubernetes `VolumeSnapshot` with the Azure Disk CSI driver, or Azure Backup for AKS. See Azure backup and snapshot options — options and trade-offs rather than commands.
Quota names to check before scaling	✓ `CPUS`, `SSD_TOTAL_GB`, and `IN_USE_ADDRESSES` if you use a LoadBalancer or Ingress. Check disk, not just CPU — node scale-up has been observed failing on disk quota (418/500) with CPU at 11/200.	◦ vCPU limits per instance family, EBS volume and snapshot limits, Elastic IP limits. Not named in the Genesis docs.	◦ Regional vCPU quota per family, managed disk limits, public IP limits. Not named in the Genesis docs.

Whatever the cloud, the VolumeSnapshot API is the portable option where your cluster has a CSI driver and a snapshot controller installed: one procedure everywhere, at the cost of a cluster prerequisite. The cloud-native command always works.

Two facts hold on every cloud, and neither is in the snapshot commands: run the checkpoint hook before the snapshot, and back the master key up separately — it is not on the volume. Use the per-cloud pages for the snapshot mechanics, and the restore drill for the part none of them covers: proving the backup actually restores.

​Pin the identity before you touch anything

​Upgrade

​1. Pull the new image into your mirror

​2. Run the upgrade

​3. Verify by content, not by “Ready”

​What “version-gated” means

​Rollback

​Roll the release back

​What survives a rollback, and what does not

​Backup and restore

​What is on the volume

​Snapshot the volume

​The restore drill

​Secrets

​The master key

​LLM and database credentials

​Configuration precedence

​Which fields the chart pins

​What applies live and what needs a restart

​Reset to defaults

​Users: onboarding and offboarding

​Onboarding

​Offboarding: the step everyone misses

​Scaling

​The capacity formula

​Raising the cap

​Node auto-upgrade will interrupt sessions

​Health and verification

​Endpoints

​Chat working does not mean embeddings working

​The chart’s own validators

​Teardown without data loss

​Ownership is by instance label, never by release name

​Incident triage

​The cluster never comes up

​The deployment is up but something is wrong

​What the network posture actually is

​Per-cloud specifics

Pin the identity before you touch anything

Upgrade

1. Pull the new image into your mirror

2. Run the upgrade

3. Verify by content, not by “Ready”

What “version-gated” means

Rollback

Roll the release back

What survives a rollback, and what does not

Backup and restore

What is on the volume

Snapshot the volume

The restore drill

Secrets

The master key

LLM and database credentials

Configuration precedence

Which fields the chart pins

What applies live and what needs a restart

Reset to defaults

Users: onboarding and offboarding

Onboarding

Offboarding: the step everyone misses

Scaling

The capacity formula

Raising the cap

Node auto-upgrade will interrupt sessions

Health and verification

Endpoints

Chat working does not mean embeddings working

The chart’s own validators

Teardown without data loss

Ownership is by instance label, never by release name

Incident triage

The cluster never comes up

The deployment is up but something is wrong

What the network posture actually is

Per-cloud specifics