Installing Genesis on GKE with Helm

Everything needed to stand up Genesis on your own Google Kubernetes Engine cluster with Helm — the web console and the on-demand sandbox terminal, reached through one URL. Self-contained: images and chart come from a registry you are already granted; no source access required.

Chart the chart version listed for your release (for example 0.3.104 · app v1.19.95, the GA release when this page was last revised)
Helm 3.8+ · Kubernetes 1.23+ · crane or Docker (only if you mirror images)
Backend 1 × StatefulSet (single replica)
Port 8080 — serves GUI, API, and the terminal WebSocket
Sandbox mode opt-in

The install runs in six phases, and each ends in a gate — an explicit list of checks that must pass before the next phase starts. Skipping a gate is how a one-hour install becomes a one-day one: every failure this runbook has seen in the field surfaces several phases after its cause.

For everything after the install — upgrade, rollback, backup and restore, secrets, user offboarding, scaling, teardown, and the full failure-mode catalogue — see the Genesis Operations Runbook. It is cloud-agnostic, and it is the page to hand the customer’s operations team at handover.

This is the full base-plus-sandbox-terminal install runbook. For the separate sandbox_runner.runtime: kubernetes per-thread runner (one Pod per selected chat thread), see Deploying the Genesis Sandbox Runner on GKE.

What you’re installing

Two container images, one Helm release, one reachable URL. Genesis runs as a single-replica StatefulSet — a FastAPI backend serving the React web console, the REST API, and the terminal WebSocket, all on one port (8080). There is no separate service or port for the terminal.

Component	Role	Image
Genesis backend	The server: console + API + terminal WS. One StatefulSet, PVC-backed SQLite.	`genesis-dist/genesis:<release tag>`, e.g. `v1.19.95`
Sandbox workspace (opt-in)	Per-user Pod the backend spins up on demand — Node, Python, git, tmux: a full dev shell. A separate image from the backend — see Coding sandbox (CLI Terminal) image.	`genesis-dist/genesis-sandbox-claude:latest`

When a user opens a terminal, the backend creates a workspace Pod for that user in a dedicated namespace, attaches a tmux session, and streams it over the same port. That is the on-demand sandbox mode — off by default, enabled in Phase 5.

Honest scope. The base install (Phases 1–4) is standard Helm. The in-pod coding agent mode (Phase 5) is opt-in and is enabled by one cliTerminal.enabled chart setting, which renders the namespace, RBAC and quota as part of the release.

Phase 1 — Preflight and grants

Run this before installing. Anything red here is a grant to request from your GCP admin — the checks are exact so nothing surfaces mid-deploy. Set your targets once; the checks reference them:

PROJECT=your-gcp-project
REGION=us-central1
ZONE=us-central1-a
CLUSTER=genesis
NETWORK=default          # the VPC the nodes sit on — change this for a custom VPC
NODE_TAG=$CLUSTER-node   # per-cluster, so one cluster's egress route never governs another
gcloud config set project $PROJECT

Use a Standard cluster. Standard is the recommended path for an enterprise GCP org. Autopilot is simpler when your org allows it, but its fully-managed node provisioning is frequently blocked by org policies (shielded-VM, no-external-IP, SA constraints) — the symptom is no nodes ever provision and pods sit pending with no clear error. Standard lets you control the node config to satisfy those policies, so it deploys on the first try.

If you do use Autopilot, three things it does not manage away. Compute Engine quota still applies — Autopilot nodes are ordinary VMs, and a cluster that fails minutes in with STATUS: ERROR usually names SSD_TOTAL_GB only in gcloud container clusters describe … --format='value(conditions)'. Requests are rewritten — Autopilot enforces a memory-to-CPU ratio and raises CPU to match (a workspace asking 500m/4Gi is admitted at roughly 616m) and adds ephemeral storage to every container, so budget the admitted figures, not the requested ones. Nodes are consolidated on Autopilot’s schedule, so expect occasional unscheduled backend restarts and slower first workspace launches while a node provisions.

Create a Standard cluster (already have one? just get-credentials below):

# --enable-dataplane-v2 is CREATION-TIME ONLY and is what makes any NetworkPolicy enforce
# --machine-type e2-standard-8 is 8 vCPU / 32Gi per node
# --tags=$NODE_TAG is what the registry-egress route below matches on
gcloud container clusters create $CLUSTER --zone $ZONE \
  --enable-dataplane-v2 \
  --machine-type e2-standard-8 \
  --tags=$NODE_TAG \
  --num-nodes 1 --enable-autoscaling --min-nodes 1 --max-nodes 5
# (regional --region creates num-nodes PER zone; --zone keeps it to one for a trial)

gcloud container clusters get-credentials $CLUSTER --zone $ZONE

Dataplane V2 is what makes a NetworkPolicy mean anything — and it is creation-time only. Kubernetes accepts a NetworkPolicy object whether or not the cluster runs a policy engine. Dataplane V2 is that engine. Without it, any policy you apply is silently ignored, with no error on either side — the failure looks exactly like working containment. GKE cannot gain Dataplane V2 after creation, so create the cluster with --enable-dataplane-v2 even if you have no immediate need for it.Already created a Standard cluster without it? You don’t have to recreate — enable the legacy Calico engine on the existing cluster (a rolling node recreation, no full rebuild):

gcloud container clusters update $CLUSTER --zone $ZONE --enable-network-policy

Be accurate about what this buys you, because the reassuring version is wrong. The workspace egress restriction is off by default (cliTerminal.networkPolicy.enabled: false). By default a workspace Pod — and a per-thread runner Pod — has exactly the network access any other Pod in this cluster has, which is the same access the Genesis backend has, because the Genesis namespace carries no NetworkPolicy either. That is deliberate: these are development workspaces, and installing packages and cloning repositories is the point of them.Turning the policy on confines workspaces to DNS plus the Genesis backend, at the cost of git, gh, npm and pip timing out. Do not tell a security reviewer that agents are confined to DNS and the backend by default. They are not. If you need that confinement, enable it in Phase 5 and then verify it on the cluster — the engine is the prerequisite, not the guarantee.

Nodes need outbound egress to the container registry — this is the one that costs a day. In a locked-down VPC with no 0.0.0.0/0 route, and registry hostnames resolving outside Private Google Access scope, a node cannot pull GKE’s own system images (pause, netd, pdcsi, kube-proxy). The node then never registers and cluster creation dies on:

0 nodes out of 1 have registered

On Autopilot the same cause wears a different costume — kube-system pods stuck in ContainerCreating, CSINode reporting 0 drivers, PVCs that never bind. It reproduces identically on Autopilot and Standard, and it presents as a confusing cascade rather than an obvious pull failure. Nothing in the chart or your values file is at fault.Grant the egress before you create the cluster. Two remedies to choose between, then one caveat that applies to whichever you pick:

A tag-scoped internet route — good for a sandbox, because it opens egress for the nodes only and nothing else. Both the route and the tag are named after $CLUSTER, so this route governs exactly one cluster’s nodes and deleting it can never revoke another cluster’s egress:
gcloud compute routes create $CLUSTER-registry-egress \ --network=$NETWORK \ --destination-range=0.0.0.0/0 --next-hop-gateway=default-internet-gateway \ --tags=$NODE_TAG --project=$PROJECT
--network is spelled out on purpose: omit it and gcloud silently uses the VPC literally named default, so on a custom VPC the route is created successfully in a network your nodes are not on and the egress does not change. The route matches only nodes carrying $NODE_TAG, which the clusters create recipe above applies — on an older cluster or a node-pools create of your own, the route matches no instance and changes nothing, so confirm the tag before blaming the route. This route is yours, it outlives the cluster, and you delete it. Because it is per-cluster, teardown is safe once that cluster is gone:
gcloud compute routes list --filter='destRange=0.0.0.0/0' \ --format='table(name,network,nextHopGateway,tags.list())' # confirm the nodes actually carry the tag the route matches gcloud container node-pools describe default-pool --cluster=$CLUSTER --zone=$ZONE \ --format='value(config.tags)' # after this cluster is gone — it governs no other cluster's nodes: gcloud compute routes delete $CLUSTER-registry-egress --project=$PROJECT
Read that listing on both axes. Route names are unique per project, not per network, so a route of the same name anywhere in the project is what will collide — which is why the filter is deliberately not narrowed to one VPC. For the egress question, only a row whose network column matches $NETWORK and whose tags include $NODE_TAG counts as coverage for this cluster. If your platform team already owns a shared egress route, reuse it instead: use the tag it matches on as NODE_TAG and skip both the create and the delete above. A shared route is platform-owned and permanent — deleting one because your sandbox is finished takes every other cluster on that tag down with it.
Private Google Access, or Cloud NAT, or both — the production-shaped option. They are different mechanisms with different prerequisites, and only the first one avoids an internet route:
- Private Google Access keeps the traffic off the internet, but reaches only Google APIs. Give it private DNS records and routes to the Google API VIPs for *.googleapis.com (node bootstrap pulls from storage.googleapis.com, so this one is not optional), *.pkg.dev, and *.gcr.io plus bare gcr.io. Listing only the registry hosts leaves bootstrap failing exactly as before.
- Cloud NAT (Public NAT) still needs a route. A NAT gateway can only use a route whose next hop is default-internet-gateway, so a VPC that had that route stripped — which is the situation this whole warning is about — does not gain egress just by adding a NAT. The NAT must also cover the node subnet’s ranges: one that exists in $REGION but does not list that subnet does nothing for these nodes. Check both the subnet coverage and the route:
  gcloud compute routers nats describe <nat> --router=<router> --region=$REGION # expect the node subnet in its NAT'd ranges — a region match alone proves nothing gcloud compute routes list --filter='destRange=0.0.0.0/0' \ --format='table(name,network,nextHopGateway,tags.list())' # expect a route in $NETWORK whose next hop is default-internet-gateway

A route is not a permission. Every remedy above assumes egress is actually allowed. GCP’s implied allow-egress rule grants that by default, but a higher-priority egress deny — a VPC firewall rule, or a hierarchical policy inherited from the folder or org, which a locked-down sandbox is likely to have — blocks the pull with a perfect route in place, and the symptom is identical. Read the effective rules rather than the VPC’s own:

gcloud compute networks get-effective-firewalls $NETWORK --project=$PROJECT

If a deny applies to $NODE_TAG, where the deny lives decides who fixes it:

A VPC firewall rule you can override locally, with a higher-priority (lower number) egress allow for the tag. Name it after $CLUSTER for the same reason the route is — it is yours, it outlives the cluster, and you delete it when that cluster is gone:

gcloud compute firewall-rules create $CLUSTER-registry-egress-allow \
  --network=$NETWORK --direction=EGRESS --action=ALLOW \
  --target-tags=$NODE_TAG --destination-ranges=0.0.0.0/0 --rules=tcp:443 \
  --priority=900 --project=$PROJECT     # must beat the deny's priority number
# after this cluster is gone — it targets no other cluster's nodes:
gcloud compute firewall-rules delete $CLUSTER-registry-egress-allow --project=$PROJECT

On the Private Google Access path, point --destination-ranges at the Google API VIP ranges rather than 0.0.0.0/0.

A hierarchical policy inherited from the folder or org you cannot override at all — those rules evaluate before VPC rules, so no rule you create in $NETWORK beats them. This one goes back to whoever owns the policy, as a request to allow egress for $NODE_TAG. It is platform-owned and permanent: do not touch it at sandbox teardown, and delete only your own $CLUSTER-prefixed rule above.

Fast verify — the first minute after the cluster object exists:

gcloud container clusters get-credentials $CLUSTER --zone $ZONE
kubectl get pods -n kube-system

Every system pod should reach Running within roughly a minute. If they hang in ContainerCreating or Pending, node egress is the cause. A clusters create that timed out on 0 nodes out of 1 have registered still leaves the cluster and a usable kubeconfig behind, so this check works on that wreck too — you do not have to delete and retry to diagnose it.If there is no cluster to point kubectl at, check the configuration instead, before creating anything. One of the three has to be true, and empty output is a fail, not a pass: a 0.0.0.0/0 route in $NETWORK carrying $NODE_TAG (the routes list above), a Cloud NAT that covers the node subnet’s ranges and has that route (routers nats describe), or Private Google Access on the node subnet — privateIpGoogleAccess: true in:

gcloud compute networks subnets describe <node-subnet> --region=$REGION --format=yaml

Dataplane V2 bring-up has the same dependency. DPv2 pulls its own images too, so the --enable-dataplane-v2 cluster above cannot come up without registry egress either. The Dataplane V2 warning above and this one are two faces of one prerequisite, not two independent problems — fix the egress and both clear.

Prefer on-demand nodes until the install is stable. A Spot preemption mid-install churns the node the backend is scheduled on and wedges PVC binding — genesis-0 then sits Pending on a volume that never re-attaches, which reads as a storage fault rather than a node one. Bring the cluster up on on-demand nodes and finish the install; Spot is fine for read-only workloads afterwards.

Grants to request

What	Grant / setting	Verify
Create the cluster (only when creating one — every other row is about deploying into a cluster that already exists) Creating a cluster is a different permission from deploying to it, and GKE separately checks that you may use the node service account. Without that second one, `clusters create` fails with `User does not have access to service account <node-sa>@…` and tells you to grant `iam.serviceAccountUser`. The `roles/container.admin` in the next row already carries `container.clusters.create`; the SA grant it does not carry.	`container.clusters.create` `roles/iam.serviceAccountUser` on the node service account (project-wide also works)	`gcloud iam service-accounts get-iam-policy <node-sa>@$PROJECT.iam.gserviceaccount.com --format='table(bindings.role,bindings.members)'` lists `roles/iam.serviceAccountUser` for your identity
Node egress to the container registry Nodes pull GKE’s own system images (`pause`, `netd`, `pdcsi`, `kube-proxy`) over the network. With no route out and no Private Google Access for the registry hostnames, the node never registers — `0 nodes out of 1 have registered` on Standard, hung `kube-system` pods and unbound PVCs on Autopilot. Dataplane V2 bring-up needs the same egress. See the warning above for both remedies.	A per-cluster tag-scoped `0.0.0.0/0` route on `$NODE_TAG`, or Private Google Access reaching `.googleapis.com`, `.pkg.dev` and `*.gcr.io`, or a Cloud NAT that covers the node subnet (which itself needs a `default-internet-gateway` route) — and, on any of them, no higher-priority egress deny	Cluster exists: `kubectl get pods -n kube-system` — all `Running` inside ~1 min No cluster yet: `gcloud compute routes list --filter='destRange=0.0.0.0/0'` — a route in `$NETWORK` carrying `$NODE_TAG`, or a NAT / PGA equivalent
Deploy + create RBAC The chart renders the workspace Role, RoleBinding, ResourceQuota and (optionally) NetworkPolicy as part of the release — GKE blocks creating them unless you already hold cluster-admin. `container.developer` is not enough. There is no separate `kubectl apply` step any more.	`roles/container.admin` (on the installing user/SA)	`kubectl auth can-i create rolebindings -n genesis-cli` `kubectl auth can-i create networkpolicies -n genesis-cli` both print `yes`. The namespace need not exist yet — this is an authorization question, not a lookup.
Enabled APIs GKE + Compute are hard requirements; Artifact Registry only if you mirror images.	`container.googleapis.com` `compute.googleapis.com` `artifactregistry.googleapis.com` (if you mirror)	`gcloud services list --enabled \| grep -E 'container\|compute\|artifactregistry'` one line per required API. A missing line is the fail — the command succeeds either way.
Node-SA image pull A GKE node pulls as its own SA, not as the identity that ran `helm`. A grant made to your corporate directory does not cover the node. For the dev cluster this is already granted — nodes pull cross-project with no mirror (Phase 2). Confirm your node SA matches what was granted.	The node’s own service account holds `roles/artifactregistry.reader` on `genesis-dist`.	`gcloud container clusters describe $CLUSTER --zone $ZONE --format='value(nodeConfig.serviceAccount)'` prints one service-account address — the one the grant names. A different address is the fail.
NetworkPolicy enforced Only needed if you intend to turn the workspace egress lockdown on; it is off by default. Any NetworkPolicy is inert unless the cluster runs a policy engine (Dataplane V2, or the legacy Calico addon). Autopilot always enforces; Standard is opt-in and can’t gain Dataplane V2 without recreation.	Want `ADVANCED_DATAPATH` (or the Calico addon `true`)	`gcloud container clusters describe $CLUSTER --zone $ZONE --format='value(networkConfig.datapathProvider)'` prints `ADVANCED_DATAPATH`. `DATAPATH_PROVIDER_UNSPECIFIED` or an empty line means no engine.
Default StorageClass Backend + workspace PVCs bind to it. GKE default is `standard-rwo` (pd-balanced → counts as SSD quota).	One row marked `(default)`	`kubectl get storageclass` exactly one row’s name ends in `(default)`. Zero means every PVC will sit `Pending`.
Regional quota See the formula below — check `CPUS`, `SSD_TOTAL_GB`, and (if you use a LoadBalancer/Ingress) `IN_USE_ADDRESSES`. Disk is the one that usually runs out, and it refuses node scale-up while CPU sits at a few percent.	Limits must exceed the formula	`gcloud compute regions describe $REGION --flatten='quotas[]' --format='csv[no-heading](quotas.metric,quotas.limit,quotas.usage)' \| grep -E '^CPUS,\|^SSD_TOTAL_GB,\|^IN_USE_ADDRESSES,'` three `metric,limit,usage` lines; `limit - usage` must exceed the formula below on each.
Egress to your IdP Backend needs HTTPS out to your OIDC JWKS and your Databricks workspace. A locked-down sandbox may block it.	Expect `200`	`kubectl run t --rm -it --image=curlimages/curl --restart=Never -- curl -sSm10 -o /dev/null -w '%{http_code}\n' https://YOUR-TENANT.us.auth0.com/.well-known/jwks.json` prints `200`. A hang or `000` is blocked egress, not a bad URL.

Capacity formula

Genesis reserves the backend once, then one Pod and two volumes per sandbox user. CPU and memory scale with concurrent users; disk scales with lifetime users, because a workspace volume outlives its Pod so that a user’s saved conversations survive being reaped. Nothing reclaims a volume automatically.

vCPU  = 4  + 0.5 × Nconcurrent   # at the shipped cap of 25 -> 16.5 vCPU
RAM   = 16 + 4   × Nconcurrent   # (Gi reserved; each may grow to 8Gi) -> 116 Gi
disk  = 50 + 20  × Nlifetime     # at the shipped ceiling of 50 -> 1050 Gi

Nconcurrent is capped by cliTerminal.maxWorkspaces (default 25) and Nlifetime by cliTerminal.maxLifetimeUsers (default 50). Lower maxWorkspaces to what this cluster can actually hold — it is one value, and the chart writes it to both the namespace quota and the backend’s own cap. A count quota reserves nothing, so the disk ceiling costs only what people actually claim.

External prerequisites (you supply)

An OIDC / Auth0 app — issuer, audience, JWKS URL, SPA client ID, and two callback URLs (Phase 4). Verify JWKS reachable with curl.
Your Databricks LLM — a workspace URL + a token, with the chat model published as a serving endpoint. Verify: curl -H "Authorization: Bearer $DAPI" <workspace>/api/2.0/serving-endpoints.

Professional Services checklist (operator responsibilities, not customer infra)

Four items that are nobody’s infrastructure and are therefore the ones that get skipped. Each has an owner and a verification, and none of them is done by the chart.

Item	Why it is yours	Done when
Generate `GENESIS_MASTER_KEY` and escrow it, with a named owner Generate with `python3 -c "import secrets; print(secrets.token_hex(32))"`. The chart refuses to render without it. It encrypts every stored credential and cannot be recovered — and it is not on the data volume, so a volume snapshot alone is half a backup.	The customer will not know to ask for this, and the person who installed Genesis will eventually leave.	The key is in the customer’s corporate secret store, not only in your shell history or the Helm release record; a named owner and a named deputy are recorded in writing; and the escrow record says which deployment it belongs to. Prefer `secrets.existingSecretGenesisMasterKey`, or `config.extraEnv.GENESIS_MASTER_KEY_REF` pointing at a `gcp_*` reference so the pod holds no key at rest.
*Ensure an embedding* serving endpoint exists and is reachable — separately from chat** Chat and embeddings are different routes, often different endpoints and different credentials.	When embeddings fail, Genesis keeps answering: semantic search silently degrades to lexical matching. An install signed off on “the bot replies” can ship with semantic search dead.	The embedding endpoint is published on the customer’s Databricks workspace, its name is configured, and `POST /api/config/llm-providers/embedding-config/test` returns `"success": true` with a dimension count — after the install, in Phase 6. Not “the chip is not red”.
Name the exact images to mirror Both of them: the backend and the workspace image.	Mirroring the backend and forgetting the workspace image produces a working console in which every terminal fails to start, days later, when the first person opens one.	Both tags resolve in the registry the nodes pull from — `crane manifest <ref> > /dev/null` succeeds for each — and `image.tag` names a tag that exists there.
Record the chart version and the image digest you deployed	”Fixed in current builds” is a question about which image is running, and only the digest settles it.	The digest from `kubectl get pod … .status.containerStatuses[].imageID` is written into the handover document beside the chart version.

Gate 1 — do not create anything until all of these hold. Every one is cheap now and expensive later; the node-egress row alone is the difference between a one-hour install and a one-day one.

kubectl get pods -n kube-system → every pod Running inside ~1 minute (or, with no cluster yet, a 0.0.0.0/0 route in $NETWORK carrying $NODE_TAG, or a Cloud NAT covering the node subnet, or Private Google Access on it)
kubectl auth can-i create rolebindings -n genesis-cli → yes
gcloud services list --enabled lists container and compute
kubectl get storageclass → exactly one row marked (default)
The node service account is the one holding roles/artifactregistry.reader
CPUS and SSD_TOTAL_GB headroom exceeds the capacity formula
The JWKS probe prints 200
All four Professional Services checklist rows are done, including the escrow owner in writing

A red row here is a grant to request, not a thing to work around. Proceeding with one outstanding is how an install becomes a week.

Phase 2 — Get the software

Both images and the chart live in one private registry, already granted to databricks.com.

Access — already granted to databricks.com. The databricks.com Google Workspace domain holds roles/artifactregistry.reader on genesis-471800/genesis-dist — which covers the two images and the Helm chart (same repo). Any databricks.com identity can pull; nothing is public.

Set the release you are installing once; the commands below reference it. Use the app version and the chart version listed for your release — the values shown are the GA release when this page was last revised:

APP_VERSION=v1.19.95     # the Genesis release tag
CHART_VERSION=0.3.104    # the chart version listed for that release

# images
us-central1-docker.pkg.dev/genesis-471800/genesis-dist/genesis:$APP_VERSION
us-central1-docker.pkg.dev/genesis-471800/genesis-dist/genesis-sandbox-claude:latest
# helm chart (OCI)
oci://us-central1-docker.pkg.dev/genesis-471800/genesis-dist/charts/genesis — version $CHART_VERSION

The registry also carries floating tags (latest, main) for the backend. Prefer the release tag: it names one build, so two installs of “the same release” are the same deployment. If you do deploy a floating tag, read the image.pullPolicy warning in Phase 3 first.

Node-SA pull — already set up for the dev cluster. A GKE node pulls images as its own service account, not a databricks.com identity. For the dev cluster this is already handled: Genesis granted your project’s default compute SA roles/artifactregistry.reader on genesis-dist, so nodes pull both images and the chart cross-project with no mirror. Point the chart straight at genesis-dist (Phase 3) and go.

Optional — mirror into your own AR. For a longer-lived or air-gapped cluster you may prefer a local copy; then set image.repository / GENESIS_CLI_TERMINAL_IMAGE to your mirror path.Create the repo and set up auth once. gcloud auth configure-docker installs the credential helper that both crane and docker read, so it is the only auth step either recipe needs — but name both hosts, since a copy reads from us-central1 and writes to your region:

SRC=us-central1-docker.pkg.dev/genesis-471800/genesis-dist
DST=$REGION-docker.pkg.dev/$PROJECT/genesis
gcloud artifacts repositories create genesis --repository-format=docker --location=$REGION
gcloud auth configure-docker us-central1-docker.pkg.dev,$REGION-docker.pkg.dev -q

Then copy with crane — it streams registry-to-registry, so it needs no local Docker daemon, pulls nothing to disk, and keeps the source digest and every platform in the manifest:

for img in genesis:$APP_VERSION genesis-sandbox-claude:latest; do
  crane copy $SRC/$img $DST/$img
done

Mirror both images — the sandbox image is not part of the backend image, and it is not pulled until the first user opens a terminal, so a missing mirror surfaces days later. Coding sandbox (CLI Terminal) image covers pinning it.No crane? Docker does the same job by round-tripping through your machine. Pin the platform — an unpinned pull on an Apple-silicon laptop mirrors an arm64 image that GKE nodes cannot run, and the failure surfaces later as a crash-looping pod, not a bad copy:

for img in genesis:$APP_VERSION genesis-sandbox-claude:latest; do
  docker pull --platform linux/amd64 $SRC/$img
  docker tag $SRC/$img $DST/$img
  docker push $DST/$img
done

There is no gcloud artifacts docker images copy — that subcommand does not exist, so reach for one of the two recipes above rather than the gcloud verb you might expect.

Pull the chart

# auth as a databricks.com identity that holds the grant
gcloud auth print-access-token | \
  helm registry login us-central1-docker.pkg.dev -u oauth2accesstoken --password-stdin

helm show chart oci://us-central1-docker.pkg.dev/genesis-471800/genesis-dist/charts/genesis --version $CHART_VERSION

Pin the chart version, and read the current one rather than trusting this page. This runbook was last revised against chart 0.3.104 / app v1.19.95. Omitting --version installs whatever is newest, which makes two installs of “the same runbook” different deployments. Ask the registry what exists:

helm show chart oci://us-central1-docker.pkg.dev/genesis-471800/genesis-dist/charts/genesis \
  | grep -E '^version:|^appVersion:'

Then pin what you read, and write it into the handover document.

The chart tarball also carries its own render-time validators, which is the cheapest way to confirm the chart version you just pinned still produces a correct, data-safe manifest. Untar it once and keep it — Phase 5’s gate uses it:

helm pull oci://us-central1-docker.pkg.dev/genesis-471800/genesis-dist/charts/genesis \
  --version $CHART_VERSION --untar --untardir /tmp/genesis-chart

Gate 2 — do not install until all three hold:

helm show chart … prints a version: and an appVersion: — your registry login works and the chart is reachable
Both images resolve in the registry your nodes will pull from: crane manifest <ref> > /dev/null && echo OK for the backend and the workspace image
bash /tmp/genesis-chart/genesis/validate-retention.sh exits 0 and ends with Retention contract holds in the rendered manifest.

The validators render locally and never contact a cluster; invoke them with bash, because files inside a Helm tarball are not extracted executable.

Phase 3 — Install the base

A values file, a master key, one helm upgrade --install.

Set image.tag explicitly. Left unset, image.tag defaults to the chart’s appVersion (for chart 0.3.104 that is v1.19.95). That works only while that exact tag exists in the registry your nodes pull from — a mirror that copied a different tag, or a chart newer than the images you mirrored, fails the pull with ImagePullBackOff. Set image.tag to the release tag you mirrored or were given, and write it into the handover document.

A floating tag needs image.pullPolicy: Always. The chart ships image.pullPolicy: IfNotPresent. If you deploy a tag that moves — latest, main, or any tag you re-point in your own mirror — a node that already has an image cached under that name does not re-pull it. The Pod restarts onto the old image and nothing reports an error: the upgrade is a silent no-op. Either pin an immutable release tag (IfNotPresent is then fine), or set:

image:
  tag: latest            # or main — any floating tag
  pullPolicy: Always     # REQUIRED with a floating tag

Either way, confirm which build is actually serving by its digest (Gate 3), not by the tag.

Create my-values.yaml. Replace every highlighted value; use your mirror path if you mirrored.

image:
  repository: us-central1-docker.pkg.dev/genesis-471800/genesis-dist/genesis
  tag: v1.19.95        # REQUIRED — the release tag you mirrored or were given
  pullPolicy: IfNotPresent   # use Always if tag is a floating tag (latest, main)

persistence: { size: 50Gi, storageClassName: "" }   # "" = cluster default
resources:
  requests: { cpu: 4000m, memory: 16Gi }
  limits: {}          # none by design — avoids OOM kills

ingress:
  enabled: true
  className: nginx       # or "gce"
  hosts: [ { host: genesis.example.com, paths: [ { path: /, pathType: Prefix, port: 8080 } ] } ]
  tls: [ { secretName: genesis-tls, hosts: [ genesis.example.com ] } ]

config:
  authEnabled: "true"
  authProvider: "auth0"
  databricksHost: "https://your-workspace.cloud.databricks.com"
  databricksBaseUrl: "https://your-workspace.cloud.databricks.com/serving-endpoints"
  modelName: "databricks-claude-opus-5-5"
  extraEnv:
    AUTH_ISSUER: "https://YOUR-TENANT.us.auth0.com/"
    AUTH_AUDIENCE: "https://genesis-api"
    AUTH_JWKS_URL: "https://YOUR-TENANT.us.auth0.com/.well-known/jwks.json"
    AUTH0_CLIENT_ID: "YOUR-SPA-CLIENT-ID"

Create the namespace, then install without `--create-namespace`

The chart renders its own Namespace object — named by the chart’s namespace value, genesis by default — and marks it helm.sh/resource-policy: keep, so that a helm uninstall or a later chart change can never cascade a delete to the data volume. Two consequences:

helm install --create-namespace fails: the namespace Helm creates for the release collides with the one the chart renders.
A plain kubectl create namespace also fails on install: Helm refuses to take over an object it does not own.

Pre-create the namespace and mark it for adoption by the release, then install with no --create-namespace:

kubectl create namespace genesis
kubectl annotate namespace genesis \
  meta.helm.sh/release-name=genesis meta.helm.sh/release-namespace=genesis
kubectl label namespace genesis app.kubernetes.io/managed-by=Helm

The release name (genesis), the -n namespace and the chart’s namespace value must all agree with those annotations. If you use a different namespace, set namespace: in my-values.yaml to the same name and annotate that namespace instead.

helm upgrade --install genesis \
  oci://us-central1-docker.pkg.dev/genesis-471800/genesis-dist/charts/genesis --version $CHART_VERSION \
  -n genesis -f my-values.yaml \
  --set secrets.genesisMasterKey="$(python3 -c 'import secrets;print(secrets.token_hex(32))')" \
  --set secrets.databricksApiKey="dapi…" \
  --wait --timeout 12m

Optional — backend scheduling priority. Off by default; turn it on only on a cluster dedicated to Genesis. By default the backend runs at Kubernetes’ implicit priority 0, the same as the workspace and agent Pods it creates, so on a full cluster the scheduler has no reason to prefer it over the workloads that depend on it. On a cluster that runs only Genesis, add:

priorityClass:
  create: true     # renders a PriorityClass and assigns it to the backend
  # preemptionPolicy: Never is the shipped default — keep it

Creating the class needs rights on scheduling.k8s.io PriorityClasses. Leave it off on a shared cluster: a class above the default makes Genesis a preferred survivor over other teams’ workloads, and changing preemptionPolicy to PreemptLowerPriority would let it evict them. To reference a class your platform team already defines, set priorityClass.name and keep create: false. It does not rank above Kubernetes’ own system tiers, and is not meant to.

First boot takes up to 10 minutes. The startup probe allows a long grace (failureThreshold: 60 × periodSeconds: 10) while the backend builds its DB, warms sessions, and loads the LLM. Watch kubectl -n genesis get pod genesis-0 -w. To tell “still working” from “stuck”, read the progress rather than the pod:

kubectl -n genesis exec genesis-0 -c genesis -- \
  curl -sS http://localhost:8080/api/health/startup
# phase, progress and current_step — always 200, even mid-boot

Gate 3 — the base is installed when all four hold:

kubectl -n genesis get pods
# expect genesis-0  1/1  Running  — nothing ImagePullBackOff or CrashLoopBackOff

kubectl -n genesis get pvc
# expect one claim, STATUS Bound

kubectl -n genesis exec genesis-0 -c genesis -- \
  curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8080/api/healthcheck
# expect 200

kubectl -n genesis get pod genesis-0 \
  -o jsonpath='{.status.containerStatuses[?(@.name=="genesis")].imageID}{"\n"}'
# expect <registry>/genesis@sha256:<64 hex> — record this digest

Then prove the Pod runs the build the tag names now — the check that catches a floating tag served from a stale node cache:

EXPECTED=$(crane digest <registry>/genesis:<your image.tag>)
RUNNING=$(kubectl -n genesis get pod genesis-0 \
  -o jsonpath='{.status.containerStatuses[?(@.name=="genesis")].imageID}')
[ "${RUNNING##*@}" = "$EXPECTED" ] && echo MATCH || echo "MISMATCH: running ${RUNNING##*@}"

ImagePullBackOff here is the node-SA grant or a tag that is not in the registry; Pending on the PVC is the StorageClass row from Gate 1. A MISMATCH on a floating tag is image.pullPolicy left at IfNotPresent; set it to Always and delete the Pod so it is recreated. On a rolling upgrade, also make sure you are reading the new Pod, not the old one that is still terminating.

One port serves everything; both surfaces authenticate through the same OIDC app.

8080 — the only port: GUI, API, terminal WS
1 URL — browser and any client hit the same host
2 callbacks — GUI origin + a loopback, in the OIDC app

Browse to https://<your-host>/ and sign in. The OIDC app must list your host in Allowed Callback URLs and Web Origins, and AUTH0_CLIENT_ID must be set (the console reads it from /api/config/frontend). Register both callbacks:

https://genesis.example.com/            # the GUI origin
http://localhost:8765/callback         # the loopback sign-in for any CLI client

No public ingress? Port-forward. For a private sandbox, skip the Ingress and reach both surfaces locally:

kubectl -n genesis port-forward svc/genesis 8080:8080

Gate 4 — access works when all three hold:

curl -sS -o /dev/null -w '%{http_code}\n' https://<your-host>/api/healthcheck → 200 from outside the cluster. A 302 here is the ingress auth annotation capturing the probe — enable ingress.authExemptHealth rather than removing the auth.
A browser at https://<your-host>/ reaches the sign-in page and a real user completes the round trip back to the console.
That user sends a chat message and gets a reply. Without a reachable LLM the server starts fine and thread creation fails with “Bot not found or unavailable” — which is an LLM-configuration answer, not a deployment one.

If sign-in loops, the OIDC app is missing your host in Allowed Callback URLs or Web Origins, or AUTH0_CLIENT_ID is unset — the console reads it from /api/config/frontend.

Phase 5 — Enable the sandbox terminal

cliTerminal.enabled=true is the whole switch: it renders the cluster objects the backend needs and turns the feature on in the application. Move 2 covers the cluster objects; Move 1 is the sizing and image settings the chart does not decide for you. Read the image section first — it is the setting most often wrong.

Coding sandbox (CLI Terminal) image

The coding sandbox — the per-user workspace Pod behind the CLI Terminal — runs a separate image from the main Genesis image. Upgrading, mirroring or pinning the backend does nothing for it: it has to be mirrored and pinned in its own right.

The feature is off by default. Nothing pulls the sandbox image until cliTerminal.enabled=true and a user opens a terminal — which is why a missing or wrong image surfaces with the first user, not at install.
Where to get it. Use the sandbox image from the registry Genesis provided you — for this install, genesis-dist/genesis-sandbox-claude — or your mirror of it. The same image is also published publicly on Docker Hub as docker.io/genesiscomputingai/genesis-sandbox-claude, pullable without credentials, with the same weekly latest and dated tags. It contains the coding tools only, no Genesis server code, so it can serve clusters that have no access to a private registry.
The reference must be fully qualified. On Kubernetes, cliTerminal.image (the app setting cli_terminal.image, env GENESIS_CLI_TERMINAL_IMAGE) must name the registry host, repository and tag, in a registry your nodes can pull from: us-central1-docker.pkg.dev/<project>/<repo>/genesis-sandbox-claude:<tag> or docker.io/genesiscomputingai/genesis-sandbox-claude:<tag>. The application’s built-in default is an unqualified name, which works for a local Docker install and cannot work here: Kubernetes resolves an unqualified name against Docker Hub’s library/ namespace, where the image does not exist, and every workspace Pod sits in ImagePullBackOff.
Pin it. genesis-sandbox-claude:latest is rebuilt and refreshed weekly, and each build is also published under a dated, immutable tag (for example 20260923-35908082637-1). Pin one of the dated tags — list them with crane ls $SRC/genesis-sandbox-claude — and record its digest beside the backend’s. Workspace Pods are created with Kubernetes’ default pull policy, so a re-pointed tag other than latest is not re-pulled on a node that already has it cached.

Mirror it multi-arch, with crane. crane cp copies registry-to-registry and keeps every platform in the manifest. A docker pull / docker push round trip from a laptop can publish only the laptop’s architecture — on Apple silicon, an arm64-only image that amd64 GKE nodes cannot run:

SANDBOX_TAG=20260923-35908082637-1   # a dated tag from `crane ls`; never moves
crane cp $SRC/genesis-sandbox-claude:$SANDBOX_TAG $DST/genesis-sandbox-claude:$SANDBOX_TAG
crane digest $DST/genesis-sandbox-claude:$SANDBOX_TAG      # record this
crane config $DST/genesis-sandbox-claude:$SANDBOX_TAG | grep -o '"architecture":"[^"]*"'
# expect "architecture":"amd64" — the platform your GKE nodes run

Then set the qualified reference in exactly one place — cliTerminal.image in the values file, or GENESIS_CLI_TERMINAL_IMAGE in config.extraEnv (Move 1) with cliTerminal.image left empty. The warning under Move 1 explains which wins.

Enabled ≠ switched on, if you set this up by hand. The two halves — the backend’s own config, and the namespace/RBAC/quota it acts through — are both required. Config alone → the backend can’t create Pods and the terminal 403s. RBAC alone → the feature stays off. cliTerminal.enabled=true does both at once; a hand-applied manifest does only the second.

Move 1 — the settings the chart does not decide

Add to config.extraEnv in my-values.yaml and re-run helm upgrade (use your mirror path for the image if you mirrored):

    GENESIS_CLI_TERMINAL_IMAGE: "us-central1-docker.pkg.dev/genesis-471800/genesis-dist/genesis-sandbox-claude:latest"   # FULLY QUALIFIED; your pinned mirror if you mirrored
    GENESIS_CLI_TERMINAL_MEMORY: "8g"          # CEILING; hard cap, 4g OOM-kills under load
    GENESIS_CLI_TERMINAL_MEMORY_REQUEST: "4g"  # what each workspace RESERVES; capacity ask
    GENESIS_CLI_TERMINAL_CPU_REQUEST: "500m"   # no CPU limit is set, so this is scheduling only
    GENESIS_CLI_TERMINAL_SHM_SIZE: "1Gi"       # /dev/shm for headless Chrome
    GENESIS_CLI_TERMINAL_HOME_VOLUME_SIZE: "10Gi"   # ×2 PVCs = 20Gi/user
    GENESIS_CLI_TERMINAL_MODEL: "databricks-claude-opus-5-5"

Do not put GENESIS_CLI_TERMINAL_ENABLED, _RUNTIME or _K8S_NAMESPACE here. cliTerminal.enabled=true sets all three on the container itself and force-locks their effective values, so they override values previously stored in Advanced Settings as well as the envFrom ConfigMap that carries config.extraEnv — an entry here of those names is silently ignored. That is deliberate: each of the three has to agree with an object the chart rendered, and a disagreement fails as a 403 or a disabled feature rather than as a visibly bad value. With ecr.enabled the same applies to GENESIS_CLI_TERMINAL_IMAGE and _IMAGE_PULL_SECRET, because the chart is what knows the registry.On GKE ecr.enabled is false, so the image is yours. Two supported places, and exactly one of them: config.extraEnv above (leave cliTerminal.image empty), or cliTerminal.image in the values file — which is empty by default precisely so it cannot mask a working extraEnv value, and which wins when you do set it.

Request and limit are different numbers on purpose. MEMORY is the ceiling a workspace may grow into and the point at which Kubernetes OOM-kills it — which kills every session that user has open. MEMORY_REQUEST is what the scheduler reserves, and so what cluster capacity must actually supply: 10 concurrent users need 10 × the request, not 10 × the ceiling. Setting them equal makes every idle workspace hold the full 8Gi, which is what made this mode need a specially sized cluster.

Turning this on does not require a Helm restart. Two delivery paths reach the same settings — pick by whether you want the state in your values file or in the running server:

IaC (what this step shows). Values in config.extraEnv / genesisConfig.content, applied with helm upgrade. Best for a reproducible deployment: the enablement lives in the values file and survives a reinstall.
In-flight. On a running server an admin sets the same settings in Config → Advanced Settings, and they take effect on the next terminal attach — no restart, no rollout. Every cli_terminal.* field is declared env_leaf: true in the config schema: a panel write refreshes that field’s environment variable immediately, and the attach path reads the variable at call time rather than caching it at startup.

The prerequisite the panel cannot satisfy is Move 2. The namespace, RBAC and quota are a one-time cluster grant. Flipping enabled in Advanced Settings without them still 403s, exactly as it would from the values file.cliTerminal.enabled=true is the chart’s platform switch: it pins the effective cli_terminal.enabled, cli_terminal.runtime, and cli_terminal.k8s_namespace values to the rendered Kubernetes objects. Setting it to false removes those chart-managed objects from the release, but deliberately does not force the application’s own setting to false; deployments that provision the namespace and RBAC outside this chart can keep working.What a live change reaches. image and the CPU / memory / volume sizes apply to workspace Pods created after the change — a user with a running workspace keeps their current Pod until it is reaped and recreated, so nothing interrupts an agent mid-task. runtime and k8s_namespace are the exception: both are fixed for the life of the server process, and a live change to either is logged and ignored rather than applied, because the workspaces already created live on the old backend and in the old namespace and nothing else can find them to reap. Change those from the values file, and retire the existing workspaces first.

Move 2 — grant the cluster objects

cliTerminal.enabled=true in your values file renders the namespace, RBAC, quota and (optionally) the workspace NetworkPolicy as part of the release, and sets plus force-locks GENESIS_CLI_TERMINAL_ENABLED, _RUNTIME and _K8S_NAMESPACE on the backend for you. This one flag is what the old kubectl apply step and the enablement env vars used to be between them:

cliTerminal:
  enabled: true
  namespace: "genesis-cli"
  image: ""                                # empty = extraEnv/APP_CONFIG owns it (see below)
  maxWorkspaces: 25                        # renders the quota AND the backend's own cap
  maxLifetimeUsers: 50                     # people who may EVER attach; claims outlive Pods
  networkPolicy:
    enabled: false                         # see the egress warning below

The chart binds the ServiceAccount the backend actually runs as — default unless serviceAccount.create or serviceAccount.name is set — so there is nothing to edit for the common case.

This namespace survives helm uninstall, and survives turning the feature back off. It holds every user’s home volume and therefore their saved Claude Code conversations, so it carries helm.sh/resource-policy: keep. Without that, an ordinary helm upgrade setting cliTerminal.enabled: false would take it out of the release manifest and Kubernetes would cascade the delete to everyone’s work — an enablement flag doubling as a data-destruction flag. Disabling revokes the RBAC and releases the quota, nothing more. A chart-created workspace namespace is labeled with the Genesis namespace of the backend instance that owns it; the cleanup scripts use that label rather than a release’s computed values, which prevents one instance from draining another instance’s retained workspaces.Removal is deliberate and separate: cluster/scripts/teardown.sh <name> --delete-workspaces, which asks before it does it. To retire one departed user instead of everyone, see Operations Runbook › Offboarding. Use cliTerminal.createNamespace: false when the namespace already exists or is managed elsewhere — that flag is about creation, not survival.Disabling is access-only. The workspace Pods are retained with the namespace, and enabled: false removes the reaper that used to remove them along with the RBAC that let it act — so an agent mid-task keeps running. Drain first, which keeps every volume:

cluster/scripts/drain-workspaces.sh <release>

A separate script from teardown.sh on purpose: it leaves the release running, which is what makes it safe to run before you disable.

If your platform objects are applied outside Helm, render them from the same template rather than keeping a hand-edited copy — a second copy diverges silently, because only one of the two ever gets updated:

helm template genesis <chart> \
  --set cliTerminal.enabled=true \
  --show-only templates/cli-terminal-rbac.yaml | kubectl apply -f -

Workspace egress is unrestricted by default. A workspace gets the same network access as any other Pod in the cluster — the same access the Genesis server has. That is deliberate: these are development workspaces and installing packages and cloning repositories is the point of them. cliTerminal.networkPolicy.enabled=true confines them to DNS plus the Genesis backend, at the cost of git/gh, npm and pip timing out.A NetworkPolicy is only as good as your CNI: Kubernetes accepts the object whether or not the CNI enforces it. GKE enforces it; a stock EKS cluster with the default VPC CNI was measured applying the same policy cleanly and enforcing nothing. Verify before relying on it:Run the control first. A blocked connection proves nothing on its own: on a locked-down VPC the cluster may have no egress to that address at all, so a probe that fails tells you the destination was unreachable, not that the policy blocked it. Two Pods, identical except for the label the policy selects on:

# 1. CONTROL — no workspace label, so the policy does not select it.
kubectl -n genesis-cli run probe-control --rm --attach --restart=Never \
  --image=busybox --command -- \
  sh -c 'if nc -z -w 5 1.1.1.1 443; then echo CONTROL-REACHED; else echo CONTROL-BLOCKED; fi'

# 2. SUBJECT — identical, but labelled as a workspace.
kubectl -n genesis-cli run probe-subject --rm --attach --restart=Never \
  --image=busybox --labels=app=genesis-cli-terminal --command -- \
  sh -c 'if nc -z -w 5 1.1.1.1 443; then echo SUBJECT-REACHED; else echo SUBJECT-BLOCKED; fi'

Only CONTROL-REACHED plus SUBJECT-BLOCKED means enforced — the label is the only difference, so the policy is what blocked it. CONTROL-REACHED with SUBJECT-REACHED means the policy is decorative. CONTROL-BLOCKED makes the run inconclusive, whatever the subject said: the cluster cannot reach that destination regardless of the policy. And silence from either command is never a pass — a Pod that was not scheduled or could not pull busybox prints nothing at all.The full verdict table is in Operations Runbook › Incident triage.

Capacity is not free and the chart cannot create it. At the shipped quota this mode asks for roughly 12.5 CPU and 100Gi of reserved memory (25 workspaces reserving 500m/4Gi each, each able to grow to 8Gi), on top of the Genesis release’s own 4 CPU / 16Gi. Disk is separate and accumulates differently: 20Gi per user in concurrent use, and the claim quota allows up to 1000Gi over the life of the deployment, because a volume outlives its Pod so a user’s conversations survive being reaped. If the cluster cannot supply it, workspaces sit Pending and every attach fails, which reads as a broken feature rather than a starved one. Lower cliTerminal.maxWorkspaces to what the cluster holds — one value, which the chart writes to both the namespace quota and the backend’s own cap. Sizing is in the capacity formula above; the scheduling traps that do not look like capacity problems — zone-pinned volumes, a node pool refused for disk quota rather than CPU — are in Operations Runbook › Scaling.

Gate 5 — the terminal is genuinely on when all four hold. Skip any of these and the feature fails for the first real user rather than for you:Read all three identities off the live deployment first. The workspace namespace and the backend’s service account are both configurable, so a gate that hardcodes them checks a different deployment than the one you installed — and on a cluster with more than one Genesis instance, possibly somebody else’s:

# The namespace the backend RUNS in (values.namespace when set, else the release's)
BACKEND_NS=$(kubectl get statefulset -A -l app.kubernetes.io/instance=genesis \
  -o jsonpath='{.items[0].metadata.namespace}')
POD=$(kubectl -n $BACKEND_NS get pod -l app.kubernetes.io/instance=genesis \
  -o jsonpath='{.items[0].metadata.name}')

# The identity the backend ACTUALLY runs as. Read it from the live Pod, which
# Kubernetes always fills in — the chart leaves it unset on the defaults, and the
# Pod then runs as `default`, so assuming the chart's own name checks nobody.
BACKEND_SA=$(kubectl -n $BACKEND_NS get pod $POD -o jsonpath='{.spec.serviceAccountName}')

# The workspace namespace the backend WILL ASK IN — the value the chart pinned onto
# the container, not the chart's default.
WS_NS=$(kubectl -n $BACKEND_NS get pod $POD -o jsonpath=\
'{.spec.containers[?(@.name=="genesis")].env[?(@.name=="GENESIS_CLI_TERMINAL_K8S_NAMESPACE")].value}')

echo "backend=$BACKEND_NS/$POD sa=$BACKEND_SA workspaces=$WS_NS"

# 1. The chart rendered the platform, in the namespace the backend will ask in
kubectl -n $WS_NS get role,rolebinding,resourcequota
# expect one of each

# 2. That exact identity may actually create workspace Pods there
kubectl auth can-i create pods -n $WS_NS \
  --as=system:serviceaccount:$BACKEND_NS:$BACKEND_SA
# expect: yes

# 3. The chart still renders this feature correctly at the version you pinned
bash /tmp/genesis-chart/genesis/validate-cli-terminal.sh
# expect exit 0, ending: In-pod coding agent mode renders correctly.

An empty $WS_NS means the chart pinned nothing, which means cliTerminal.enabled is not actually true in the release you installed — fix that before reading the rest of this gate.

Attach a real terminal as a real user, from the console. The first attach pulls the ~700 MB workspace image and can take up to a minute; after that it is instant. Nothing short of a shell prompt counts — a rendered tab is not an attached session.

If the socket closes instead, the reason names the cause: “CLI terminal access is disabled on this deployment” is the effective config, “Access denied” is the user’s use_cli_terminal permission, and a Kubernetes 403 after the socket attaches is the namespace or RBAC. The full symptom → cause → fix table is in the Operations Runbook.

Phase 6 — Final verification

Confirm the deploy by content, then hand the console to a user.

Gate 6 — the deployment is ready to hand over when all six hold. Five of these are cheap; the fifth is the one installs skip, and it is the one that silently ships a broken deployment.

Health, from outside. Expect 200:

curl -sS -o /dev/null -w '%{http_code}\n' https://<host>/api/healthcheck

No stuck pulls, in either namespace. kubectl get pods -n genesis and kubectl get pods -n genesis-cli — nothing in ImagePullBackOff, CrashLoopBackOff or Pending. A stuck pull here is the node-SA row from Gate 1 or a tag that is not in the registry.
Console. https://<host>/ signs a real user in, and that user gets a reply to a chat message. Admins also get Pulse → CLI Workspaces (/pulse/cli-workspaces) to list running workspace Pods with restart and cancel.
Terminal. A user opens a terminal and gets a shell prompt. The first attach pulls the ~700 MB image and can take a minute; after that it is instant. The Pod is reaped when idle and the volume persists, so reconnecting restores their work.
Embeddings, verified separately from chat. Chat replying proves nothing about semantic search — they are different routes and often different endpoints and credentials. Force a real embedding call:
curl -sS -X POST https://<host>/api/config/llm-providers/embedding-config/test \ -H "Authorization: Bearer <token>" -H 'Content-Type: application/json' -d '{}' # expect {"success": true, ..., "dimensions": <n>}
"success": false carries the provider’s own error text. Do not sign off on a status chip that merely is not red.
Recorded. The chart version and the running image digest are written into the handover document, along with the escrow owner for GENESIS_MASTER_KEY.

Then hand over the Operations Runbook, which is what the customer’s operator will need at 02:00.

When it goes wrong

The failure-mode catalogue lives in one place — symptom → cause → fix for every failure below, in Operations Runbook › Incident triage. It is not repeated here, because a description kept in two places goes stale in one of them. What this page keeps is the GCP-specific remedy detail each symptom points at. Skim this index before you start; each row names the phase where the cause is fixed.

Symptom	Phase	GCP remedy on this page
`0 nodes out of 1 have registered`, or `kube-system` pods stuck `ContainerCreating` with `CSINode` at 0 drivers and unbound PVCs	1	The node-egress warning — tag-scoped route, Private Google Access, or Cloud NAT, plus the effective-firewall check. This is the expensive one: `kubectl get pods -n kube-system` first; a minute there saves a day.
Backend `Pending` on a volume that never attaches, mid-install	1	Install on on-demand nodes; a Spot preemption wedges PVC binding and reads as a storage fault.
Workspace Pods `ImagePullBackOff`	1, 5	The node-SA row — a node pulls as its own service account, which a databricks.com grant does not cover — and the sandbox image reference: an unqualified name resolves to Docker Hub `library/` and fails; an `arm64`-only mirror fails on `amd64` nodes.
Backend `ImagePullBackOff`	3	`image.tag` unset (it defaults to the chart’s `appVersion`) or naming a tag that is not in the registry the nodes pull from.
An upgrade “succeeded” but the old build is still serving	3	A floating tag (`latest`, `main`) with `image.pullPolicy: IfNotPresent`. Set `Always`, delete the Pod, and compare the running digest (Gate 3).
`helm install` fails on the `Namespace`, or with `--create-namespace`	3	Pre-create the namespace with the Helm adoption annotations and label, and drop `--create-namespace`.
NetworkPolicy applied cleanly and enforces nothing	1	Dataplane V2 is creation-time only; retrofit Calico with `gcloud container clusters update … --enable-network-policy`. Note the lockdown is off by default.
Terminal returns a 403 or “disabled” though the chart rendered everything	5	Gate 5 — the close reason distinguishes effective config, the user’s `use_cli_terminal` permission, and a Kubernetes 403 on workspace creation.
Chat works, semantic search silently degrades	6	Gate 6 item 5 — verify the embedding endpoint independently of chat.
`helm upgrade` fails with `invalid ownership metadata`	5	The migration note — set `cliTerminal.createNamespace=false` and delete only the Role, RoleBinding, quota and policy. Never the namespace.
”Reset to defaults” reports off while the terminal stays live	—	Version-gated; fixed in current builds. Upgrade the image, or verify the real state after a reset rather than trusting the panel.

Two facts that are not symptoms but are worth knowing before you tear anything down:

PVCs are retained on uninstall, by design. The backend volume and every workspace volume survive helm uninstall; workspace volumes are per-user home directories and nothing auto-deletes them. Reclaiming them is deliberate and explicit — see Teardown without data loss.
Chart version matters for the retention contract. Helm reads the retention policy from the live object, so it protects nothing until an upgrade has written it. On a deployment that enabled the terminal on a chart older than 0.3.83, going straight to cliTerminal.enabled: false in one helm upgrade still deletes the namespace. Upgrade with the feature still on first, then disable in a second step. Deployments enabling it for the first time on 0.3.83 or later are never exposed to this.

Connecting remotely to the session

To watch or drive a Claude Code session managing this cluster from another device, or to coordinate two sessions inside the same pod, see Remote Connections to Claude Code — including the caveat that Remote Control does not work with the in-pod claude that routes its LLM through the Genesis proxy.

Genesis on GKE · last revised against chart 0.3.104 / app v1.19.95 — use the chart version listed for your release, confirm it with helm show chart, and pin what you read · images + chart in us-central1-docker.pkg.dev/genesis-471800/genesis-dist. Replace every highlighted placeholder with your values. Preflight (Phase 1) is safe to run against any cluster before you commit. Day-2 operations: Genesis Operations Runbook.

Getting Started

Getting to Know Genesis

Creating A Mission

User Guide

Blueprint

Genesis Data Agents

Genesis Prover

Genesis Apps

Genesis REST API

Genesis Agent Security Controls

Genesis Data Agent's Toolkit

Setup

Slack and Teams

Data Connections

Deployment Options

Installing Genesis on GKE with Helm

What you’re installing

Phase 1 — Preflight and grants

Grants to request

Capacity formula

External prerequisites (you supply)

Professional Services checklist (operator responsibilities, not customer infra)

Phase 2 — Get the software

Pull the chart

Phase 3 — Install the base

Create the namespace, then install without `--create-namespace`

Phase 5 — Enable the sandbox terminal

Coding sandbox (CLI Terminal) image

Move 1 — the settings the chart does not decide

Move 2 — grant the cluster objects

Phase 6 — Final verification

When it goes wrong

Connecting remotely to the session

​What you’re installing

​Phase 1 — Preflight and grants

​Grants to request

​Capacity formula

​External prerequisites (you supply)

​Professional Services checklist (operator responsibilities, not customer infra)

​Phase 2 — Get the software

​Pull the chart

​Phase 3 — Install the base

​Create the namespace, then install without --create-namespace

​Phase 4 — Access and sign-in

​Phase 5 — Enable the sandbox terminal

​Coding sandbox (CLI Terminal) image

​Move 1 — the settings the chart does not decide

​Move 2 — grant the cluster objects

​Phase 6 — Final verification

​When it goes wrong

​Connecting remotely to the session

What you’re installing

Phase 1 — Preflight and grants

Grants to request

Capacity formula

External prerequisites (you supply)

Professional Services checklist (operator responsibilities, not customer infra)

Phase 2 — Get the software

Pull the chart

Phase 3 — Install the base

Create the namespace, then install without `--create-namespace`

Phase 4 — Access and sign-in

Phase 5 — Enable the sandbox terminal

Coding sandbox (CLI Terminal) image

Move 1 — the settings the chart does not decide

Move 2 — grant the cluster objects

Phase 6 — Final verification

When it goes wrong

Connecting remotely to the session