> ## Documentation Index
> Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Installing Genesis on GKE with Helm

> Self-contained runbook for deploying Genesis on a GKE cluster with Helm, including the in-pod sandbox terminal.

Everything needed to stand up Genesis on your own Google Kubernetes Engine cluster
with Helm — the web console and the on-demand sandbox terminal, reached through one
URL. Self-contained: images and chart come from a registry you are already granted;
no source access required.

* **Chart** the chart version listed for your release (for example `0.3.104` · **app**
  `v1.19.95`, the GA release when this page was last revised)
* **Helm** 3.8+ · **Kubernetes** 1.23+ · **`crane`** or Docker *(only if you mirror images)*
* **Backend** 1 × StatefulSet (single replica)
* **Port** `8080` — serves GUI, API, and the terminal WebSocket
* **Sandbox mode** opt-in

The install runs in **six phases**, and each ends in a gate — an explicit list of checks
that must pass before the next phase starts. Skipping a gate is how a one-hour install
becomes a one-day one: every failure this runbook has seen in the field surfaces several
phases after its cause.

<Note>
  **For everything after the install** — upgrade, rollback, backup and restore, secrets,
  user offboarding, scaling, teardown, and the full failure-mode catalogue — see the
  [Genesis Operations Runbook](/docs/deployments/Genesis-Operations-Runbook). It is
  cloud-agnostic, and it is the page to hand the customer's operations team at handover.
</Note>

<Note>
  This is the full base-plus-sandbox-terminal install runbook. For the separate
  `sandbox_runner.runtime: kubernetes` per-thread runner (one Pod per selected chat
  thread), see [Deploying the Genesis Sandbox Runner on GKE](/docs/deployments/GCP/Genesis-On-GCP-GKE).
</Note>

## What you're installing

Two container images, one Helm release, one reachable URL.

Genesis runs as a **single-replica StatefulSet** — a FastAPI backend serving the
React web console, the REST API, *and* the terminal WebSocket, all on **one port
(`8080`)**. There is no separate service or port for the terminal.

| Component | Role | Image |
| - | - | - |
| **Genesis backend** | The server: console + API + terminal WS. One StatefulSet, PVC-backed SQLite. | `genesis-dist/genesis:<release tag>`, e.g. `v1.19.95` |
| **Sandbox workspace** *(opt-in)* | Per-user Pod the backend spins up on demand — Node, Python, git, tmux: a full dev shell. A **separate image** from the backend — see [Coding sandbox (CLI Terminal) image](#coding-sandbox-cli-terminal-image). | `genesis-dist/genesis-sandbox-claude:latest` |

When a user opens a terminal, the backend creates a **workspace Pod for that user**
in a dedicated namespace, attaches a tmux session, and streams it over the same
port. That is the on-demand sandbox mode — off by default, enabled in Phase 5.

<Note>
  **Honest scope.** The base install (Phases 1–4) is standard Helm. The in-pod coding
  agent mode (Phase 5) is opt-in and is enabled by one `cliTerminal.enabled` chart
  setting, which renders the namespace, RBAC and quota as part of the release.
</Note>

## Phase 1 — Preflight and grants

Run this before installing. Anything red here is a grant to request from your GCP
admin — the checks are exact so nothing surfaces mid-deploy.

Set your targets once; the checks reference them:

```bash theme={null}
PROJECT=your-gcp-project
REGION=us-central1
ZONE=us-central1-a
CLUSTER=genesis
NETWORK=default          # the VPC the nodes sit on — change this for a custom VPC
NODE_TAG=$CLUSTER-node   # per-cluster, so one cluster's egress route never governs another
gcloud config set project $PROJECT
```

<Tip>
  **Use a Standard cluster.** Standard is the recommended path for an enterprise GCP
  org. Autopilot is simpler when your org allows it, but its fully-managed node
  provisioning is frequently blocked by org policies (shielded-VM, no-external-IP, SA
  constraints) — the symptom is *no nodes ever provision and pods sit pending with no
  clear error*. Standard lets you control the node config to satisfy those policies,
  so it deploys on the first try.
</Tip>

<Note>
  **If you do use Autopilot**, three things it does not manage away. **Compute Engine
  quota** still applies — Autopilot nodes are ordinary VMs, and a cluster that fails
  minutes in with `STATUS: ERROR` usually names `SSD_TOTAL_GB` only in
  `gcloud container clusters describe … --format='value(conditions)'`. **Requests are
  rewritten** — Autopilot enforces a memory-to-CPU ratio and raises CPU to match (a
  workspace asking 500m/4Gi is admitted at roughly 616m) and adds ephemeral storage to
  every container, so budget the admitted figures, not the requested ones. **Nodes are
  consolidated** on Autopilot's schedule, so expect occasional unscheduled backend
  restarts and slower first workspace launches while a node provisions.
</Note>

Create a Standard cluster (already have one? just `get-credentials` below):

```bash theme={null}
# --enable-dataplane-v2 is CREATION-TIME ONLY and is what makes any NetworkPolicy enforce
# --machine-type e2-standard-8 is 8 vCPU / 32Gi per node
# --tags=$NODE_TAG is what the registry-egress route below matches on
gcloud container clusters create $CLUSTER --zone $ZONE \
  --enable-dataplane-v2 \
  --machine-type e2-standard-8 \
  --tags=$NODE_TAG \
  --num-nodes 1 --enable-autoscaling --min-nodes 1 --max-nodes 5
# (regional --region creates num-nodes PER zone; --zone keeps it to one for a trial)

gcloud container clusters get-credentials $CLUSTER --zone $ZONE
```

<Warning>
  **Dataplane V2 is what makes a NetworkPolicy mean anything — and it is creation-time
  only.** Kubernetes accepts a NetworkPolicy object whether or not the cluster runs a
  policy engine. **Dataplane V2** is that engine. Without it, any policy you apply is
  silently ignored, with no error on either side — the failure looks exactly like
  working containment. GKE cannot gain Dataplane V2 after creation, so create the
  cluster with `--enable-dataplane-v2` even if you have no immediate need for it.

  **Already created a Standard cluster without it?** You don't have to recreate —
  enable the legacy Calico engine on the existing cluster (a rolling node recreation,
  no full rebuild):

  ```bash theme={null}
  gcloud container clusters update $CLUSTER --zone $ZONE --enable-network-policy
  ```

  **Be accurate about what this buys you, because the reassuring version is wrong.**
  The workspace egress restriction is **off by default**
  (`cliTerminal.networkPolicy.enabled: false`). By default a workspace Pod — and a
  per-thread runner Pod — has exactly the network access any other Pod in this cluster
  has, which is the same access the Genesis backend has, because the Genesis namespace
  carries no NetworkPolicy either. That is deliberate: these are development
  workspaces, and installing packages and cloning repositories is the point of them.

  Turning the policy on confines workspaces to DNS plus the Genesis backend, at the
  cost of `git`, `gh`, npm and pip timing out. Do **not** tell a security reviewer that
  agents are confined to DNS and the backend by default. They are not. If you need that
  confinement, enable it in Phase 5 and then *verify it on the cluster* — the engine is
  the prerequisite, not the guarantee.
</Warning>

<Warning>
  **Nodes need outbound egress to the container registry — this is the one that costs a
  day.** In a locked-down VPC with no `0.0.0.0/0` route, and registry hostnames resolving
  outside Private Google Access scope, a node cannot pull **GKE's own system images**
  (`pause`, `netd`, `pdcsi`, `kube-proxy`). The node then never registers and cluster
  creation dies on:

  ```text theme={null}
  0 nodes out of 1 have registered
  ```

  On Autopilot the same cause wears a different costume — `kube-system` pods stuck in
  `ContainerCreating`, `CSINode` reporting 0 drivers, PVCs that never bind. It reproduces
  identically on Autopilot and Standard, and it presents as a confusing cascade rather than
  an obvious pull failure. Nothing in the chart or your values file is at fault.

  **Grant the egress before you create the cluster.** Two remedies to choose between, then
  one caveat that applies to whichever you pick:

  * **A tag-scoped internet route** — good for a sandbox, because it opens egress for the
    nodes only and nothing else. Both the route and the tag are named after `$CLUSTER`, so
    this route governs exactly one cluster's nodes and deleting it can never revoke another
    cluster's egress:

    ```bash theme={null}
    gcloud compute routes create $CLUSTER-registry-egress \
      --network=$NETWORK \
      --destination-range=0.0.0.0/0 --next-hop-gateway=default-internet-gateway \
      --tags=$NODE_TAG --project=$PROJECT
    ```

    `--network` is spelled out on purpose: omit it and `gcloud` silently uses the VPC
    literally named `default`, so on a custom VPC the route is created successfully in a
    network your nodes are not on and the egress does not change. The route matches only
    nodes carrying `$NODE_TAG`, which the `clusters create` recipe above applies — on an
    older cluster or a `node-pools create` of your own, the route matches no instance and
    changes nothing, so confirm the tag before blaming the route.

    **This route is yours, it outlives the cluster, and you delete it.** Because it is
    per-cluster, teardown is safe once *that* cluster is gone:

    ```bash theme={null}
    gcloud compute routes list --filter='destRange=0.0.0.0/0' \
      --format='table(name,network,nextHopGateway,tags.list())'
    # confirm the nodes actually carry the tag the route matches
    gcloud container node-pools describe default-pool --cluster=$CLUSTER --zone=$ZONE \
      --format='value(config.tags)'
    # after this cluster is gone — it governs no other cluster's nodes:
    gcloud compute routes delete $CLUSTER-registry-egress --project=$PROJECT
    ```

    Read that listing on both axes. Route names are unique **per project**, not per
    network, so a route of the same name anywhere in the project is what will collide —
    which is why the filter is deliberately not narrowed to one VPC. For the egress
    question, only a row whose `network` column matches `$NETWORK` **and** whose tags
    include `$NODE_TAG` counts as coverage for this cluster.

    **If your platform team already owns a shared egress route**, reuse it instead: use the
    tag *it* matches on as `NODE_TAG` and skip both the create and the delete above. A
    shared route is platform-owned and permanent — deleting one because your sandbox is
    finished takes every other cluster on that tag down with it.

  * **Private Google Access, or Cloud NAT, or both** — the production-shaped option. They
    are different mechanisms with different prerequisites, and only the first one avoids an
    internet route:

    * **Private Google Access** keeps the traffic off the internet, but reaches *only*
      Google APIs. Give it private DNS records and routes to the Google API VIPs for
      **`*.googleapis.com`** (node bootstrap pulls from `storage.googleapis.com`, so this
      one is not optional), **`*.pkg.dev`**, and **`*.gcr.io` plus bare `gcr.io`**. Listing
      only the registry hosts leaves bootstrap failing exactly as before.
    * **Cloud NAT (Public NAT) still needs a route.** A NAT gateway can only use a route
      whose next hop is `default-internet-gateway`, so a VPC that had that route stripped —
      which is the situation this whole warning is about — does not gain egress just by
      adding a NAT. The NAT must also cover the node subnet's ranges: one that exists in
      `$REGION` but does not list that subnet does nothing for these nodes. Check both the
      subnet coverage and the route:

      ```bash theme={null}
      gcloud compute routers nats describe <nat> --router=<router> --region=$REGION
      # expect the node subnet in its NAT'd ranges — a region match alone proves nothing
      gcloud compute routes list --filter='destRange=0.0.0.0/0' \
        --format='table(name,network,nextHopGateway,tags.list())'
      # expect a route in $NETWORK whose next hop is default-internet-gateway
      ```

  **A route is not a permission.** Every remedy above assumes egress is actually allowed.
  GCP's implied allow-egress rule grants that by default, but a higher-priority egress deny
  — a VPC firewall rule, or a hierarchical policy inherited from the folder or org, which a
  locked-down sandbox is likely to have — blocks the pull with a perfect route in place, and
  the symptom is identical. Read the *effective* rules rather than the VPC's own:

  ```bash theme={null}
  gcloud compute networks get-effective-firewalls $NETWORK --project=$PROJECT
  ```

  If a deny applies to `$NODE_TAG`, **where the deny lives decides who fixes it**:

  * **A VPC firewall rule** you can override locally, with a higher-priority (lower number)
    egress allow for the tag. Name it after `$CLUSTER` for the same reason the route is —
    it is yours, it outlives the cluster, and you delete it when that cluster is gone:

    ```bash theme={null}
    gcloud compute firewall-rules create $CLUSTER-registry-egress-allow \
      --network=$NETWORK --direction=EGRESS --action=ALLOW \
      --target-tags=$NODE_TAG --destination-ranges=0.0.0.0/0 --rules=tcp:443 \
      --priority=900 --project=$PROJECT     # must beat the deny's priority number
    # after this cluster is gone — it targets no other cluster's nodes:
    gcloud compute firewall-rules delete $CLUSTER-registry-egress-allow --project=$PROJECT
    ```

    On the Private Google Access path, point `--destination-ranges` at the Google API VIP
    ranges rather than `0.0.0.0/0`.

  * **A hierarchical policy inherited from the folder or org** you cannot override at all —
    those rules evaluate before VPC rules, so no rule you create in `$NETWORK` beats them.
    This one goes back to whoever owns the policy, as a request to allow egress for
    `$NODE_TAG`. It is platform-owned and permanent: do not touch it at sandbox teardown,
    and delete only your own `$CLUSTER-`prefixed rule above.

  **Fast verify — the first minute after the cluster object exists:**

  ```bash theme={null}
  gcloud container clusters get-credentials $CLUSTER --zone $ZONE
  kubectl get pods -n kube-system
  ```

  Every system pod should reach `Running` within roughly a minute. If they hang in
  `ContainerCreating` or `Pending`, node egress is the cause. A `clusters create` that timed
  out on `0 nodes out of 1 have registered` still leaves the cluster and a usable kubeconfig
  behind, so this check works on that wreck too — you do not have to delete and retry to
  diagnose it.

  **If there is no cluster to point `kubectl` at**, check the configuration instead, before
  creating anything. One of the three has to be true, and **empty output is a fail, not a
  pass**: a `0.0.0.0/0` route **in `$NETWORK`** carrying `$NODE_TAG` (the `routes list`
  above), a Cloud NAT that covers the node subnet's ranges *and* has that route
  (`routers nats describe`), or Private Google Access on the node subnet —
  `privateIpGoogleAccess: true` in:

  ```bash theme={null}
  gcloud compute networks subnets describe <node-subnet> --region=$REGION --format=yaml
  ```

  **Dataplane V2 bring-up has the same dependency.** DPv2 pulls its own images too, so the
  `--enable-dataplane-v2` cluster above cannot come up without registry egress either. The
  Dataplane V2 warning above and this one are two faces of one prerequisite, not two
  independent problems — fix the egress and both clear.
</Warning>

<Warning>
  **Prefer on-demand nodes until the install is stable.** A Spot preemption mid-install
  churns the node the backend is scheduled on and wedges PVC binding — `genesis-0` then
  sits `Pending` on a volume that never re-attaches, which reads as a storage fault rather
  than a node one. Bring the cluster up on on-demand nodes and finish the install; Spot is
  fine for read-only workloads afterwards.
</Warning>

### Grants to request

| What | Grant / setting | Verify |
| - | - | - |
| **Create the cluster** *(only when creating one — every other row is about deploying into a cluster that already exists)* <br /> Creating a cluster is a different permission from deploying to it, and GKE separately checks that you may *use* the node service account. Without that second one, `clusters create` fails with `User does not have access to service account <node-sa>@…` and tells you to grant `iam.serviceAccountUser`. The `roles/container.admin` in the next row already carries `container.clusters.create`; the SA grant it does **not** carry. | `container.clusters.create` <br /> `roles/iam.serviceAccountUser` **on the node service account** *(project-wide also works)* | `gcloud iam service-accounts get-iam-policy <node-sa>@$PROJECT.iam.gserviceaccount.com --format='table(bindings.role,bindings.members)'` <br /> *lists `roles/iam.serviceAccountUser` for your identity* |
| **Node egress to the container registry** <br /> Nodes pull GKE's own system images (`pause`, `netd`, `pdcsi`, `kube-proxy`) over the network. With no route out and no Private Google Access for the registry hostnames, the node never registers — `0 nodes out of 1 have registered` on Standard, hung `kube-system` pods and unbound PVCs on Autopilot. Dataplane V2 bring-up needs the same egress. See the warning above for both remedies. | A per-cluster tag-scoped `0.0.0.0/0` route on `$NODE_TAG`, **or** Private Google Access reaching `*.googleapis.com`, `*.pkg.dev` and `*.gcr.io`, **or** a Cloud NAT that covers the node subnet (which itself needs a `default-internet-gateway` route) — and, on any of them, no higher-priority egress deny | Cluster exists: `kubectl get pods -n kube-system` — *all `Running` inside \~1 min* <br /> No cluster yet: `gcloud compute routes list --filter='destRange=0.0.0.0/0'` — *a route in `$NETWORK` carrying `$NODE_TAG`, or a NAT / PGA equivalent* |
| **Deploy + create RBAC** <br /> The chart renders the workspace Role, RoleBinding, ResourceQuota and (optionally) NetworkPolicy as part of the release — GKE blocks creating them unless you already hold cluster-admin. `container.developer` is **not** enough. There is no separate `kubectl apply` step any more. | `roles/container.admin` <br /> *(on the installing user/SA)* | `kubectl auth can-i create rolebindings -n genesis-cli` <br /> `kubectl auth can-i create networkpolicies -n genesis-cli` <br /> *both print `yes`. The namespace need not exist yet — this is an authorization question, not a lookup.* |
| **Enabled APIs** <br /> GKE + Compute are hard requirements; Artifact Registry only if you mirror images. | `container.googleapis.com` <br /> `compute.googleapis.com` <br /> `artifactregistry.googleapis.com` *(if you mirror)* | `gcloud services list --enabled \| grep -E 'container\|compute\|artifactregistry'` <br /> *one line per required API. A missing line is the fail — the command succeeds either way.* |
| **Node-SA image pull** <br /> A GKE node pulls as its own SA, not as the identity that ran `helm`. A grant made to your corporate directory does not cover the node. For the dev cluster this is already granted — nodes pull cross-project with no mirror (Phase 2). Confirm your node SA matches what was granted. | *The node's own service account holds `roles/artifactregistry.reader` on `genesis-dist`.* | `gcloud container clusters describe $CLUSTER --zone $ZONE --format='value(nodeConfig.serviceAccount)'` <br /> *prints one service-account address — the one the grant names. A different address is the fail.* |
| **NetworkPolicy enforced** <br /> Only needed if you intend to turn the workspace egress lockdown on; it is off by default. Any NetworkPolicy is **inert** unless the cluster runs a policy engine (Dataplane V2, or the legacy Calico addon). Autopilot always enforces; Standard is opt-in and can't gain Dataplane V2 without recreation. | Want `ADVANCED_DATAPATH` (or the Calico addon `true`) | `gcloud container clusters describe $CLUSTER --zone $ZONE --format='value(networkConfig.datapathProvider)'` <br /> *prints `ADVANCED_DATAPATH`. `DATAPATH_PROVIDER_UNSPECIFIED` or an empty line means no engine.* |
| **Default StorageClass** <br /> Backend + workspace PVCs bind to it. GKE default is `standard-rwo` (pd-balanced → counts as SSD quota). | One row marked `(default)` | `kubectl get storageclass` <br /> *exactly one row's name ends in `(default)`. Zero means every PVC will sit `Pending`.* |
| **Regional quota** <br /> See the formula below — check `CPUS`, `SSD_TOTAL_GB`, and (if you use a LoadBalancer/Ingress) `IN_USE_ADDRESSES`. **Disk is the one that usually runs out**, and it refuses node scale-up while CPU sits at a few percent. | Limits must exceed the formula | `gcloud compute regions describe $REGION --flatten='quotas[]' --format='csv[no-heading](quotas.metric,quotas.limit,quotas.usage)' \| grep -E '^CPUS,\|^SSD_TOTAL_GB,\|^IN_USE_ADDRESSES,'` <br /> *three `metric,limit,usage` lines; `limit - usage` must exceed the formula below on each.* |
| **Egress to your IdP** <br /> Backend needs HTTPS out to your OIDC JWKS and your Databricks workspace. A locked-down sandbox may block it. | Expect `200` | `kubectl run t --rm -it --image=curlimages/curl --restart=Never -- curl -sSm10 -o /dev/null -w '%{http_code}\n' https://YOUR-TENANT.us.auth0.com/.well-known/jwks.json` <br /> *prints `200`. A hang or `000` is blocked egress, not a bad URL.* |

### Capacity formula

Genesis reserves the backend once, then one Pod and two volumes per sandbox user. CPU
and memory scale with **concurrent** users; disk scales with **lifetime** users,
because a workspace volume outlives its Pod so that a user's saved conversations
survive being reaped. Nothing reclaims a volume automatically.

```text theme={null}
vCPU  = 4  + 0.5 × Nconcurrent   # at the shipped cap of 25 -> 16.5 vCPU
RAM   = 16 + 4   × Nconcurrent   # (Gi reserved; each may grow to 8Gi) -> 116 Gi
disk  = 50 + 20  × Nlifetime     # at the shipped ceiling of 50 -> 1050 Gi
```

`Nconcurrent` is capped by `cliTerminal.maxWorkspaces` (default **25**) and
`Nlifetime` by `cliTerminal.maxLifetimeUsers` (default **50**). Lower
`maxWorkspaces` to what this cluster can actually hold — it is one value, and the
chart writes it to both the namespace quota and the backend's own cap. A count quota
reserves nothing, so the disk ceiling costs only what people actually claim.

### External prerequisites (you supply)

* **An OIDC / Auth0 app** — issuer, audience, JWKS URL, SPA client ID, and two
  callback URLs (Phase 4). Verify JWKS reachable with `curl`.
* **Your Databricks LLM** — a workspace URL + a token, with the chat model published
  as a serving endpoint. Verify:
  `curl -H "Authorization: Bearer $DAPI" <workspace>/api/2.0/serving-endpoints`.

### Professional Services checklist (operator responsibilities, not customer infra)

Four items that are nobody's infrastructure and are therefore the ones that get
skipped. Each has an owner and a verification, and none of them is done by the chart.

| Item | Why it is yours | Done when |
| - | - | - |
| **Generate `GENESIS_MASTER_KEY` and escrow it, with a named owner** <br /> Generate with `python3 -c "import secrets; print(secrets.token_hex(32))"`. The chart refuses to render without it. It encrypts every stored credential and **cannot be recovered** — and it is **not on the data volume**, so a volume snapshot alone is half a backup. | The customer will not know to ask for this, and the person who installed Genesis will eventually leave. | The key is in the customer's corporate secret store, **not** only in your shell history or the Helm release record; a named owner **and a named deputy** are recorded in writing; and the escrow record says which deployment it belongs to. Prefer `secrets.existingSecretGenesisMasterKey`, or `config.extraEnv.GENESIS_MASTER_KEY_REF` pointing at a `gcp_*` reference so the pod holds no key at rest. |
| **Ensure an *embedding* serving endpoint exists and is reachable — separately from chat** <br /> Chat and embeddings are different routes, often different endpoints and different credentials. | When embeddings fail, Genesis keeps answering: semantic search silently degrades to lexical matching. An install signed off on "the bot replies" can ship with semantic search dead. | The embedding endpoint is published on the customer's Databricks workspace, its name is configured, and `POST /api/config/llm-providers/embedding-config/test` returns `"success": true` with a dimension count — **after** the install, in Phase 6. Not "the chip is not red". |
| **Name the exact images to mirror** <br /> Both of them: the backend **and** the workspace image. | Mirroring the backend and forgetting the workspace image produces a working console in which every terminal fails to start, days later, when the first person opens one. | Both tags resolve in the registry the **nodes** pull from — `crane manifest <ref> > /dev/null` succeeds for each — and `image.tag` names a tag that exists there. |
| **Record the chart version and the image digest you deployed** | "Fixed in current builds" is a question about which image is running, and only the digest settles it. | The digest from `kubectl get pod … .status.containerStatuses[].imageID` is written into the handover document beside the chart version. |

<Check>
  **Gate 1 — do not create anything until all of these hold.** Every one is cheap now
  and expensive later; the node-egress row alone is the difference between a one-hour
  install and a one-day one.

  1. `kubectl get pods -n kube-system` → every pod `Running` inside \~1 minute *(or, with
     no cluster yet, a `0.0.0.0/0` route in `$NETWORK` carrying `$NODE_TAG`, or a Cloud
     NAT covering the node subnet, or Private Google Access on it)*
  2. `kubectl auth can-i create rolebindings -n genesis-cli` → `yes`
  3. `gcloud services list --enabled` lists `container` and `compute`
  4. `kubectl get storageclass` → exactly one row marked `(default)`
  5. The node service account is the one holding `roles/artifactregistry.reader`
  6. `CPUS` and `SSD_TOTAL_GB` headroom exceeds the capacity formula
  7. The JWKS probe prints `200`
  8. All four Professional Services checklist rows are done, **including the escrow
     owner in writing**

  A red row here is a grant to request, not a thing to work around. Proceeding with one
  outstanding is how an install becomes a week.
</Check>

## Phase 2 — Get the software

Both images and the chart live in one private registry, already granted to
databricks.com.

<Check>
  **Access — already granted to databricks.com.** The `databricks.com` Google
  Workspace domain holds `roles/artifactregistry.reader` on
  `genesis-471800/genesis-dist` — which covers the two images *and* the Helm chart
  (same repo). Any databricks.com identity can pull; nothing is public.
</Check>

Set the release you are installing once; the commands below reference it. Use the app
version and the chart version listed for your release — the values shown are the GA
release when this page was last revised:

```bash theme={null}
APP_VERSION=v1.19.95     # the Genesis release tag
CHART_VERSION=0.3.104    # the chart version listed for that release
```

```text theme={null}
# images
us-central1-docker.pkg.dev/genesis-471800/genesis-dist/genesis:$APP_VERSION
us-central1-docker.pkg.dev/genesis-471800/genesis-dist/genesis-sandbox-claude:latest
# helm chart (OCI)
oci://us-central1-docker.pkg.dev/genesis-471800/genesis-dist/charts/genesis — version $CHART_VERSION
```

The registry also carries floating tags (`latest`, `main`) for the backend. Prefer the
release tag: it names one build, so two installs of "the same release" are the same
deployment. If you do deploy a floating tag, read the `image.pullPolicy` warning in
Phase 3 first.

<Check>
  **Node-SA pull — already set up for the dev cluster.** A GKE node pulls images as
  its own service account, not a databricks.com identity. For the dev cluster this is
  already handled: Genesis granted your project's default compute SA
  `roles/artifactregistry.reader` on `genesis-dist`, so nodes pull both images and the
  chart cross-project with **no mirror**. Point the chart straight at `genesis-dist`
  (Phase 3) and go.
</Check>

<Note>
  **Optional — mirror into your own AR.** For a longer-lived or air-gapped cluster you
  may prefer a local copy; then set `image.repository` / `GENESIS_CLI_TERMINAL_IMAGE`
  to your mirror path.

  Create the repo and set up auth once. `gcloud auth configure-docker` installs the
  credential helper that **both** `crane` and `docker` read, so it is the only auth step
  either recipe needs — but name **both** hosts, since a copy reads from `us-central1` and
  writes to your region:

  ```bash theme={null}
  SRC=us-central1-docker.pkg.dev/genesis-471800/genesis-dist
  DST=$REGION-docker.pkg.dev/$PROJECT/genesis
  gcloud artifacts repositories create genesis --repository-format=docker --location=$REGION
  gcloud auth configure-docker us-central1-docker.pkg.dev,$REGION-docker.pkg.dev -q
  ```

  Then copy with [`crane`](https://github.com/google/go-containerregistry) — it streams
  registry-to-registry, so it needs no local Docker daemon, pulls nothing to disk, and
  keeps the source digest and every platform in the manifest:

  ```bash theme={null}
  for img in genesis:$APP_VERSION genesis-sandbox-claude:latest; do
    crane copy $SRC/$img $DST/$img
  done
  ```

  Mirror **both** images — the sandbox image is not part of the backend image, and it is
  not pulled until the first user opens a terminal, so a missing mirror surfaces days
  later. [Coding sandbox (CLI Terminal) image](#coding-sandbox-cli-terminal-image) covers
  pinning it.

  **No `crane`?** Docker does the same job by round-tripping through your machine. Pin the
  platform — an unpinned pull on an Apple-silicon laptop mirrors an `arm64` image that GKE
  nodes cannot run, and the failure surfaces later as a crash-looping pod, not a bad copy:

  ```bash theme={null}
  for img in genesis:$APP_VERSION genesis-sandbox-claude:latest; do
    docker pull --platform linux/amd64 $SRC/$img
    docker tag $SRC/$img $DST/$img
    docker push $DST/$img
  done
  ```

  There is no `gcloud artifacts docker images copy` — that subcommand does not exist, so
  reach for one of the two recipes above rather than the `gcloud` verb you might expect.
</Note>

### Pull the chart

```bash theme={null}
# auth as a databricks.com identity that holds the grant
gcloud auth print-access-token | \
  helm registry login us-central1-docker.pkg.dev -u oauth2accesstoken --password-stdin

helm show chart oci://us-central1-docker.pkg.dev/genesis-471800/genesis-dist/charts/genesis --version $CHART_VERSION
```

<Note>
  **Pin the chart version, and read the current one rather than trusting this page.**
  This runbook was last revised against chart `0.3.104` / app `v1.19.95`. Omitting
  `--version` installs whatever is newest, which makes two installs of "the same
  runbook" different deployments. Ask the registry what exists:

  ```bash theme={null}
  helm show chart oci://us-central1-docker.pkg.dev/genesis-471800/genesis-dist/charts/genesis \
    | grep -E '^version:|^appVersion:'
  ```

  Then pin what you read, and write it into the handover document.
</Note>

The chart tarball also carries its own render-time validators, which is the cheapest
way to confirm the chart version you just pinned still produces a correct, data-safe
manifest. Untar it once and keep it — Phase 5's gate uses it:

```bash theme={null}
helm pull oci://us-central1-docker.pkg.dev/genesis-471800/genesis-dist/charts/genesis \
  --version $CHART_VERSION --untar --untardir /tmp/genesis-chart
```

<Check>
  **Gate 2 — do not install until all three hold:**

  1. `helm show chart …` prints a `version:` and an `appVersion:` — your registry login
     works and the chart is reachable
  2. Both images resolve in the registry your **nodes** will pull from:
     `crane manifest <ref> > /dev/null && echo OK` for the backend **and** the workspace
     image
  3. `bash /tmp/genesis-chart/genesis/validate-retention.sh` exits `0` and ends with
     `Retention contract holds in the rendered manifest.`

  The validators render locally and never contact a cluster; invoke them with `bash`,
  because files inside a Helm tarball are not extracted executable.
</Check>

## Phase 3 — Install the base

A values file, a master key, one `helm upgrade --install`.

<Warning>
  **Set `image.tag` explicitly.** Left unset, `image.tag` defaults to the chart's
  `appVersion` (for chart `0.3.104` that is `v1.19.95`). That works only while that exact
  tag exists in the registry your nodes pull from — a mirror that copied a different tag,
  or a chart newer than the images you mirrored, fails the pull with `ImagePullBackOff`.
  Set `image.tag` to the release tag you mirrored or were given, and write it into the
  handover document.
</Warning>

<Warning>
  **A floating tag needs `image.pullPolicy: Always`.** The chart ships
  `image.pullPolicy: IfNotPresent`. If you deploy a tag that moves — `latest`, `main`, or
  any tag you re-point in your own mirror — a node that already has an image cached under
  that name does **not** re-pull it. The Pod restarts onto the old image and nothing
  reports an error: the upgrade is a silent no-op. Either pin an immutable release tag
  (`IfNotPresent` is then fine), or set:

  ```yaml theme={null}
  image:
    tag: latest            # or main — any floating tag
    pullPolicy: Always     # REQUIRED with a floating tag
  ```

  Either way, confirm which build is actually serving by its digest (Gate 3), not by the
  tag.
</Warning>

Create `my-values.yaml`. Replace every highlighted value; use your mirror path if you
mirrored.

```yaml theme={null}
image:
  repository: us-central1-docker.pkg.dev/genesis-471800/genesis-dist/genesis
  tag: v1.19.95        # REQUIRED — the release tag you mirrored or were given
  pullPolicy: IfNotPresent   # use Always if tag is a floating tag (latest, main)

persistence: { size: 50Gi, storageClassName: "" }   # "" = cluster default
resources:
  requests: { cpu: 4000m, memory: 16Gi }
  limits: {}          # none by design — avoids OOM kills

ingress:
  enabled: true
  className: nginx       # or "gce"
  hosts: [ { host: genesis.example.com, paths: [ { path: /, pathType: Prefix, port: 8080 } ] } ]
  tls: [ { secretName: genesis-tls, hosts: [ genesis.example.com ] } ]

config:
  authEnabled: "true"
  authProvider: "auth0"
  databricksHost: "https://your-workspace.cloud.databricks.com"
  databricksBaseUrl: "https://your-workspace.cloud.databricks.com/serving-endpoints"
  modelName: "databricks-claude-opus-5-5"
  extraEnv:
    AUTH_ISSUER: "https://YOUR-TENANT.us.auth0.com/"
    AUTH_AUDIENCE: "https://genesis-api"
    AUTH_JWKS_URL: "https://YOUR-TENANT.us.auth0.com/.well-known/jwks.json"
    AUTH0_CLIENT_ID: "YOUR-SPA-CLIENT-ID"
```

### Create the namespace, then install without `--create-namespace`

The chart renders its **own** `Namespace` object — named by the chart's `namespace`
value, `genesis` by default — and marks it `helm.sh/resource-policy: keep`, so that a
`helm uninstall` or a later chart change can never cascade a delete to the data volume.
Two consequences:

* `helm install --create-namespace` **fails**: the namespace Helm creates for the release
  collides with the one the chart renders.
* A plain `kubectl create namespace` **also fails** on install: Helm refuses to take over
  an object it does not own.

Pre-create the namespace and mark it for adoption by the release, then install with no
`--create-namespace`:

```bash theme={null}
kubectl create namespace genesis
kubectl annotate namespace genesis \
  meta.helm.sh/release-name=genesis meta.helm.sh/release-namespace=genesis
kubectl label namespace genesis app.kubernetes.io/managed-by=Helm
```

The release name (`genesis`), the `-n` namespace and the chart's `namespace` value must
all agree with those annotations. If you use a different namespace, set `namespace:` in
`my-values.yaml` to the same name and annotate that namespace instead.

```bash theme={null}
helm upgrade --install genesis \
  oci://us-central1-docker.pkg.dev/genesis-471800/genesis-dist/charts/genesis --version $CHART_VERSION \
  -n genesis -f my-values.yaml \
  --set secrets.genesisMasterKey="$(python3 -c 'import secrets;print(secrets.token_hex(32))')" \
  --set secrets.databricksApiKey="dapi…" \
  --wait --timeout 12m
```

<Note>
  **Optional — backend scheduling priority. Off by default; turn it on only on a cluster
  dedicated to Genesis.** By default the backend runs at Kubernetes' implicit priority 0,
  the same as the workspace and agent Pods it creates, so on a full cluster the scheduler
  has no reason to prefer it over the workloads that depend on it. On a cluster that runs
  only Genesis, add:

  ```yaml theme={null}
  priorityClass:
    create: true     # renders a PriorityClass and assigns it to the backend
    # preemptionPolicy: Never is the shipped default — keep it
  ```

  Creating the class needs rights on `scheduling.k8s.io` PriorityClasses. Leave it **off on
  a shared cluster**: a class above the default makes Genesis a preferred survivor over
  other teams' workloads, and changing `preemptionPolicy` to `PreemptLowerPriority` would
  let it evict them. To reference a class your platform team already defines, set
  `priorityClass.name` and keep `create: false`. It does not rank above Kubernetes' own
  system tiers, and is not meant to.
</Note>

<Note>
  **First boot takes up to 10 minutes.** The startup probe allows a long grace
  (`failureThreshold: 60` × `periodSeconds: 10`) while the backend builds its DB, warms
  sessions, and loads the LLM. Watch `kubectl -n genesis get pod genesis-0 -w`. To tell
  "still working" from "stuck", read the progress rather than the pod:

  ```bash theme={null}
  kubectl -n genesis exec genesis-0 -c genesis -- \
    curl -sS http://localhost:8080/api/health/startup
  # phase, progress and current_step — always 200, even mid-boot
  ```
</Note>

<Check>
  **Gate 3 — the base is installed when all four hold:**

  ```bash theme={null}
  kubectl -n genesis get pods
  # expect genesis-0  1/1  Running  — nothing ImagePullBackOff or CrashLoopBackOff

  kubectl -n genesis get pvc
  # expect one claim, STATUS Bound

  kubectl -n genesis exec genesis-0 -c genesis -- \
    curl -sS -o /dev/null -w '%{http_code}\n' http://localhost:8080/api/healthcheck
  # expect 200

  kubectl -n genesis get pod genesis-0 \
    -o jsonpath='{.status.containerStatuses[?(@.name=="genesis")].imageID}{"\n"}'
  # expect <registry>/genesis@sha256:<64 hex> — record this digest
  ```

  Then prove the Pod runs the build the tag names **now** — the check that catches a
  floating tag served from a stale node cache:

  ```bash theme={null}
  EXPECTED=$(crane digest <registry>/genesis:<your image.tag>)
  RUNNING=$(kubectl -n genesis get pod genesis-0 \
    -o jsonpath='{.status.containerStatuses[?(@.name=="genesis")].imageID}')
  [ "${RUNNING##*@}" = "$EXPECTED" ] && echo MATCH || echo "MISMATCH: running ${RUNNING##*@}"
  ```

  `ImagePullBackOff` here is the node-SA grant or a tag that is not in the registry;
  `Pending` on the PVC is the StorageClass row from Gate 1. A `MISMATCH` on a floating tag
  is `image.pullPolicy` left at `IfNotPresent`; set it to `Always` and delete the Pod so it
  is recreated. On a rolling upgrade, also make sure you are reading the **new** Pod, not
  the old one that is still terminating.
</Check>

## Phase 4 — Access and sign-in

One port serves everything; both surfaces authenticate through the same OIDC app.

* **`8080`** — the only port: GUI, API, terminal WS
* **1 URL** — browser and any client hit the same host
* **2 callbacks** — GUI origin + a loopback, in the OIDC app

Browse to `https://<your-host>/` and sign in. The OIDC app must list your host in
**Allowed Callback URLs** and **Web Origins**, and `AUTH0_CLIENT_ID` must be set (the
console reads it from `/api/config/frontend`). Register both callbacks:

```text theme={null}
https://genesis.example.com/            # the GUI origin
http://localhost:8765/callback         # the loopback sign-in for any CLI client
```

<Note>
  **No public ingress? Port-forward.** For a private sandbox, skip the Ingress and
  reach both surfaces locally:

  ```bash theme={null}
  kubectl -n genesis port-forward svc/genesis 8080:8080
  ```
</Note>

<Check>
  **Gate 4 — access works when all three hold:**

  1. `curl -sS -o /dev/null -w '%{http_code}\n' https://<your-host>/api/healthcheck` →
     `200` **from outside the cluster**. A `302` here is the ingress auth annotation
     capturing the probe — enable `ingress.authExemptHealth` rather than removing the
     auth.
  2. A browser at `https://<your-host>/` reaches the sign-in page and a real user
     completes the round trip back to the console.
  3. That user sends a chat message and gets a reply. Without a reachable LLM the server
     starts fine and thread creation fails with *"Bot not found or unavailable"* — which
     is an LLM-configuration answer, not a deployment one.

  If sign-in loops, the OIDC app is missing your host in **Allowed Callback URLs** or
  **Web Origins**, or `AUTH0_CLIENT_ID` is unset — the console reads it from
  `/api/config/frontend`.
</Check>

## Phase 5 — Enable the sandbox terminal

`cliTerminal.enabled=true` is the whole switch: it renders the cluster objects the
backend needs **and** turns the feature on in the application. Move 2 covers the cluster
objects; Move 1 is the sizing and image settings the chart does not decide for you.
Read the image section first — it is the setting most often wrong.

### Coding sandbox (CLI Terminal) image

The coding sandbox — the per-user workspace Pod behind the CLI Terminal — runs a
**separate image** from the main Genesis image. Upgrading, mirroring or pinning the
backend does nothing for it: it has to be mirrored and pinned in its own right.

* **The feature is off by default.** Nothing pulls the sandbox image until
  `cliTerminal.enabled=true` and a user opens a terminal — which is why a missing or
  wrong image surfaces with the first user, not at install.
* **Where to get it.** Use the sandbox image from the registry Genesis provided you — for
  this install, `genesis-dist/genesis-sandbox-claude` — or your mirror of it. The same
  image is also published publicly on Docker Hub as
  `docker.io/genesiscomputingai/genesis-sandbox-claude`, pullable without credentials, with
  the same weekly `latest` and dated tags. It contains the coding tools only, no Genesis
  server code, so it can serve clusters that have no access to a private registry.
* **The reference must be fully qualified.** On Kubernetes, `cliTerminal.image` (the app
  setting `cli_terminal.image`, env `GENESIS_CLI_TERMINAL_IMAGE`) must name the registry
  host, repository and tag, in a registry your **nodes** can pull from:
  `us-central1-docker.pkg.dev/<project>/<repo>/genesis-sandbox-claude:<tag>` or
  `docker.io/genesiscomputingai/genesis-sandbox-claude:<tag>`. The
  application's built-in default is an unqualified name, which works for a local Docker
  install and cannot work here: Kubernetes resolves an unqualified name against Docker
  Hub's `library/` namespace, where the image does not exist, and every workspace Pod
  sits in `ImagePullBackOff`.
* **Pin it.** `genesis-sandbox-claude:latest` is rebuilt and refreshed **weekly**, and
  each build is also published under a dated, immutable tag (for example
  `20260923-35908082637-1`). Pin one of the dated tags — list them with
  `crane ls $SRC/genesis-sandbox-claude` — and record its digest beside the backend's.
  Workspace Pods are created with Kubernetes' default pull policy, so a re-pointed tag
  other than `latest` is not re-pulled on a node that already has it cached.
* **Mirror it multi-arch, with `crane`.** `crane cp` copies registry-to-registry and keeps
  every platform in the manifest. A `docker pull` / `docker push` round trip from a laptop
  can publish only the laptop's architecture — on Apple silicon, an `arm64`-only image
  that `amd64` GKE nodes cannot run:

  ```bash theme={null}
  SANDBOX_TAG=20260923-35908082637-1   # a dated tag from `crane ls`; never moves
  crane cp $SRC/genesis-sandbox-claude:$SANDBOX_TAG $DST/genesis-sandbox-claude:$SANDBOX_TAG
  crane digest $DST/genesis-sandbox-claude:$SANDBOX_TAG      # record this
  crane config $DST/genesis-sandbox-claude:$SANDBOX_TAG | grep -o '"architecture":"[^"]*"'
  # expect "architecture":"amd64" — the platform your GKE nodes run
  ```

Then set the qualified reference in exactly one place — `cliTerminal.image` in the
values file, or `GENESIS_CLI_TERMINAL_IMAGE` in `config.extraEnv` (Move 1) with
`cliTerminal.image` left empty. The warning under Move 1 explains which wins.

<Warning>
  **Enabled ≠ switched on, if you set this up by hand.** The two halves — the backend's
  own config, and the namespace/RBAC/quota it acts through — are both required. Config
  alone → the backend can't create Pods and the terminal 403s. RBAC alone → the feature
  stays off. `cliTerminal.enabled=true` does both at once; a hand-applied manifest does
  only the second.
</Warning>

### Move 1 — the settings the chart does not decide

Add to `config.extraEnv` in `my-values.yaml` and re-run `helm upgrade` (use your
mirror path for the image if you mirrored):

```yaml theme={null}
    GENESIS_CLI_TERMINAL_IMAGE: "us-central1-docker.pkg.dev/genesis-471800/genesis-dist/genesis-sandbox-claude:latest"   # FULLY QUALIFIED; your pinned mirror if you mirrored
    GENESIS_CLI_TERMINAL_MEMORY: "8g"          # CEILING; hard cap, 4g OOM-kills under load
    GENESIS_CLI_TERMINAL_MEMORY_REQUEST: "4g"  # what each workspace RESERVES; capacity ask
    GENESIS_CLI_TERMINAL_CPU_REQUEST: "500m"   # no CPU limit is set, so this is scheduling only
    GENESIS_CLI_TERMINAL_SHM_SIZE: "1Gi"       # /dev/shm for headless Chrome
    GENESIS_CLI_TERMINAL_HOME_VOLUME_SIZE: "10Gi"   # ×2 PVCs = 20Gi/user
    GENESIS_CLI_TERMINAL_MODEL: "databricks-claude-opus-5-5"
```

<Warning>
  **Do not put `GENESIS_CLI_TERMINAL_ENABLED`, `_RUNTIME` or `_K8S_NAMESPACE` here.**
  `cliTerminal.enabled=true` sets all three on the container itself and force-locks their
  effective values, so they override values previously stored in Advanced Settings as well
  as the `envFrom` ConfigMap that carries `config.extraEnv` — an entry here of those names
  is silently ignored. That is deliberate: each of the three has to agree with
  an object the chart rendered, and a disagreement fails as a 403 or a disabled feature
  rather than as a visibly bad value. With `ecr.enabled` the same applies to
  `GENESIS_CLI_TERMINAL_IMAGE` and `_IMAGE_PULL_SECRET`, because the chart is what knows
  the registry.

  **On GKE `ecr.enabled` is false**, so the image is yours. Two supported places, and
  exactly one of them: `config.extraEnv` above (leave `cliTerminal.image` empty), or
  `cliTerminal.image` in the values file — which is empty by default precisely so it cannot
  mask a working `extraEnv` value, and which wins when you do set it.
</Warning>

<Note>
  **Request and limit are different numbers on purpose.** `MEMORY` is the ceiling a
  workspace may grow into and the point at which Kubernetes OOM-kills it — which kills
  every session that user has open. `MEMORY_REQUEST` is what the scheduler reserves, and
  so what cluster capacity must actually supply: 10 concurrent users need 10 × the
  request, not 10 × the ceiling. Setting them equal makes every idle workspace hold the
  full 8Gi, which is what made this mode need a specially sized cluster.
</Note>

<Note>
  **Turning this on does not require a Helm restart.** Two delivery paths reach the same
  settings — pick by whether you want the state in your values file or in the running
  server:

  * **IaC (what this step shows).** Values in `config.extraEnv` / `genesisConfig.content`,
    applied with `helm upgrade`. Best for a reproducible deployment: the enablement lives
    in the values file and survives a reinstall.
  * **In-flight.** On a **running** server an admin sets the same settings in **Config →
    Advanced Settings**, and they take effect on the **next terminal attach** — no restart,
    no rollout. Every `cli_terminal.*` field is declared `env_leaf: true` in the config
    schema: a panel write refreshes that field's environment variable immediately, and the
    attach path reads the variable at call time rather than caching it at startup.

  **The prerequisite the panel cannot satisfy is Move 2.** The namespace, RBAC and quota
  are a one-time cluster grant. Flipping `enabled` in Advanced Settings without them still
  403s, exactly as it would from the values file.

  `cliTerminal.enabled=true` is the chart's platform switch: it pins the effective
  `cli_terminal.enabled`, `cli_terminal.runtime`, and `cli_terminal.k8s_namespace` values to
  the rendered Kubernetes objects. Setting it to `false` removes those chart-managed objects
  from the release, but deliberately does not force the application's own setting to `false`;
  deployments that provision the namespace and RBAC outside this chart can keep working.

  **What a live change reaches.** `image` and the CPU / memory / volume sizes apply to
  workspace Pods created *after* the change — a user with a running workspace keeps their
  current Pod until it is reaped and recreated, so nothing interrupts an agent mid-task.
  `runtime` and `k8s_namespace` are the exception: both are fixed for the life of the
  server process, and a live change to either is logged and ignored rather than applied,
  because the workspaces already created live on the old backend and in the old namespace
  and nothing else can find them to reap. Change those from the values file, and retire the
  existing workspaces first.
</Note>

### Move 2 — grant the cluster objects

`cliTerminal.enabled=true` in your values file renders the namespace, RBAC, quota and
(optionally) the workspace NetworkPolicy as part of the release, and sets plus force-locks
`GENESIS_CLI_TERMINAL_ENABLED`, `_RUNTIME` and `_K8S_NAMESPACE` on the backend for you.
This one flag is what the old `kubectl apply` step and the enablement env vars used to be
between them:

```yaml theme={null}
cliTerminal:
  enabled: true
  namespace: "genesis-cli"
  image: ""                                # empty = extraEnv/APP_CONFIG owns it (see below)
  maxWorkspaces: 25                        # renders the quota AND the backend's own cap
  maxLifetimeUsers: 50                     # people who may EVER attach; claims outlive Pods
  networkPolicy:
    enabled: false                         # see the egress warning below
```

The chart binds the ServiceAccount the backend **actually** runs as — `default` unless
`serviceAccount.create` or `serviceAccount.name` is set — so there is nothing to edit for
the common case.

<Note>
  **This namespace survives `helm uninstall`, and survives turning the feature back off.**
  It holds every user's home volume and therefore their saved Claude Code conversations, so
  it carries `helm.sh/resource-policy: keep`. Without that, an ordinary `helm upgrade`
  setting `cliTerminal.enabled: false` would take it out of the release manifest and
  Kubernetes would cascade the delete to everyone's work — an enablement flag doubling as a
  data-destruction flag. Disabling revokes the RBAC and releases the quota, nothing more.
  A chart-created workspace namespace is labeled with the Genesis namespace of the backend
  instance that owns it; the cleanup scripts use that label rather than a release's computed
  values, which prevents one instance from draining another instance's retained workspaces.

  Removal is deliberate and separate: `cluster/scripts/teardown.sh <name>   --delete-workspaces`, which asks before it does it. To retire one departed user instead of
  everyone, see
  [Operations Runbook › Offboarding](/docs/deployments/Genesis-Operations-Runbook#offboarding-the-step-everyone-misses).
  Use
  `cliTerminal.createNamespace: false` when the namespace already exists or is managed
  elsewhere — that flag is about creation, not survival.

  **Disabling is access-only.** The workspace Pods are retained with the namespace, and
  `enabled: false` removes the reaper that used to remove them along with the RBAC that let
  it act — so an agent mid-task keeps running. Drain first, which keeps every volume:

  ```bash theme={null}
  cluster/scripts/drain-workspaces.sh <release>
  ```

  A separate script from `teardown.sh` on purpose: it leaves the release running, which is
  what makes it safe to run before you disable.
</Note>

<Note>
  **If your platform objects are applied outside Helm**, render them from the same
  template rather than keeping a hand-edited copy — a second copy diverges silently,
  because only one of the two ever gets updated:

  ```bash theme={null}
  helm template genesis <chart> \
    --set cliTerminal.enabled=true \
    --show-only templates/cli-terminal-rbac.yaml | kubectl apply -f -
  ```
</Note>

<Warning>
  **Workspace egress is unrestricted by default.** A workspace gets the same network
  access as any other Pod in the cluster — the same access the Genesis server has. That is
  deliberate: these are development workspaces and installing packages and cloning
  repositories is the point of them. `cliTerminal.networkPolicy.enabled=true` confines
  them to DNS plus the Genesis backend, at the cost of `git`/`gh`, npm and pip timing out.

  A NetworkPolicy is only as good as your CNI: Kubernetes accepts the object whether or
  not the CNI enforces it. GKE enforces it; a stock EKS cluster with the default VPC CNI
  was measured applying the same policy cleanly and enforcing nothing. Verify before
  relying on it:

  **Run the control first.** A blocked connection proves nothing on its own: on a
  locked-down VPC the cluster may have no egress to that address at all, so a probe that
  fails tells you the destination was unreachable, not that the policy blocked it. Two
  Pods, identical except for the label the policy selects on:

  ```bash theme={null}
  # 1. CONTROL — no workspace label, so the policy does not select it.
  kubectl -n genesis-cli run probe-control --rm --attach --restart=Never \
    --image=busybox --command -- \
    sh -c 'if nc -z -w 5 1.1.1.1 443; then echo CONTROL-REACHED; else echo CONTROL-BLOCKED; fi'

  # 2. SUBJECT — identical, but labelled as a workspace.
  kubectl -n genesis-cli run probe-subject --rm --attach --restart=Never \
    --image=busybox --labels=app=genesis-cli-terminal --command -- \
    sh -c 'if nc -z -w 5 1.1.1.1 443; then echo SUBJECT-REACHED; else echo SUBJECT-BLOCKED; fi'
  ```

  Only `CONTROL-REACHED` **plus** `SUBJECT-BLOCKED` means enforced — the label is the
  only difference, so the policy is what blocked it. `CONTROL-REACHED` with
  `SUBJECT-REACHED` means the policy is decorative. **`CONTROL-BLOCKED` makes the run
  inconclusive**, whatever the subject said: the cluster cannot reach that destination
  regardless of the policy. And silence from either command is never a pass — a Pod that
  was not scheduled or could not pull `busybox` prints nothing at all.

  The full verdict table is in
  [Operations Runbook › Incident triage](/docs/deployments/Genesis-Operations-Runbook#incident-triage).
</Warning>

Capacity is not free and the chart cannot create it. At the shipped quota this mode asks
for roughly **12.5 CPU and 100Gi of reserved memory** (25 workspaces reserving 500m/4Gi
each, each able to grow to 8Gi), on top of the Genesis release's own 4 CPU / 16Gi. Disk is
separate and accumulates differently: 20Gi per user in concurrent use, and the claim quota
allows up to 1000Gi over the life of the deployment, because a volume outlives its Pod so
a user's conversations survive being reaped. If the cluster cannot supply it, workspaces sit `Pending` and
every attach fails, which reads as a broken feature rather than a starved one. Lower
`cliTerminal.maxWorkspaces` to what the cluster holds — one value, which the chart writes
to both the namespace quota and the backend's own cap. Sizing is in the
[capacity formula](#capacity-formula) above; the scheduling traps that do not look like
capacity problems — zone-pinned volumes, a node pool refused for disk quota rather than
CPU — are in
[Operations Runbook › Scaling](/docs/deployments/Genesis-Operations-Runbook#scaling).

<Check>
  **Gate 5 — the terminal is genuinely on when all four hold.** Skip any of these and
  the feature fails for the first real user rather than for you:

  Read all three identities off the live deployment first. The workspace namespace and
  the backend's service account are both configurable, so a gate that hardcodes them
  checks a different deployment than the one you installed — and on a cluster with more
  than one Genesis instance, possibly somebody else's:

  ```bash theme={null}
  # The namespace the backend RUNS in (values.namespace when set, else the release's)
  BACKEND_NS=$(kubectl get statefulset -A -l app.kubernetes.io/instance=genesis \
    -o jsonpath='{.items[0].metadata.namespace}')
  POD=$(kubectl -n $BACKEND_NS get pod -l app.kubernetes.io/instance=genesis \
    -o jsonpath='{.items[0].metadata.name}')

  # The identity the backend ACTUALLY runs as. Read it from the live Pod, which
  # Kubernetes always fills in — the chart leaves it unset on the defaults, and the
  # Pod then runs as `default`, so assuming the chart's own name checks nobody.
  BACKEND_SA=$(kubectl -n $BACKEND_NS get pod $POD -o jsonpath='{.spec.serviceAccountName}')

  # The workspace namespace the backend WILL ASK IN — the value the chart pinned onto
  # the container, not the chart's default.
  WS_NS=$(kubectl -n $BACKEND_NS get pod $POD -o jsonpath=\
  '{.spec.containers[?(@.name=="genesis")].env[?(@.name=="GENESIS_CLI_TERMINAL_K8S_NAMESPACE")].value}')

  echo "backend=$BACKEND_NS/$POD sa=$BACKEND_SA workspaces=$WS_NS"
  ```

  ```bash theme={null}
  # 1. The chart rendered the platform, in the namespace the backend will ask in
  kubectl -n $WS_NS get role,rolebinding,resourcequota
  # expect one of each

  # 2. That exact identity may actually create workspace Pods there
  kubectl auth can-i create pods -n $WS_NS \
    --as=system:serviceaccount:$BACKEND_NS:$BACKEND_SA
  # expect: yes

  # 3. The chart still renders this feature correctly at the version you pinned
  bash /tmp/genesis-chart/genesis/validate-cli-terminal.sh
  # expect exit 0, ending: In-pod coding agent mode renders correctly.
  ```

  An empty `$WS_NS` means the chart pinned nothing, which means `cliTerminal.enabled` is
  not actually `true` in the release you installed — fix that before reading the rest of
  this gate.

  4. **Attach a real terminal as a real user**, from the console. The first attach pulls
     the \~700 MB workspace image and can take up to a minute; after that it is instant.
     Nothing short of a shell prompt counts — a rendered tab is not an attached session.

  If the socket closes instead, the reason names the cause: *"CLI terminal access is
  disabled on this deployment"* is the effective config, *"Access denied"* is the user's
  `use_cli_terminal` permission, and a Kubernetes 403 after the socket attaches is the
  namespace or RBAC. The full symptom → cause → fix table is in the
  [Operations Runbook](/docs/deployments/Genesis-Operations-Runbook#incident-triage).
</Check>

## Phase 6 — Final verification

Confirm the deploy by content, then hand the console to a user.

<Check>
  **Gate 6 — the deployment is ready to hand over when all six hold.** Five of these
  are cheap; the fifth is the one installs skip, and it is the one that silently ships a
  broken deployment.

  1. **Health, from outside.** Expect `200`:

     ```bash theme={null}
     curl -sS -o /dev/null -w '%{http_code}\n' https://<host>/api/healthcheck
     ```

  2. **No stuck pulls, in either namespace.** `kubectl get pods -n genesis` and
     `kubectl get pods -n genesis-cli` — nothing in `ImagePullBackOff`,
     `CrashLoopBackOff` or `Pending`. A stuck pull here is the node-SA row from Gate 1
     or a tag that is not in the registry.

  3. **Console.** `https://<host>/` signs a real user in, and that user gets a reply to
     a chat message. Admins also get **Pulse → CLI Workspaces** (`/pulse/cli-workspaces`)
     to list running workspace Pods with restart and cancel.

  4. **Terminal.** A user opens a terminal and gets a shell prompt. The first attach
     pulls the \~700 MB image and can take a minute; after that it is instant. The Pod is
     reaped when idle and the volume persists, so reconnecting restores their work.

  5. **Embeddings, verified separately from chat.** Chat replying proves nothing about
     semantic search — they are different routes and often different endpoints and
     credentials. Force a real embedding call:

     ```bash theme={null}
     curl -sS -X POST https://<host>/api/config/llm-providers/embedding-config/test \
       -H "Authorization: Bearer <token>" -H 'Content-Type: application/json' -d '{}'
     # expect {"success": true, ..., "dimensions": <n>}
     ```

     `"success": false` carries the provider's own error text. Do not sign off on a
     status chip that merely is not red.

  6. **Recorded.** The chart version and the running **image digest** are written into
     the handover document, along with the escrow owner for `GENESIS_MASTER_KEY`.
</Check>

Then hand over the [Operations Runbook](/docs/deployments/Genesis-Operations-Runbook),
which is what the customer's operator will need at 02:00.

## When it goes wrong

**The failure-mode catalogue lives in one place** — symptom → cause → fix for every
failure below, in
[Operations Runbook › Incident triage](/docs/deployments/Genesis-Operations-Runbook#incident-triage).
It is not repeated here, because a description kept in two places goes stale in one of
them. What this page keeps is the GCP-specific *remedy* detail each symptom points at.

Skim this index before you start; each row names the phase where the cause is fixed.

| Symptom | Phase | GCP remedy on this page |
| - | - | - |
| `0 nodes out of 1 have registered`, or `kube-system` pods stuck `ContainerCreating` with `CSINode` at 0 drivers and unbound PVCs | 1 | The node-egress warning — tag-scoped route, Private Google Access, or Cloud NAT, plus the effective-firewall check. **This is the expensive one**: `kubectl get pods -n kube-system` first; a minute there saves a day. |
| Backend `Pending` on a volume that never attaches, mid-install | 1 | Install on on-demand nodes; a Spot preemption wedges PVC binding and reads as a storage fault. |
| Workspace Pods `ImagePullBackOff` | 1, 5 | The node-SA row — a node pulls as its **own** service account, which a databricks.com grant does not cover — and the [sandbox image](#coding-sandbox-cli-terminal-image) reference: an unqualified name resolves to Docker Hub `library/` and fails; an `arm64`-only mirror fails on `amd64` nodes. |
| Backend `ImagePullBackOff` | 3 | `image.tag` unset (it defaults to the chart's `appVersion`) or naming a tag that is not in the registry the nodes pull from. |
| An upgrade "succeeded" but the old build is still serving | 3 | A floating tag (`latest`, `main`) with `image.pullPolicy: IfNotPresent`. Set `Always`, delete the Pod, and compare the running digest (Gate 3). |
| `helm install` fails on the `Namespace`, or with `--create-namespace` | 3 | Pre-create the namespace with the Helm adoption annotations and label, and drop `--create-namespace`. |
| NetworkPolicy applied cleanly and enforces nothing | 1 | Dataplane V2 is creation-time only; retrofit Calico with `gcloud container clusters update … --enable-network-policy`. Note the lockdown is **off by default**. |
| Terminal returns a 403 or "disabled" though the chart rendered everything | 5 | Gate 5 — the close reason distinguishes effective config, the user's `use_cli_terminal` permission, and a Kubernetes 403 on workspace creation. |
| Chat works, semantic search silently degrades | 6 | Gate 6 item 5 — verify the embedding endpoint independently of chat. |
| `helm upgrade` fails with `invalid ownership metadata` | 5 | The migration note — set `cliTerminal.createNamespace=false` and delete only the Role, RoleBinding, quota and policy. **Never** the namespace. |
| "Reset to defaults" reports off while the terminal stays live | — | Version-gated; fixed in current builds. Upgrade the image, or verify the real state after a reset rather than trusting the panel. |

Two facts that are not symptoms but are worth knowing before you tear anything down:

* **PVCs are retained on uninstall, by design.** The backend volume and every workspace
  volume survive `helm uninstall`; workspace volumes are per-user home directories and
  nothing auto-deletes them. Reclaiming them is deliberate and explicit — see
  [Teardown without data loss](/docs/deployments/Genesis-Operations-Runbook#teardown-without-data-loss).
* **Chart version matters for the retention contract.** Helm reads the retention policy
  from the *live* object, so it protects nothing until an upgrade has written it. On a
  deployment that enabled the terminal on a chart older than `0.3.83`, going straight to
  `cliTerminal.enabled: false` in one `helm upgrade` still deletes the namespace. Upgrade
  with the feature still **on** first, then disable in a second step. Deployments enabling
  it for the first time on `0.3.83` or later are never exposed to this.

## Connecting remotely to the session

To watch or drive a Claude Code session managing this cluster from another device, or
to coordinate two sessions inside the same pod, see
[Remote Connections to Claude Code](/docs/deployments/Remote-Claude-Connections) —
including the caveat that Remote Control does **not** work with the in-pod `claude`
that routes its LLM through the Genesis proxy.

***

Genesis on GKE · last revised against chart `0.3.104` / app `v1.19.95` — use the chart
version listed for your release, confirm it with `helm show chart`, and pin what you read ·
images + chart in
`us-central1-docker.pkg.dev/genesis-471800/genesis-dist`. Replace every highlighted
placeholder with your values. Preflight (Phase 1) is safe to run against any cluster
before you commit. Day-2 operations:
[Genesis Operations Runbook](/docs/deployments/Genesis-Operations-Runbook).