> ## Documentation Index
> Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Deploying Genesis - Kubernetes on AWS

> Deploy the Genesis container to an existing AWS Elastic Kubernetes Service cluster

This guide walks you through deploying the Genesis container to an existing AWS EKS (Elastic Kubernetes Service) cluster.

<Note>
  **For everything after the install** — upgrade, rollback, backup and restore, secrets, user offboarding, scaling, teardown, and the failure-mode catalogue — see the [Genesis Operations Runbook](/docs/deployments/Genesis-Operations-Runbook). It is cloud-agnostic, with a per-cloud section for the EKS-specific substrate. EBS snapshot procedures stay in [Backup and Recovery](/docs/deployments/AWS/Backup-And-Recovery).
</Note>

<Note>
  Genesis will provide you with the container image URL. Replace `<container-registry-url>` throughout this document with the URL provided by Genesis. Helm charts are published as private OCI artifacts on Docker Hub (`oci://registry-1.docker.io/genesiscomputingai/genesis`), ECR (`oci://<account>.dkr.ecr.<region>.amazonaws.com/charts/genesis`), and GHCR (`oci://ghcr.io/genesis-computing-ai/charts/genesis`). Docker Hub and ECR credentials are provided during onboarding; GHCR access uses a GitHub PAT with `read:packages`. If you use the bundled `aws-eks/` helper scripts, copy `aws-eks/eks-config-example.env` to `aws-eks/eks-config.env`, set `ECR_REGISTRY`, and ensure `AWS_DEFAULT_REGION` matches the ECR region so `aws ecr get-login-password` succeeds. Pin `image.tag` in your values file, and set `GENESIS_IMAGE_TAG` in `aws-eks/eks-config.env` only when you want a one-off override for a deploy.
</Note>

## Prerequisites

### AWS EKS Cluster Requirements

<CardGroup cols={2}>
  <Card title="EKS Cluster" icon="cube">
    Kubernetes version 1.23 or higher
  </Card>

  <Card title="kubectl" icon="terminal">
    Configured to access your EKS cluster
  </Card>

  <Card title="Helm 3.8+" icon="dharmachakra">
    For deploying the Genesis Helm chart
  </Card>

  <Card title="AWS CLI" icon="cloud">
    Configured with appropriate credentials
  </Card>
</CardGroup>

**Verify cluster access:**

```bash theme={null}
aws eks update-kubeconfig --name <cluster-name> --region <region>
kubectl get nodes  # Verify access
```

**Verify Helm installation:**

```bash theme={null}
helm version
```

### Additional Requirements

* **EBS CSI Driver**: Required for persistent storage (see [EBS CSI Driver Setup](#ebs-csi-driver-setup))
* **Ingress Controller**: For external access (see [Ingress Controller Setup](#ingress-controller-setup))
* **`metrics-server` and VPA controller (optional)**: Required only if you enable the chart's `vpa` settings (see [Vertical Pod Autoscaler (Optional)](#vertical-pod-autoscaler-optional))

### Network Access

| Requirement | Purpose |
| - | - |
| Outbound Internet | Cluster nodes must pull container images from the registry URL provided by Genesis |
| DNS Resolution | For accessing external APIs (OpenAI, Databricks, etc.) if using those LLM providers |

### IAM Permissions

The EKS cluster node group IAM role needs the following permissions:

* `AmazonEKSWorkerNodePolicy`
* `AmazonEKS_CNI_Policy`
* `AmazonEC2ContainerRegistryReadOnly`

<Info>
  If using **AWS Bedrock** as the LLM provider, use IAM Roles for Service
  Accounts (IRSA) for production least-privilege access. Granting Bedrock
  permissions to the node group role can work for simple non-production
  deployments, but it gives every workload on that node role the same access.
</Info>

## EBS CSI Driver Setup

Genesis requires persistent storage for its database, git repositories, and uploaded files. The Amazon EBS CSI driver must be installed in your cluster to provision EBS volumes.

<Warning>
  The EBS CSI driver is **not automatically installed** by default on standard EKS clusters (even for Kubernetes 1.23+). You must install it manually. However, if you're using **EKS Auto Mode**, the EBS CSI driver is automatically included and no installation is needed.
</Warning>

**Check if installed:**

```bash theme={null}
kubectl get pods -n kube-system | grep ebs-csi
```

<Card title="Installation Instructions" icon="book" href="https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html">
  Refer to the Amazon EKS User Guide for detailed EBS CSI driver installation steps, including IAM role setup and add-on installation.
</Card>

## Ingress Controller Setup

Genesis requires an ingress controller for external access. Choose one based on your requirements:

<Tabs>
  <Tab title="AWS Load Balancer Controller (Recommended)">
    The AWS Load Balancer Controller creates Application Load Balancers (ALB) or Network Load Balancers (NLB) for ingress resources.

    **Install AWS Load Balancer Controller:**

    ```bash theme={null}
    # Add the EKS chart repository
    helm repo add eks https://aws.github.io/eks-charts
    helm repo update

    # Install the controller
    helm install aws-load-balancer-controller eks/aws-load-balancer-controller \
      -n kube-system \
      --set clusterName=<cluster-name> \
      --set serviceAccount.create=false \
      --set serviceAccount.name=aws-load-balancer-controller
    ```

    **Verify installation:**

    ```bash theme={null}
    kubectl get deployment -n kube-system aws-load-balancer-controller
    ```
  </Tab>

  <Tab title="NGINX Ingress Controller">
    **Install NGINX Ingress:**

    ```bash theme={null}
    kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/main/deploy/static/provider/cloud/deploy.yaml

    # Wait for it to be ready
    kubectl wait --namespace ingress-nginx \
      --for=condition=ready pod \
      --selector=app.kubernetes.io/component=controller \
      --timeout=90s
    ```
  </Tab>
</Tabs>

<Note>
  **Single-instance and load balancer choice:** Genesis runs as a single pod (`genesis-0`) and does not support multiple instances. ALB/NLB therefore do not add value for *load distribution* across replicas. They still add value for: exposing the app with a stable external endpoint, TLS termination (e.g. ACM with ALB), health checks, and integration with Route53, WAF, etc. You can also expose Genesis using a `LoadBalancer`-type Service (one NLB/ELB per service) if you prefer not to use an ingress controller.
</Note>

## Genesis Container Configuration

### Container Image

| Setting | Description |
| - | - |
| Repository | Genesis will provide the container image URL (e.g., `<container-registry-url>/genesis`) |
| Tags | Genesis will specify the appropriate tag to use (for example `v1.18.2`, `edge`, or another specific version) |
| Registry Access | Genesis will provide details about authentication requirements, if any |

<Info>
  Current Genesis container images include `ripgrep` (`rg`) out of the box for in-container file-content searches. After `kubectl exec`, you can use it to inspect mounted files under `/app/bot_storage` or `/app/bot_git` without installing extra packages. The same images also bundle the in-repo `genesis_sdk` package under `/app/genesis_sdk` (for `execute_python_code` local execution — not installable from PyPI) and the `llm_wiki` package under `/app/llm_wiki`, so the SDK, wiki APIs, and tools shipped with Genesis are available without extra image customization.
</Info>

### Image Pull Secrets (If Required)

If the container registry requires authentication, Genesis will provide the necessary credentials. Create a Kubernetes secret:

```bash theme={null}
# Create namespace first if it doesn't exist
kubectl create namespace genesis

# Create Docker registry secret (if credentials provided by Genesis)
kubectl create secret docker-registry genesis-registry-credentials \
  --docker-server=<container-registry-url> \
  --docker-username=<username> \
  --docker-password=<password> \
  --docker-email=<email> \
  -n genesis
```

Then reference it in your Helm values file:

```yaml theme={null}
imagePullSecrets:
  - name: genesis-registry-credentials
```

### Exposed Ports

| Port | Service | Description |
| - | - | - |
| 8080 | FastAPI | Primary application with React GUI and modern APIs |
| 8082 | Flask | UDF proxy, OAuth, external integrations |
| 8501 | Streamlit | Legacy Streamlit interface |

### Persistent Storage Requirements

Genesis requires persistent storage for:

| Path | Purpose |
| - | - |
| `/app/.genesis/db/genesis.db` | SQLite database |
| `/app/bot_git` | Cloned git repositories |
| `/app/bot_storage` | Uploaded files and bot storage |
| `/app/tmp` | Runtime temp files |

<Info>
  **Recommended Storage Size:** 50Gi minimum (adjust based on expected usage)

  **Storage Class:** Uses cluster default (typically `gp2` or `gp3` on EKS)
</Info>

<Warning>
  Before enabling Claude Code on EKS, review the worktree and run-evidence
  persistence notes in [Cursor and Claude Code coding backends](/docs/Setup/Cursor#claude-code-backend).
</Warning>

### Environment Variables

<AccordionGroup>
  <Accordion title="Required Configuration">
    | Variable | Default | Description |
    | - | - | - |
    | `SQLITE_DB_PATH` | `/app/.genesis/db/genesis.db` | Path to SQLite database |
    | `DATABASE_URL` | `sqlite:///app/.genesis/db/genesis.db` | SQLite connection string |
  </Accordion>

  <Accordion title="AWS Bedrock Configuration">
    The active LLM provider belongs to `APP_CONFIG`. Configure Bedrock after the
    pod is running with the config API or **Config -> AI Providers** — see
    [Configuring a provider through the config API](/docs/Setup/Setup-AI-Model#configuring-a-provider-through-the-config-api)
    and the worked example under [Deployment Steps](#deployment-steps) below.

    The CloudFormation `BedrockModelId` parameter and Terraform
    `bedrock_model_id` input default to `""`. Leave them empty to use the
    Bedrock catalog defaults, including Claude Opus 5.5
    (`global.anthropic.claude-opus-5-5`) for chat, or set a model id to pin it.
    The Terraform AWS and ECS templates seed an unset provider configuration with
    `ACTIVE_LLM_PROVIDER=litellm_bedrock` and `BEDROCK_AUTH_MODE=iam_role`.
    They also set `BEDROCK_ENABLED=true` so images through `v1.19.95` recognize
    the provider when no configured marker is stored; newer images ignore this
    switch. A non-empty model input also seeds `BEDROCK_MODEL_NAME`,
    `BEDROCK_FAST_MODEL_NAME`, and `BEDROCK_PREMIER_MODEL`. When the input is
    empty, those templates omit the model overrides. Saved configuration or a
    mounted `genesis-config.yaml` takes precedence over these deployment
    defaults.

    ```yaml theme={null}
    AWS_REGION: "<aws-region>"          # standard AWS SDK region for the pod
    AWS_DEFAULT_REGION: "<aws-region>"  # SDK fallback for the same
    ```

    These are the standard AWS SDK region variables that boto3 reads; they do not
    select or configure a provider. Bedrock's own region is
    `aws_region` in the provider configuration.

    Recommended: use IRSA for the Genesis service account, so no API key is stored
    at all — configure Bedrock with `"auth_mode": "iam_role"` and the pod's role
    supplies the credential.
  </Accordion>

  <Accordion title="Authentication (Optional)">
    | Variable | Value | Description |
    | - | - | - |
    | `AUTH_ENABLED` | `"true"` | Enable authentication |
    | `AUTH_PROVIDER` | `"proxy"` | For ingress-based auth |
  </Accordion>

  <Accordion title="Git repository runtime">
    Genesis prepares `GIT_PATH` on startup and ensures `repos/`, `worktrees/`,
    and `.genesis_runtime/` exist before shared Git-backed workflows use the
    mount. The shared group-write repair starts in the background so slow
    recursive permission fixes do not block container startup.

    | Variable | Default | Description |
    | - | - | - |
    | `GIT_PATH` | `/app/bot_git` | Mounted Git workspace Genesis prepares at startup. |
    | `GENESIS_BOT_GIT_SHARED_GROUP` | `cortex_sandbox` | Shared Linux group Genesis applies when repairing the mounted Git tree so repositories remain group writable. The repair starts in the background during container startup. |
    | `GENESIS_TRUST_ALL_GIT_DIRS` | `true` | When `true`, Genesis adds `*` to system Git `safe.directory` during startup. Set to `false` only when you manage Git trust outside Genesis. |
  </Accordion>

  <Accordion title="Logging">
    | Variable | Options | Description |
    | - | - | - |
    | `LOG_LEVEL` | `DEBUG`, `INFO`, `WARNING`, `ERROR` | Log verbosity |
    | `LOGS_FORMAT` | `json`, `text` | Log output format |
  </Accordion>
</AccordionGroup>

### Inbound SCIM for AWS/EKS deployments

SCIM is optional. Enable it only when you want an external IdP to provision
Genesis users and role membership automatically. Genesis acts as a SCIM 2.0
**service provider**, so the customer's IdP pushes users and groups to Genesis
at `https://<genesis-host>/scim/v2`.

For the full SCIM documentation set, start with
[SCIM Inbound Provisioning](/docs/Setup/SCIM-Inbound-Provisioning). This
section only covers the AWS/EKS deployment specifics.

#### Helm values for SCIM

Add the SCIM environment variables to your Helm values (or your
secret-management workflow):

```yaml theme={null}
config:
  extraEnv:
    GENESIS_SCIM_BEARER_TOKEN: "<generated-scim-token>"
    GENESIS_SCIM_GROUP_ROLE_MAPPING: '{"company-admins":"admin","company-users":"user"}'
    GENESIS_SCIM_REQUIRE_PROVISIONED_USERS: "false"
    GENESIS_SCIM_RATE_LIMIT_PER_MINUTE: "600"
    GENESIS_SCIM_TENANT_ID: "default"
```

For the description of each variable, see the
[SCIM environment variables table](/docs/Setup/SCIM-Inbound-Provisioning#environment-variables).

<Warning>
  Treat the SCIM bearer token like a password. Store it in your IdP's secret
  field and your deployment secret-management system. Do not place it in
  screenshots, shared tickets, or shell history.
</Warning>

#### EKS rate limiting note

For multi-replica EKS deployments, enforce strict global SCIM rate limiting at
ingress (for example, AWS WAF, NGINX rate-limit annotations, or an API
gateway). The in-process per-token limiter is per-pod and is not sufficient as
the only control across replicas. See
[SCIM Security](/docs/Setup/SCIM-Security#risks-and-recommended-production-controls)
for details.

#### AWS Identity Center caveat

AWS IAM Identity Center is primarily a SCIM **target**, not a SCIM client. It
does not natively push outbound SCIM into arbitrary applications such as
Genesis.

For customers who want to drive Genesis from AWS Identity Store, the
supported pattern is an AWS-hosted provisioner that reads Identity Store via
AWS APIs and pushes SCIM into Genesis. See
[SCIM IdP Setup Guides — AWS-sourced provisioning](/docs/Setup/SCIM-IdP-Setup-Guides#aws-sourced-provisioning)
for the architecture and step-by-step setup.

For production rollouts that need direct outbound SCIM, use an IdP that
natively pushes SCIM to custom applications, such as Okta, Azure Entra ID,
or OneLogin.

#### Pilot, validation, and rollback

For the validation procedure, conformance harness, and rollback steps, see
the [SCIM Pilot Runbook](/docs/Setup/SCIM-Pilot-Runbook).

## Deployment Steps

<Steps>
  <Step title="Prepare Helm Chart">
    Genesis publishes the Helm chart to private OCI registries. After authenticating with the registry Genesis assigned to your deployment, pull and unpack the chart locally. If Genesis provided a `.tgz` archive instead, extract it and continue with the same chart directory layout:

    ```bash theme={null}
    # Option A: Pull and unpack the OCI chart locally
    helm pull oci://registry-1.docker.io/genesiscomputingai/genesis --version <chart-version> --untar

    # Option B: Extract the Helm chart archive provided by Genesis
    tar -xzf genesis-<version>.tgz

    # Navigate to the chart directory
    cd genesis
    ```

    <Info>
      Chart releases are published from `main`. On each GA release, Genesis updates `appVersion` in `Chart.yaml` to the promoted container version and patch-bumps the chart version automatically. To find the chart version for a specific GA release, query the `GA_HELM_MAP` repo variable: `gh variable get GA_HELM_MAP -R genesis-computing-ai/genesis-agents | jq -r '.["vX.Y.Z"]'`.
    </Info>
  </Step>

  <Step title="Create Values File">
    Create a `genesis-values.yaml` file with your configuration:

    ```yaml theme={null}
    # Genesis EKS Deployment Values

    # Container image (Genesis will provide the repository URL)
    image:
      repository: <container-registry-url>/genesis  # Replace with URL provided by Genesis
      tag: v1.18.2  # Example GA pin; use the version provided by Genesis
      pullPolicy: IfNotPresent

    # Genesis EKS helper scripts enable ECR and inject the registry/token at deploy time
    ecr:
      enabled: true
      registry: ""

    # Image pull secrets (if the container registry requires authentication)
    # Genesis will provide instructions if credentials are needed
    # imagePullSecrets:
    #   - name: genesis-registry-credentials

    # Secrets
    secrets:
      # Master encryption key (REQUIRED on first install unless
      # existingSecretGenesisMasterKey or config.extraEnv.GENESIS_MASTER_KEY_REF)
      genesisMasterKey: ""  # Generate: python -c "import secrets; print(secrets.token_hex(32))"

    # Non-sensitive configuration
    config:
      sqliteDbPath: "/app/.genesis/db/genesis.db"
      logLevel: "INFO"
      logsFormat: "json"
      authEnabled: "true"
      authProvider: "proxy"  # For ingress-based authentication
      
      # Standard AWS SDK region for the pod. The LLM provider is configured
      # after install through the config API, not here.
      extraEnv:
        AWS_REGION: "us-east-1"
        AWS_DEFAULT_REGION: "us-east-1"

    # Recommended for IaC-managed Genesis settings
    genesisConfig:
      content: ""  # Set with --set-file genesisConfig.content=genesis-config.yaml
      existingConfigMap: ""  # Or reference a pre-created ConfigMap, not both

    # Resource requests
    resources:
      requests:
        cpu: "2000m"
        memory: "6Gi"
      limits: {}  # No limits by default

    # Optional Vertical Pod Autoscaler (VPA)
    vpa:
      enabled: false
      updateMode: "Off"  # "Off", "Initial", or "Auto"
      controlledResources:
        - cpu
        - memory
      minAllowed:
        cpu: "500m"
        memory: "2Gi"
      maxAllowed:
        cpu: "8000m"
        memory: "32Gi"

    # Persistent storage
    persistence:
      storageClassName: ""  # Uses cluster default (gp2/gp3)
      size: 50Gi  # Kubernetes uses binary units (Gi = Gibibytes)
      accessMode: ReadWriteOnce

    # Service configuration
    service:
      type: ClusterIP
      ports:
        fastapi: 8080
        flask: 8082
        streamlit: 8501

    # Ingress configuration
    ingress:
      enabled: true
      className: "alb"  # For AWS Load Balancer Controller, or "nginx" for NGINX
      
      # For AWS Load Balancer Controller
      annotations:
        alb.ingress.kubernetes.io/scheme: internet-facing
        alb.ingress.kubernetes.io/target-type: ip
        alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS": 443}]'
        alb.ingress.kubernetes.io/ssl-redirect: "443"
        alb.ingress.kubernetes.io/healthcheck-path: "/api/health"
        alb.ingress.kubernetes.io/healthcheck-protocol: "HTTP"
        alb.ingress.kubernetes.io/healthcheck-port: "8080"
        # For HTTPS, add certificate ARN:
        # alb.ingress.kubernetes.io/certificate-arn: "arn:aws:acm:region:account:certificate/cert-id"
      
      hosts:
        - host: genesis.yourdomain.com  # Replace with your domain
          paths:
            - path: /
              pathType: Prefix
              port: 8080
      
      # TLS configuration (if using HTTPS)
      tls: []

    # Optional: agent sandbox execution in Kubernetes pods
    sandbox:
      enabled: true
      defaultBackend: "eks"

    # Optional extra Kubernetes resources rendered as separate YAML documents
    extraManifests: []
    ```

    <Note>
      For IaC-managed deployments, keep cluster settings in
      `genesis-values.yaml` and copy the chart's `genesis-config-example.yaml`
      to a separate `genesis-config.yaml` for `providers`, `llm`, `embedding`,
      `auth`, `oauth_providers`, and related app settings (including LiteLLM
      Proxy OAuth). Deploy with
      `--set-file genesisConfig.content=genesis-config.yaml`, or set
      `genesisConfig.existingConfigMap` when the file already exists as a
      ConfigMap. Genesis mounts the file at
      `/app/.genesis/config/genesis-config.yaml`, and explicit environment
      variables still take precedence over file-backed values.

      For Helm-driven RBAC on OIDC deployments, you can also pass
      `--set-file groupRoleMapping.content=group-role-mapping.yaml` and
      `--set-file rbacSeedPolicy.content=rbac-policies.csv`. See
      [Deploying Genesis - Azure AKS with OIDC Authentication](/docs/deployments/Azure/Genesis-On-Azure-AKS-OIDC#helm-driven-rbac).
    </Note>

    <Tip>
      `extraManifests` defaults to `[]`. Add complete Kubernetes resources here when you want Helm to deploy objects such as `PodDisruptionBudget`, `NetworkPolicy`, or `PodMonitor` alongside Genesis without forking the chart. Each list item is rendered as its own YAML document.
    </Tip>

    <Tip>
      For NGINX Ingress Controller, use these annotations instead:

      ```yaml theme={null}
      annotations:
        nginx.ingress.kubernetes.io/proxy-read-timeout: "3600"
        nginx.ingress.kubernetes.io/proxy-send-timeout: "3600"
        nginx.ingress.kubernetes.io/proxy-body-size: "100m"
      ```
    </Tip>

    <Info>
      If you deploy with `aws-eks/deploy-eks-auth0.sh`, `aws-eks/deploy-eks-oidc.sh`, or `aws-eks/deploy-eks-proxy-stack.sh`, pin `image.tag` in the corresponding `values-eks-*.yaml` file. Those scripts pass `ecr.enabled`, `ecr.registry`, a fresh `ecr.token`, and `image.pullPolicy=Always` to Helm on every run, and they only pass `--set image.tag` when `GENESIS_IMAGE_TAG` is set in `aws-eks/eks-config.env`.
    </Info>
  </Step>

  <Step title="Deploy Genesis">
    ```bash theme={null}
    # Install using Helm
    helm install genesis ./genesis -f genesis-values.yaml

    # Or if deploying from chart directory
    helm install genesis . -f genesis-values.yaml
    ```
  </Step>

  <Step title="Verify Deployment">
    ```bash theme={null}
    # Check namespace
    kubectl get namespace genesis

    # Check StatefulSet
    kubectl get statefulset -n genesis

    # Check pod status
    kubectl get pods -n genesis -w

    # Check persistent volume claim
    kubectl get pvc -n genesis

    # Check service
    kubectl get svc -n genesis

    # Check ingress
    kubectl get ingress -n genesis

    # View pod logs
    kubectl logs -n genesis genesis-0 --follow
    ```
  </Step>

  <Step title="Access Genesis">
    <Tabs>
      <Tab title="Via Ingress (Recommended)">
        **For AWS Load Balancer Controller:** Get the ALB address:

        ```bash theme={null}
        kubectl get ingress -n genesis genesis -o jsonpath='{.status.loadBalancer.ingress[0].hostname}'
        ```

        Access via: `http://<alb-hostname>` or `https://genesis.yourdomain.com`

        **For NGINX Ingress:** Get the external IP:

        ```bash theme={null}
        kubectl get svc -n ingress-nginx ingress-nginx-controller
        ```

        Access via: `http://<external-ip>` or `http://genesis.yourdomain.com` (if DNS configured)
      </Tab>

      <Tab title="Port Forward (For Testing)">
        ```bash theme={null}
        # Forward port 8080
        kubectl port-forward -n genesis svc/genesis 8080:8080

        # Access at http://localhost:8080
        ```
      </Tab>
    </Tabs>
  </Step>

  <Step title="Configure the LLM Provider">
    Genesis starts without a provider and reports **No suitable LLM found** until
    one is configured. Configure it against the **running** pod, either from
    **Config -> AI Providers** in the UI or through the config API. Keep
    credentials in a request-body file — process arguments are visible to every
    process on the host and land in shell history.

    ```bash theme={null}
    umask 077
    set -o pipefail   # json.tool must not mask a failed curl
    GENESIS_URL=https://genesis.yourdomain.com
    BODY=$(mktemp "${TMPDIR:-/tmp}/genesis-llm-body.XXXXXX.json")
    CURL_CONFIG=$(mktemp "${TMPDIR:-/tmp}/genesis-curl.XXXXXX")
    trap 'rm -f "$BODY" "$CURL_CONFIG"' EXIT
    chmod 600 "$BODY" "$CURL_CONFIG"

    # With authEnabled: "true", every call needs a bearer token for a user holding
    # access_config_panel. read -rs keeps it out of shell history. Drop these lines
    # and --config only on an auth-disabled pod.
    read -rsp 'Genesis access token: ' GENESIS_TOKEN && echo
    printf 'header = "Authorization: Bearer %s"\n' "$GENESIS_TOKEN" > "$CURL_CONFIG"
    unset GENESIS_TOKEN

    # IRSA deployments use "auth_mode": "iam_role" and omit aws_bearer_token.
    cat > "$BODY" <<'JSON'
    {"provider": "bedrock",
     "auth_mode": "api_key",
     "endpoint_mode": "default",
     "aws_bearer_token": "<aws-bearer-token-bedrock>",
     "aws_region": "us-east-1",
     "aws_account_id": "<aws-account-id>",
     "test_connection": true}
    JSON
    curl -sS --fail-with-body --config "$CURL_CONFIG" \
      -X POST -H 'Content-Type: application/json' --data @"$BODY" \
      "$GENESIS_URL/api/config/llm-providers/other" | python3 -m json.tool

    curl -sS --fail-with-body --config "$CURL_CONFIG" \
      -X POST -H 'Content-Type: application/json' -d '{"force": true}' \
      "$GENESIS_URL/api/config/llm-providers/litellm_bedrock/activate" | python3 -m json.tool

    curl -sS --fail-with-body --config "$CURL_CONFIG" \
      -X PUT -H 'Content-Type: application/json' \
      -d '{"provider": "litellm_bedrock", "use_defaults": true, "activate_provider": true}' \
      "$GENESIS_URL/api/config/models/main/assignments" | python3 -m json.tool
    ```

    Read each response before running the next call: these routes also answer `200`
    with `success: false` (for example when the provider already exists — repeat
    that call as a `PUT`). For the full endpoint reference, the card / llm-type
    naming table and the other providers, see
    [Configuring a provider through the config API](/docs/Setup/Setup-AI-Model#configuring-a-provider-through-the-config-api).

    <Note>
      Some schema fields carry a `default_env_var` binding that seeds them from the
      environment while nothing is stored, so an image can come up pre-seeded. That
      is bootstrap only — it is not the way to configure a provider, and anything
      saved through the UI or the config API outranks it.
    </Note>
  </Step>
</Steps>

## Kubernetes-Specific Configuration

### StatefulSet

Genesis is deployed as a **StatefulSet** (not Deployment) to ensure:

* Stable network identity (pod name: `genesis-0`)
* Ordered, graceful deployment and scaling
* Stable persistent storage (PVC name: `genesis-data-genesis-0`)

### Persistent Volume Claim

The StatefulSet creates a PersistentVolumeClaim with:

| Property | Value |
| - | - |
| Name | `genesis-data-genesis-0` |
| Storage Class | Uses cluster default (typically `gp2` or `gp3`) |
| Access Mode | `ReadWriteOnce` (single pod access) |
| Retention | PVC is retained when StatefulSet is deleted (data preservation) |

### Helm resource retention

Chart `0.3.70` and later annotate the Genesis Namespace, sandbox Namespace and
RBAC, and both ServiceAccounts with `helm.sh/resource-policy: keep`. This keeps
those objects in place when they move to a separate platform release in a later
chart revision; it also means `helm uninstall` does not delete the Genesis
Namespace. Delete the namespace explicitly when you intend to remove the
deployment:

```bash theme={null}
kubectl delete namespace <genesis-namespace>
```

If sandbox access is enabled, the per-instance RoleBinding and
ClusterRoleBinding are retained outside the Genesis Namespace as well. Turning
`sandbox.enabled` off or changing `sandbox.namespace` does not revoke those
bindings, and the old bindings can continue granting pod management and
namespace-read access to the instance's ServiceAccount. Revoke all bindings for
an instance with:

```bash theme={null}
cluster/scripts/cleanup-retained-rbac.sh <genesis-namespace>
```

After changing `sandbox.namespace`, run the cleanup command and then re-apply
the Helm release so the current bindings are recreated. The command is also
needed after a direct `helm uninstall`; `cluster/scripts/teardown.sh` runs it as
part of teardown. Shared sandbox resources are retained for other instances
and should only be removed with the corresponding shared-resource retirement
workflow after the last instance is gone.

The chart reserves `helm.sh/resource-policy` on retained objects. Do not set
that key under `serviceAccount.annotations` or
`sandbox.podServiceAccount.annotations`; Helm rejects the render and names the
values path. A rollback to a chart version older than `0.3.70` removes the
annotation from live objects, so re-check it after rollback:

```bash theme={null}
kubectl get namespace <genesis-namespace> \
  -o jsonpath='{.metadata.annotations.helm\.sh/resource-policy}'
```

### Volume Mounts

The Genesis container mounts the persistent volume at multiple paths:

| Mount Path | Purpose |
| - | - |
| `/app/.genesis/db` | Database files |
| `/app/bot_git` | Git repositories |
| `/app/bot_storage` | File storage and uploads |
| `/app/tmp` | Temporary files |

### Health Checks

Genesis uses three Kubernetes probes so slow startups do not trigger liveness restarts before the app is ready:

<CardGroup cols={3}>
  <Card title="Startup Probe" icon="hourglass">
    * **Endpoint:** `GET /api/health` on port 8080
    * **Period:** 10 seconds
    * **Timeout:** 5 seconds
    * **Failure threshold:** 60 (up to \~10 minutes boot grace)
    * Gates liveness and readiness until startup completes (`503` with progress, then `200`)
  </Card>

  <Card title="Liveness Probe" icon="heart-pulse">
    * **Endpoint:** `GET /api/health/live` on port 8080
    * **Period:** 30 seconds
    * **Timeout:** 5 seconds
    * **Failure threshold:** 3
    * Always `200` while the process is alive — does not require Genesis to be fully ready
  </Card>

  <Card title="Readiness Probe" icon="check">
    * **Endpoint:** `GET /api/health` on port 8080
    * **Period:** 10 seconds
    * **Timeout:** 5 seconds
    * **Failure threshold:** 3
    * Returns `503` while booting; `200` when ready to serve traffic
  </Card>
</CardGroup>

When Genesis sits behind **OAuth2 Proxy** (`authProvider: "proxy"`, for example via `aws-eks/deploy-eks-proxy-stack.sh`), configure the proxy so GET health probes and the dashboard `healthcheck` endpoint bypass authentication. The bundled `aws-eks/oauth2-proxy-values-eks-example.yaml` includes:

```yaml theme={null}
cookie_secure = true
cookie_samesite = "lax"
skip_auth_routes = ["GET=^/api/health$", "GET=^/api/health/startup$", "GET=^/api/health/live$", "GET=^/api/healthcheck$"]
cookie_csrf_per_request = true
cookie_csrf_expire = "5m"
```

`cookie_secure = true` is correct when HTTPS terminates at the ingress controller or external load balancer, even if the OAuth2 Proxy chart ingress has `tls: []`. Routes are GET-scoped and anchored so other `/api/health/*` paths (errors, threads, slots, and so on) remain authenticated. `cookie_csrf_per_request` isolates concurrent login flows and prevents invalid-CSRF `403` responses.

#### Auth-exempt health Ingress for `auth_request` deployments

If an NGINX ingress uses `nginx.ingress.kubernetes.io/auth-url` and
`nginx.ingress.kubernetes.io/auth-signin`, an external load balancer or uptime monitor
can receive the proxy's login redirect instead of Genesis health status. Enable the
chart's second, unauthenticated health Ingress for this deployment pattern:

```yaml theme={null}
ingress:
  annotations:
    nginx.ingress.kubernetes.io/auth-url: "https://oauth2-proxy.yourcompany.com/oauth2/auth"
    nginx.ingress.kubernetes.io/auth-signin: "https://oauth2-proxy.yourcompany.com/oauth2/start?rd=$escaped_request_uri"
  authExemptHealth:
    enabled: true
```

`ingress.authExemptHealth.enabled` defaults to `false`. When enabled, the chart renders
`<release>-health` on the same hosts and TLS configuration as the main Ingress, routing
these exact paths to the FastAPI Service: `/api/health`, `/api/health/startup`,
`/api/health/live`, and `/api/healthcheck`, each with and without a trailing slash.
`ingress.authExemptHealth.pathType` defaults to `Exact`; keep it that way so the
authenticated `/api/health/detailed` and `/api/health/slots` routes are not exposed.
The backend port defaults to `service.ports.fastapi` (`8080`) and can be overridden with
`ingress.authExemptHealth.port`.

The health Ingress does not inherit `ingress.annotations`. Put only its own annotations
under `ingress.authExemptHealth.annotations`. For the AWS Load Balancer Controller,
add the ALB group annotation there so both Ingress resources use the same load balancer:

```yaml theme={null}
ingress:
  authExemptHealth:
    annotations:
      alb.ingress.kubernetes.io/group.name: genesis
```

### Graceful shutdown (`podLifecycle`)

The chart sets a `preStop` sleep and a pod-level termination grace period so ingress load balancers can drain in-flight requests before Genesis receives `SIGTERM`.

| Value | Type | Default | Description |
| - | - | - | - |
| `podLifecycle.preStopSleepSeconds` | `int` | `30` | Seconds the main container sleeps in `preStop` before `SIGTERM`, giving the load balancer time to stop sending new traffic. |
| `podLifecycle.terminationGracePeriodSeconds` | `int` | `90` | Total grace period before `SIGKILL`. Must be **greater than** `preStopSleepSeconds` so Genesis can shut down after the sleep. Helm fails template rendering if this constraint is violated. |

```yaml theme={null}
podLifecycle:
  preStopSleepSeconds: 30
  terminationGracePeriodSeconds: 90
```

Increase both values if your ingress controller or ALB target deregistration is slower than the defaults.

### Extension containers (`extraContainers`, `extraInitContainers`)

Pass additional Kubernetes container specs through the chart without forking templates. `extraInitContainers` is rendered when `customCACerts` is enabled, when `extraInitContainers` is non-empty, or both.

| Value | Type | Default | Description |
| - | - | - | - |
| `extraContainers` | `list` | `[]` | Sidecar containers in the Genesis pod (for example, a log shipper mounting `genesis-data`). |
| `extraInitContainers` | `list` | `[]` | Init containers that run before the `genesis` container starts (for example, wait for an external dependency). |

```yaml theme={null}
extraContainers:
  - name: fluent-bit
    image: fluent/fluent-bit:latest
    volumeMounts:
      - name: genesis-data
        mountPath: /app/.genesis/db
        subPath: db
        readOnly: true

extraInitContainers:
  - name: wait-for-db
    image: busybox:1.36
    command: ["sh", "-c", "until nc -z db-host 5432; do sleep 2; done"]
```

### Resource Requirements

**Default resource requests:**

| Resource | Request |
| - | - |
| CPU | 2000m (2 cores) |
| Memory | 6Gi |

<Note>
  No resource limits are set by default to avoid OOM kills. Adjust based on your workload.
</Note>

### Vertical Pod Autoscaler (Optional)

Genesis runs as a single-replica StatefulSet backed by a `ReadWriteOnce`
persistent volume, so horizontal pod autoscaling is not supported. If you want
automatic right-sizing for `resources.requests`, enable the chart's optional
Vertical Pod Autoscaler (VPA) support instead.

**Prerequisites:**

1. `metrics-server` must be installed in the cluster.
2. Install a VPA controller. From the Helm chart directory, you can use the bundled helper:

```bash theme={null}
./aws-eks/setup-eks-vpa.sh
```

<Warning>
  `vpa.updateMode: "Auto"` can cause brief downtime because the single Genesis
  pod must be evicted and recreated to apply new resource requests. Start with
  `vpa.updateMode: "Off"` unless brief resizes are acceptable in your
  environment.
</Warning>

| Value | Type | Default | Description |
| - | - | - | - |
| `vpa.enabled` | `boolean` | `false` | Creates a `VerticalPodAutoscaler` resource for the Genesis StatefulSet. |
| `vpa.updateMode` | `string` | `"Off"` | Controls how recommendations are applied. Supported values are `"Off"`, `"Initial"`, and `"Auto"`. |
| `vpa.controlledResources` | `string[]` | `["cpu", "memory"]` | Resources VPA manages for the `genesis` container. |
| `vpa.minAllowed.cpu` | `string` | `"500m"` | Minimum CPU request VPA can set. |
| `vpa.minAllowed.memory` | `string` | `"2Gi"` | Minimum memory request VPA can set. |
| `vpa.maxAllowed.cpu` | `string` | `"8000m"` | Maximum CPU request VPA can set. |
| `vpa.maxAllowed.memory` | `string` | `"32Gi"` | Maximum memory request VPA can set. |

**Example: recommendation-only mode**

```yaml theme={null}
vpa:
  enabled: true
  updateMode: "Off"
  controlledResources:
    - cpu
    - memory
  minAllowed:
    cpu: "500m"
    memory: "2Gi"
  maxAllowed:
    cpu: "8000m"
    memory: "32Gi"
```

Apply the updated values and then verify the VPA object:

```bash theme={null}
helm upgrade genesis ./genesis -f genesis-values.yaml
kubectl get vpa -n genesis
kubectl describe vpa genesis -n genesis
```

<Tip>
  The examples on this page use Helm release name `genesis` in namespace
  `genesis`. If you deploy with a different release name or namespace, check the
  generated VPA name with `kubectl get vpa -n <namespace>` before describing it.
</Tip>

## AWS-Specific Considerations

### IAM Roles for Service Accounts (IRSA)

For production, configure IRSA so the Genesis pod can access AWS services
without storing long-lived credentials. This is the recommended pattern for AWS
Bedrock IAM authentication.

1. Create IAM Role with Bedrock permissions
2. Create Service Account with role annotation:

```yaml theme={null}
apiVersion: v1
kind: ServiceAccount
metadata:
  name: genesis
  namespace: genesis
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/genesis-bedrock-role
```

3. Configure the Helm release to use the annotated service account:

```yaml theme={null}
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/genesis-bedrock-role
```

Genesis detects the AWS web-identity environment variables injected by EKS and
can use IAM Role authentication for Bedrock without an API key.

### Agent sandbox execution (optional)

Genesis agents can run long-running Python jobs, shell commands, and scripts
in isolated Kubernetes pods instead of the application container's local Docker
runtime. On EKS, enable this through Helm `sandbox` values. The Genesis
ServiceAccount needs IRSA for the Genesis pod itself (Bedrock, Secrets Manager,
and other control-plane AWS calls). Sandbox workload pods run in a separate
namespace and need their own ServiceAccount with IRSA when agents call AWS APIs
from inside the sandbox (for example `boto3`, S3, or Bedrock).

| Value | Type | Default | Description |
| - | - | - | - |
| `sandbox.enabled` | `boolean` | `false` | Enable sandbox RBAC and default-backend wiring. |
| `sandbox.namespace` | `string` | `"genesis-sandbox"` | Kubernetes namespace for sandbox pods. |
| `sandbox.defaultBackend` | `string` | `"local"` | Sets `GENESIS__SANDBOX__DEFAULT_BACKEND` when `sandbox.enabled` is `true`. Use `"eks"` on EKS so `sandbox_action` defaults to Kubernetes pods. Valid values: `local`, `eks`, `k8s`, `ecs_ec2`, `fargate`, `railway`. |
| `sandbox.createSharedResources` | `boolean` | `true` | When `true`, the chart creates the sandbox namespace, a `genesis-sandbox-manager` Role, a `genesis-sandbox-ns-reader` ClusterRole, and per-release bindings for the Genesis ServiceAccount. Set to `false` when an external provisioning script manages shared resources (for example, multi-instance clusters). |
| `sandbox.podServiceAccount.create` | `boolean` | `false` | When `true` and `sandbox.createSharedResources` is `true`, create a dedicated ServiceAccount in the sandbox namespace for sandbox pods. |
| `sandbox.podServiceAccount.name` | `string` | `"genesis"` | Name of the sandbox-pod ServiceAccount in `sandbox.namespace`. |
| `sandbox.podServiceAccount.annotations` | `object` | `{}` | Annotations on the sandbox-pod ServiceAccount. On EKS, set `eks.amazonaws.com/role-arn` to the IAM role sandbox pods should assume via IRSA. |

When `sandbox.enabled` is `true`, the chart grants the release ServiceAccount
pod lifecycle permissions in the sandbox namespace (`create`, `exec`, `delete`,
`get`, `list`, `watch` on pods and pod logs) and read access to the sandbox
namespace object itself.

**Example: EKS lab instance**

```yaml theme={null}
sandbox:
  enabled: true
  defaultBackend: "eks"
  podServiceAccount:
    create: true
    name: "genesis"
    annotations:
      eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/genesis-sandbox-pod-role

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/genesis-runtime-role
```

`serviceAccount.create` must be `true` so the chart can bind sandbox RBAC to the
Genesis ServiceAccount. Set `sandbox.podServiceAccount.create` to `true` and
annotate `sandbox.podServiceAccount.annotations` with an IRSA role trusted for
`<sandbox.namespace>:<sandbox.podServiceAccount.name>` so sandbox pods receive
AWS credentials. When starting sandboxes, pass the same name via
`backend_options.service_account` (or `spec.k8s.service_account`) — for example
`genesis` when using the default `sandbox.podServiceAccount.name`.

The configured `defaultBackend` maps to `sandbox.default_backend` in Genesis
APP\_CONFIG. Agents that call `sandbox_action` without an explicit `backend` (or
with `backend="local"`) use that operator default instead of local Docker. See
[sandbox\_action](/docs/Genbot-Toolkit/Additional-Tool-Groups#sandbox-tools).

### Bootstrap secrets via `config.extraEnv` (AWS Secrets Manager)

On EKS with IRSA, you can reference AWS Secrets Manager secrets in Helm
`config.extraEnv`. Genesis resolves `aws_*` shorthand or full ARNs at **pod
startup** using the pod's IRSA credentials — no init container, Kubernetes
`Secret`, or CSI driver is required for the reference itself.

Grant `secretsmanager:GetSecretValue` on the secret to the IRSA role (same as
[Custom CA Certificates](#custom-ca-certificates-from-aws-secrets-manager) and
[External Secrets](/docs/Setup/External-Secrets)). After you rotate a value in
AWS Secrets Manager, restart the StatefulSet so the pod re-resolves it:

```bash theme={null}
kubectl rollout restart statefulset genesis -n genesis
```

**Example: Databricks API key on EKS** (chart file `values_databricks_llm.yaml`):

```yaml theme={null}
config:
  extraEnv:
    DATABRICKS_API_KEY: "arn:aws:secretsmanager:REGION:ACCOUNT:secret:NAME"
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT:role/ROLE"
```

Short form is also supported:

```yaml theme={null}
config:
  extraEnv:
    DATABRICKS_API_KEY: "aws_my-org/genesis/databricks_token"
```

For the same Databricks LLM deployment, you can instead pass the token at
install time (`--set secrets.databricksApiKey="dapi..."`) or mount a
pre-created Kubernetes secret via `secrets.existingSecretDatabricksApiKey`.

The bundled `values_databricks_llm.yaml` sample leaves `config.modelName`,
`config.harvesterModel`, `config.fastModelName` and `config.embeddingModel`
empty, so the catalog default for each role applies: `databricks-claude-opus-5-5`
for the main and harvester models, `databricks-claude-haiku-4-5` for the fast
model, and `databricks-gte-large-en` for embeddings. Set any of these fields to
seed a different model when no corresponding model is already saved in AppConfig.

All three forms are `default_env_var` **bootstrap seeding**: the value is captured
only while nothing is stored for the field, and it is outranked by anything saved
through the UI or the config API. Seeding a credential does not configure or
activate the provider — do that from
[Configuring a provider through the config API](/docs/Setup/Setup-AI-Model#configuring-a-provider-through-the-config-api).

**Example: Master encryption key from AWS Secrets Manager** (leave
`genesisMasterKey` empty and resolve via IRSA at pod startup):

```yaml theme={null}
secrets:
  genesisMasterKey: ""

config:
  extraEnv:
    GENESIS_MASTER_KEY_REF: "aws_prod/genesis-keys#master_key"

serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: "arn:aws:iam::ACCOUNT:role/genesis-secrets-reader"
```

Grant `secretsmanager:GetSecretValue` on the referenced secret to the IRSA
role. If `GENESIS_MASTER_KEY` is also set in the pod environment, it takes
precedence over `GENESIS_MASTER_KEY_REF`. See
[Master encryption key](/docs/Setup/External-Secrets#master-encryption-key)
for the full resolution order.

<Note>
  Some non-production deployments grant Bedrock permissions to the EKS node
  role. That is acceptable for isolated testing, but IRSA is preferred for
  production because it scopes permissions to the Genesis service account rather
  than all pods on a node.
</Note>

### Custom CA Certificates from AWS Secrets Manager

If Genesis must connect to internal HTTPS endpoints signed by a private CA, the
Helm chart can fetch CA certificates from AWS Secrets Manager during pod
startup and merge them into the container trust store. It uses the same secret
reference syntax documented in [External Secrets](/docs/Setup/External-Secrets).

```yaml theme={null}
serviceAccount:
  create: true
  annotations:
    eks.amazonaws.com/role-arn: arn:aws:iam::<account-id>:role/genesis-runtime-role

customCACerts:
  enabled: true
  secrets:
    - ref: aws_prod/certs/corporate-root-ca
      fileName: corporate-root-ca.crt
    - ref: arn:aws:secretsmanager:us-east-1:123456789012:secret:prod/certs/internal-ca
      fileName: internal-ca.crt
```

| Parameter | Type | Default | Description |
| - | - | - | - |
| `customCACerts.enabled` | `bool` | `false` | Enables custom CA fetching at pod startup. When set to `true`, `customCACerts.secrets` must not be empty or Helm will fail validation. |
| `customCACerts.secrets` | `list` | `[]` | CA certificates to fetch and merge into the system trust store. |
| `customCACerts.secrets[].ref` | `string` | | On EKS, use `aws_*` or a full AWS Secrets Manager ARN. The chart also accepts `gcp_*` references when the pod has GCP Secret Manager credentials. |
| `customCACerts.secrets[].fileName` | `string` | | Required filename for the fetched certificate. It must be a simple `*.crt` name containing only letters, numbers, `.`, `_`, or `-`. Path separators are rejected. |

<Note>
  Grant `secretsmanager:GetSecretValue` on each referenced secret to the IRSA
  role. When enabled, the chart mounts the merged CA bundle and sets
  `REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, `SSL_CERT_FILE`, `GIT_SSL_CAINFO`, and
  `NODE_EXTRA_CA_CERTS` so Python HTTP clients, curl, Git clone/fetch, and
  Node.js trust the same custom CAs as API calls.
</Note>

The `fetch-ca-certs` init container uses the image's
`/etc/ssl/certs/ca-certificates.crt` system bundle to contact Secrets Manager.
In that container only, the chart overrides `SSL_CERT_FILE`,
`REQUESTS_CA_BUNDLE`, `CURL_CA_BUNDLE`, and `AWS_CA_BUNDLE` from
`config.extraEnv`; this prevents paths to a custom bundle that is not available
until Genesis starts from breaking certificate retrieval. The Genesis
container continues to use the merged custom CA bundle.

<Tip>
  If you already distribute CA files through Kubernetes Secrets or a CSI
  driver, continue using `extraVolumes` and `extraVolumeMounts` to mount them
  directly instead of `customCACerts`.
</Tip>

When you rotate a certificate in Secrets Manager, restart the StatefulSet so
the pod rebuilds the trust bundle:

```bash theme={null}
kubectl rollout restart statefulset genesis -n genesis
```

### EBS Volume Encryption

EBS volumes created by the EBS CSI driver on EKS are encrypted by default if your cluster has encryption enabled. Verify:

```bash theme={null}
kubectl get storageclass -o yaml
```

### VPC Configuration

Ensure your EKS cluster nodes have:

| Requirement | Purpose |
| - | - |
| Outbound Internet Access | For pulling images from the container registry provided by Genesis |
| DNS Resolution | For external API calls (OpenAI, Databricks, etc.) |
| Security Group Rules | Allow ingress on ports 80/443 if using ALB, or node port if using NodePort service |

## Troubleshooting

<AccordionGroup>
  <Accordion title="Pod Not Starting">
    ```bash theme={null}
    # Check pod status
    kubectl describe pod -n genesis genesis-0

    # Check events
    kubectl get events -n genesis --sort-by='.lastTimestamp'
    ```

    **Common issues:**

    * **Image pull errors:** Check network connectivity to the container registry provided by Genesis
    * **PVC issues:** Verify EBS CSI driver is installed and storage class exists
    * **Resource constraints:** Check node resources
  </Accordion>

  <Accordion title="Persistent Volume Issues">
    ```bash theme={null}
    # Check PVC status
    kubectl get pvc -n genesis
    kubectl describe pvc genesis-data-genesis-0 -n genesis

    # Check PV
    kubectl get pv
    kubectl describe pv <pv-name>

    # Check EBS CSI driver logs
    kubectl logs -n kube-system -l app=ebs-csi-controller
    ```
  </Accordion>

  <Accordion title="Ingress Not Working">
    **For AWS Load Balancer Controller:**

    ```bash theme={null}
    # Check controller logs
    kubectl logs -n kube-system -l app.kubernetes.io/name=aws-load-balancer-controller

    # Check ingress status
    kubectl describe ingress -n genesis genesis

    # Verify ALB was created
    aws elbv2 describe-load-balancers --region <region>
    ```

    **For NGINX Ingress:**

    ```bash theme={null}
    # Check controller pods
    kubectl get pods -n ingress-nginx

    # Check controller logs
    kubectl logs -n ingress-nginx -l app.kubernetes.io/component=controller
    ```
  </Accordion>

  <Accordion title="Database Issues">
    ```bash theme={null}
    # Access pod shell
    kubectl exec -it -n genesis genesis-0 -- /bin/bash

    # Check database file
    ls -lh /app/.genesis/db/

    # Check database integrity (if SQLite tools available)
    sqlite3 /app/.genesis/db/genesis.db "PRAGMA integrity_check;"
    ```
  </Accordion>

  <Accordion title="Viewing Logs">
    ```bash theme={null}
    # View current logs
    kubectl logs -n genesis genesis-0

    # Follow logs
    kubectl logs -n genesis genesis-0 --follow

    # View previous container logs (if pod restarted)
    kubectl logs -n genesis genesis-0 --previous
    ```
  </Accordion>
</AccordionGroup>

## Upgrading Genesis

When a new Genesis version is released:

```bash theme={null}
# Update image tag in values file or override
helm upgrade genesis ./genesis -f genesis-values.yaml --set image.tag=v1.3.2

# Or update values.yaml and upgrade
helm upgrade genesis ./genesis -f genesis-values.yaml

# Verify upgrade
kubectl get pods -n genesis -o jsonpath='{.items[0].spec.containers[0].image}'
kubectl get pods -n genesis
```

<Info>
  The StatefulSet will perform a rolling update, and the persistent volume will be reattached to the new pod.
</Info>

<Note>
  For values-driven upgrades, update `image.tag` in the relevant `values-eks-*.yaml` file and rerun the matching `aws-eks/deploy-eks-*.sh` helper. If you use `aws-eks/update-eks-image.sh` or `aws-eks/update-eks-helm.sh`, both helpers require `ECR_REGISTRY` in `aws-eks/eks-config.env`, default to `v1.18.2` when no tag argument is provided, refresh the ECR token on every run, and perform a `kubectl rollout restart` after `helm upgrade` so EKS nodes pull the updated image and pull secret.
</Note>

<Tip>
  Purely numeric `image.tag` values (for example ECR build IDs) are coerced to strings automatically in chart **0.3.55** and later. On older charts, quote the tag in YAML or use `--set-string image.tag=...` instead of `--set`.
</Tip>

## Uninstalling

```bash theme={null}
# Uninstall Helm release
helm uninstall genesis

# Note: the PVC, namespace, ServiceAccounts, and retained sandbox RBAC are kept
# by the chart; delete the namespace explicitly when removing the deployment.
# If sandbox access was enabled, revoke its retained per-instance bindings first:
cluster/scripts/cleanup-retained-rbac.sh genesis

# To delete PVC and all data:
kubectl delete pvc genesis-data-genesis-0 -n genesis

# Delete namespace
kubectl delete namespace genesis
```

<Warning>
  Deleting the PVC will permanently remove all Genesis data including the database, git repositories, and uploaded files.
</Warning>

## Additional Resources

<CardGroup cols={3}>
  <Card title="AWS EKS Documentation" icon="aws" href="https://docs.aws.amazon.com/eks/">
    Official Amazon EKS documentation
  </Card>

  <Card title="EBS CSI Driver" icon="hard-drive" href="https://docs.aws.amazon.com/eks/latest/userguide/ebs-csi.html">
    EBS CSI Driver setup guide
  </Card>

  <Card title="AWS Load Balancer Controller" icon="scale-balanced" href="https://kubernetes-sigs.github.io/aws-load-balancer-controller/">
    AWS Load Balancer Controller documentation
  </Card>
</CardGroup>