For everything after the install — upgrade, rollback, backup and restore, secrets, user offboarding, scaling, teardown, and the failure-mode catalogue — see the Genesis Operations Runbook. It is cloud-agnostic, with a per-cloud section for the EKS-specific substrate. EBS snapshot procedures stay in Backup and Recovery.
Genesis will provide you with the container image URL. Replace
<container-registry-url> throughout this document with the URL provided by Genesis. Helm charts are published as private OCI artifacts on Docker Hub (oci://registry-1.docker.io/genesiscomputingai/genesis), ECR (oci://<account>.dkr.ecr.<region>.amazonaws.com/charts/genesis), and GHCR (oci://ghcr.io/genesis-computing-ai/charts/genesis). Docker Hub and ECR credentials are provided during onboarding; GHCR access uses a GitHub PAT with read:packages. If you use the bundled aws-eks/ helper scripts, copy aws-eks/eks-config-example.env to aws-eks/eks-config.env, set ECR_REGISTRY, and ensure AWS_DEFAULT_REGION matches the ECR region so aws ecr get-login-password succeeds. Pin image.tag in your values file, and set GENESIS_IMAGE_TAG in aws-eks/eks-config.env only when you want a one-off override for a deploy.Prerequisites
AWS EKS Cluster Requirements
EKS Cluster
Kubernetes version 1.23 or higher
kubectl
Configured to access your EKS cluster
Helm 3.8+
For deploying the Genesis Helm chart
AWS CLI
Configured with appropriate credentials
Additional Requirements
- EBS CSI Driver: Required for persistent storage (see EBS CSI Driver Setup)
- Ingress Controller: For external access (see Ingress Controller Setup)
metrics-serverand VPA controller (optional): Required only if you enable the chart’svpasettings (see Vertical Pod Autoscaler (Optional))
Network Access
IAM Permissions
The EKS cluster node group IAM role needs the following permissions:AmazonEKSWorkerNodePolicyAmazonEKS_CNI_PolicyAmazonEC2ContainerRegistryReadOnly
If using AWS Bedrock as the LLM provider, use IAM Roles for Service
Accounts (IRSA) for production least-privilege access. Granting Bedrock
permissions to the node group role can work for simple non-production
deployments, but it gives every workload on that node role the same access.
EBS CSI Driver Setup
Genesis requires persistent storage for its database, git repositories, and uploaded files. The Amazon EBS CSI driver must be installed in your cluster to provision EBS volumes. Check if installed:Installation Instructions
Refer to the Amazon EKS User Guide for detailed EBS CSI driver installation steps, including IAM role setup and add-on installation.
Ingress Controller Setup
Genesis requires an ingress controller for external access. Choose one based on your requirements:- AWS Load Balancer Controller (Recommended)
- NGINX Ingress Controller
The AWS Load Balancer Controller creates Application Load Balancers (ALB) or Network Load Balancers (NLB) for ingress resources.Install AWS Load Balancer Controller:Verify installation:
Single-instance and load balancer choice: Genesis runs as a single pod (
genesis-0) and does not support multiple instances. ALB/NLB therefore do not add value for load distribution across replicas. They still add value for: exposing the app with a stable external endpoint, TLS termination (e.g. ACM with ALB), health checks, and integration with Route53, WAF, etc. You can also expose Genesis using a LoadBalancer-type Service (one NLB/ELB per service) if you prefer not to use an ingress controller.Genesis Container Configuration
Container Image
Current Genesis container images include
ripgrep (rg) out of the box for in-container file-content searches. After kubectl exec, you can use it to inspect mounted files under /app/bot_storage or /app/bot_git without installing extra packages. The same images also bundle the in-repo genesis_sdk package under /app/genesis_sdk (for execute_python_code local execution — not installable from PyPI) and the llm_wiki package under /app/llm_wiki, so the SDK, wiki APIs, and tools shipped with Genesis are available without extra image customization.Image Pull Secrets (If Required)
If the container registry requires authentication, Genesis will provide the necessary credentials. Create a Kubernetes secret:Exposed Ports
Persistent Storage Requirements
Genesis requires persistent storage for:Recommended Storage Size: 50Gi minimum (adjust based on expected usage)Storage Class: Uses cluster default (typically
gp2 or gp3 on EKS)Environment Variables
Required Configuration
Required Configuration
AWS Bedrock Configuration
AWS Bedrock Configuration
The active LLM provider belongs to These are the standard AWS SDK region variables that boto3 reads; they do not
select or configure a provider. Bedrock’s own region is
APP_CONFIG. Configure Bedrock after the
pod is running with the config API or Config -> AI Providers — see
Configuring a provider through the config API
and the worked example under Deployment Steps below.The CloudFormation BedrockModelId parameter and Terraform
bedrock_model_id input default to "". Leave them empty to use the
Bedrock catalog defaults, including Claude Opus 5.5
(global.anthropic.claude-opus-5-5) for chat, or set a model id to pin it.
The Terraform AWS and ECS templates seed an unset provider configuration with
ACTIVE_LLM_PROVIDER=litellm_bedrock and BEDROCK_AUTH_MODE=iam_role.
They also set BEDROCK_ENABLED=true so images through v1.19.95 recognize
the provider when no configured marker is stored; newer images ignore this
switch. A non-empty model input also seeds BEDROCK_MODEL_NAME,
BEDROCK_FAST_MODEL_NAME, and BEDROCK_PREMIER_MODEL. When the input is
empty, those templates omit the model overrides. Saved configuration or a
mounted genesis-config.yaml takes precedence over these deployment
defaults.aws_region in the provider configuration.Recommended: use IRSA for the Genesis service account, so no API key is stored
at all — configure Bedrock with "auth_mode": "iam_role" and the pod’s role
supplies the credential.Authentication (Optional)
Authentication (Optional)
Git repository runtime
Git repository runtime
Genesis prepares
GIT_PATH on startup and ensures repos/, worktrees/,
and .genesis_runtime/ exist before shared Git-backed workflows use the
mount. The shared group-write repair starts in the background so slow
recursive permission fixes do not block container startup.Logging
Logging
Inbound SCIM for AWS/EKS deployments
SCIM is optional. Enable it only when you want an external IdP to provision Genesis users and role membership automatically. Genesis acts as a SCIM 2.0 service provider, so the customer’s IdP pushes users and groups to Genesis athttps://<genesis-host>/scim/v2.
For the full SCIM documentation set, start with
SCIM Inbound Provisioning. This
section only covers the AWS/EKS deployment specifics.
Helm values for SCIM
Add the SCIM environment variables to your Helm values (or your secret-management workflow):EKS rate limiting note
For multi-replica EKS deployments, enforce strict global SCIM rate limiting at ingress (for example, AWS WAF, NGINX rate-limit annotations, or an API gateway). The in-process per-token limiter is per-pod and is not sufficient as the only control across replicas. See SCIM Security for details.AWS Identity Center caveat
AWS IAM Identity Center is primarily a SCIM target, not a SCIM client. It does not natively push outbound SCIM into arbitrary applications such as Genesis. For customers who want to drive Genesis from AWS Identity Store, the supported pattern is an AWS-hosted provisioner that reads Identity Store via AWS APIs and pushes SCIM into Genesis. See SCIM IdP Setup Guides — AWS-sourced provisioning for the architecture and step-by-step setup. For production rollouts that need direct outbound SCIM, use an IdP that natively pushes SCIM to custom applications, such as Okta, Azure Entra ID, or OneLogin.Pilot, validation, and rollback
For the validation procedure, conformance harness, and rollback steps, see the SCIM Pilot Runbook.Deployment Steps
1
Prepare Helm Chart
Genesis publishes the Helm chart to private OCI registries. After authenticating with the registry Genesis assigned to your deployment, pull and unpack the chart locally. If Genesis provided a
.tgz archive instead, extract it and continue with the same chart directory layout:Chart releases are published from
main. On each GA release, Genesis updates appVersion in Chart.yaml to the promoted container version and patch-bumps the chart version automatically. To find the chart version for a specific GA release, query the GA_HELM_MAP repo variable: gh variable get GA_HELM_MAP -R genesis-computing-ai/genesis-agents | jq -r '.["vX.Y.Z"]'.2
Create Values File
Create a
genesis-values.yaml file with your configuration:For IaC-managed deployments, keep cluster settings in
genesis-values.yaml and copy the chart’s genesis-config-example.yaml
to a separate genesis-config.yaml for providers, llm, embedding,
auth, oauth_providers, and related app settings (including LiteLLM
Proxy OAuth). Deploy with
--set-file genesisConfig.content=genesis-config.yaml, or set
genesisConfig.existingConfigMap when the file already exists as a
ConfigMap. Genesis mounts the file at
/app/.genesis/config/genesis-config.yaml, and explicit environment
variables still take precedence over file-backed values.For Helm-driven RBAC on OIDC deployments, you can also pass
--set-file groupRoleMapping.content=group-role-mapping.yaml and
--set-file rbacSeedPolicy.content=rbac-policies.csv. See
Deploying Genesis - Azure AKS with OIDC Authentication.If you deploy with
aws-eks/deploy-eks-auth0.sh, aws-eks/deploy-eks-oidc.sh, or aws-eks/deploy-eks-proxy-stack.sh, pin image.tag in the corresponding values-eks-*.yaml file. Those scripts pass ecr.enabled, ecr.registry, a fresh ecr.token, and image.pullPolicy=Always to Helm on every run, and they only pass --set image.tag when GENESIS_IMAGE_TAG is set in aws-eks/eks-config.env.3
Deploy Genesis
4
Verify Deployment
5
Access Genesis
- Via Ingress (Recommended)
- Port Forward (For Testing)
For AWS Load Balancer Controller: Get the ALB address:Access via: Access via:
http://<alb-hostname> or https://genesis.yourdomain.comFor NGINX Ingress: Get the external IP:http://<external-ip> or http://genesis.yourdomain.com (if DNS configured)6
Configure the LLM Provider
Genesis starts without a provider and reports No suitable LLM found until
one is configured. Configure it against the running pod, either from
Config -> AI Providers in the UI or through the config API. Keep
credentials in a request-body file — process arguments are visible to every
process on the host and land in shell history.Read each response before running the next call: these routes also answer
200
with success: false (for example when the provider already exists — repeat
that call as a PUT). For the full endpoint reference, the card / llm-type
naming table and the other providers, see
Configuring a provider through the config API.Some schema fields carry a
default_env_var binding that seeds them from the
environment while nothing is stored, so an image can come up pre-seeded. That
is bootstrap only — it is not the way to configure a provider, and anything
saved through the UI or the config API outranks it.Kubernetes-Specific Configuration
StatefulSet
Genesis is deployed as a StatefulSet (not Deployment) to ensure:- Stable network identity (pod name:
genesis-0) - Ordered, graceful deployment and scaling
- Stable persistent storage (PVC name:
genesis-data-genesis-0)
Persistent Volume Claim
The StatefulSet creates a PersistentVolumeClaim with:Helm resource retention
Chart0.3.70 and later annotate the Genesis Namespace, sandbox Namespace and
RBAC, and both ServiceAccounts with helm.sh/resource-policy: keep. This keeps
those objects in place when they move to a separate platform release in a later
chart revision; it also means helm uninstall does not delete the Genesis
Namespace. Delete the namespace explicitly when you intend to remove the
deployment:
sandbox.enabled off or changing sandbox.namespace does not revoke those
bindings, and the old bindings can continue granting pod management and
namespace-read access to the instance’s ServiceAccount. Revoke all bindings for
an instance with:
sandbox.namespace, run the cleanup command and then re-apply
the Helm release so the current bindings are recreated. The command is also
needed after a direct helm uninstall; cluster/scripts/teardown.sh runs it as
part of teardown. Shared sandbox resources are retained for other instances
and should only be removed with the corresponding shared-resource retirement
workflow after the last instance is gone.
The chart reserves helm.sh/resource-policy on retained objects. Do not set
that key under serviceAccount.annotations or
sandbox.podServiceAccount.annotations; Helm rejects the render and names the
values path. A rollback to a chart version older than 0.3.70 removes the
annotation from live objects, so re-check it after rollback:
Volume Mounts
The Genesis container mounts the persistent volume at multiple paths:Health Checks
Genesis uses three Kubernetes probes so slow startups do not trigger liveness restarts before the app is ready:Startup Probe
- Endpoint:
GET /api/healthon port 8080 - Period: 10 seconds
- Timeout: 5 seconds
- Failure threshold: 60 (up to ~10 minutes boot grace)
- Gates liveness and readiness until startup completes (
503with progress, then200)
Liveness Probe
- Endpoint:
GET /api/health/liveon port 8080 - Period: 30 seconds
- Timeout: 5 seconds
- Failure threshold: 3
- Always
200while the process is alive — does not require Genesis to be fully ready
Readiness Probe
- Endpoint:
GET /api/healthon port 8080 - Period: 10 seconds
- Timeout: 5 seconds
- Failure threshold: 3
- Returns
503while booting;200when ready to serve traffic
authProvider: "proxy", for example via aws-eks/deploy-eks-proxy-stack.sh), configure the proxy so GET health probes and the dashboard healthcheck endpoint bypass authentication. The bundled aws-eks/oauth2-proxy-values-eks-example.yaml includes:
cookie_secure = true is correct when HTTPS terminates at the ingress controller or external load balancer, even if the OAuth2 Proxy chart ingress has tls: []. Routes are GET-scoped and anchored so other /api/health/* paths (errors, threads, slots, and so on) remain authenticated. cookie_csrf_per_request isolates concurrent login flows and prevents invalid-CSRF 403 responses.
Auth-exempt health Ingress for auth_request deployments
If an NGINX ingress uses nginx.ingress.kubernetes.io/auth-url and
nginx.ingress.kubernetes.io/auth-signin, an external load balancer or uptime monitor
can receive the proxy’s login redirect instead of Genesis health status. Enable the
chart’s second, unauthenticated health Ingress for this deployment pattern:
ingress.authExemptHealth.enabled defaults to false. When enabled, the chart renders
<release>-health on the same hosts and TLS configuration as the main Ingress, routing
these exact paths to the FastAPI Service: /api/health, /api/health/startup,
/api/health/live, and /api/healthcheck, each with and without a trailing slash.
ingress.authExemptHealth.pathType defaults to Exact; keep it that way so the
authenticated /api/health/detailed and /api/health/slots routes are not exposed.
The backend port defaults to service.ports.fastapi (8080) and can be overridden with
ingress.authExemptHealth.port.
The health Ingress does not inherit ingress.annotations. Put only its own annotations
under ingress.authExemptHealth.annotations. For the AWS Load Balancer Controller,
add the ALB group annotation there so both Ingress resources use the same load balancer:
Graceful shutdown (podLifecycle)
The chart sets a preStop sleep and a pod-level termination grace period so ingress load balancers can drain in-flight requests before Genesis receives SIGTERM.
Extension containers (extraContainers, extraInitContainers)
Pass additional Kubernetes container specs through the chart without forking templates. extraInitContainers is rendered when customCACerts is enabled, when extraInitContainers is non-empty, or both.
Resource Requirements
Default resource requests:No resource limits are set by default to avoid OOM kills. Adjust based on your workload.
Vertical Pod Autoscaler (Optional)
Genesis runs as a single-replica StatefulSet backed by aReadWriteOnce
persistent volume, so horizontal pod autoscaling is not supported. If you want
automatic right-sizing for resources.requests, enable the chart’s optional
Vertical Pod Autoscaler (VPA) support instead.
Prerequisites:
metrics-servermust be installed in the cluster.- Install a VPA controller. From the Helm chart directory, you can use the bundled helper:
Example: recommendation-only mode
AWS-Specific Considerations
IAM Roles for Service Accounts (IRSA)
For production, configure IRSA so the Genesis pod can access AWS services without storing long-lived credentials. This is the recommended pattern for AWS Bedrock IAM authentication.- Create IAM Role with Bedrock permissions
- Create Service Account with role annotation:
- Configure the Helm release to use the annotated service account:
Agent sandbox execution (optional)
Genesis agents can run long-running Python jobs, shell commands, and scripts in isolated Kubernetes pods instead of the application container’s local Docker runtime. On EKS, enable this through Helmsandbox values. The Genesis
ServiceAccount needs IRSA for the Genesis pod itself (Bedrock, Secrets Manager,
and other control-plane AWS calls). Sandbox workload pods run in a separate
namespace and need their own ServiceAccount with IRSA when agents call AWS APIs
from inside the sandbox (for example boto3, S3, or Bedrock).
When
sandbox.enabled is true, the chart grants the release ServiceAccount
pod lifecycle permissions in the sandbox namespace (create, exec, delete,
get, list, watch on pods and pod logs) and read access to the sandbox
namespace object itself.
Example: EKS lab instance
serviceAccount.create must be true so the chart can bind sandbox RBAC to the
Genesis ServiceAccount. Set sandbox.podServiceAccount.create to true and
annotate sandbox.podServiceAccount.annotations with an IRSA role trusted for
<sandbox.namespace>:<sandbox.podServiceAccount.name> so sandbox pods receive
AWS credentials. When starting sandboxes, pass the same name via
backend_options.service_account (or spec.k8s.service_account) — for example
genesis when using the default sandbox.podServiceAccount.name.
The configured defaultBackend maps to sandbox.default_backend in Genesis
APP_CONFIG. Agents that call sandbox_action without an explicit backend (or
with backend="local") use that operator default instead of local Docker. See
sandbox_action.
Bootstrap secrets via config.extraEnv (AWS Secrets Manager)
On EKS with IRSA, you can reference AWS Secrets Manager secrets in Helm
config.extraEnv. Genesis resolves aws_* shorthand or full ARNs at pod
startup using the pod’s IRSA credentials — no init container, Kubernetes
Secret, or CSI driver is required for the reference itself.
Grant secretsmanager:GetSecretValue on the secret to the IRSA role (same as
Custom CA Certificates and
External Secrets). After you rotate a value in
AWS Secrets Manager, restart the StatefulSet so the pod re-resolves it:
values_databricks_llm.yaml):
--set secrets.databricksApiKey="dapi...") or mount a
pre-created Kubernetes secret via secrets.existingSecretDatabricksApiKey.
The bundled values_databricks_llm.yaml sample leaves config.modelName,
config.harvesterModel, config.fastModelName and config.embeddingModel
empty, so the catalog default for each role applies: databricks-claude-opus-5-5
for the main and harvester models, databricks-claude-haiku-4-5 for the fast
model, and databricks-gte-large-en for embeddings. Set any of these fields to
seed a different model when no corresponding model is already saved in AppConfig.
All three forms are default_env_var bootstrap seeding: the value is captured
only while nothing is stored for the field, and it is outranked by anything saved
through the UI or the config API. Seeding a credential does not configure or
activate the provider — do that from
Configuring a provider through the config API.
Example: Master encryption key from AWS Secrets Manager (leave
genesisMasterKey empty and resolve via IRSA at pod startup):
secretsmanager:GetSecretValue on the referenced secret to the IRSA
role. If GENESIS_MASTER_KEY is also set in the pod environment, it takes
precedence over GENESIS_MASTER_KEY_REF. See
Master encryption key
for the full resolution order.
Some non-production deployments grant Bedrock permissions to the EKS node
role. That is acceptable for isolated testing, but IRSA is preferred for
production because it scopes permissions to the Genesis service account rather
than all pods on a node.
Custom CA Certificates from AWS Secrets Manager
If Genesis must connect to internal HTTPS endpoints signed by a private CA, the Helm chart can fetch CA certificates from AWS Secrets Manager during pod startup and merge them into the container trust store. It uses the same secret reference syntax documented in External Secrets.Grant
secretsmanager:GetSecretValue on each referenced secret to the IRSA
role. When enabled, the chart mounts the merged CA bundle and sets
REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, SSL_CERT_FILE, GIT_SSL_CAINFO, and
NODE_EXTRA_CA_CERTS so Python HTTP clients, curl, Git clone/fetch, and
Node.js trust the same custom CAs as API calls.fetch-ca-certs init container uses the image’s
/etc/ssl/certs/ca-certificates.crt system bundle to contact Secrets Manager.
In that container only, the chart overrides SSL_CERT_FILE,
REQUESTS_CA_BUNDLE, CURL_CA_BUNDLE, and AWS_CA_BUNDLE from
config.extraEnv; this prevents paths to a custom bundle that is not available
until Genesis starts from breaking certificate retrieval. The Genesis
container continues to use the merged custom CA bundle.
When you rotate a certificate in Secrets Manager, restart the StatefulSet so
the pod rebuilds the trust bundle:
EBS Volume Encryption
EBS volumes created by the EBS CSI driver on EKS are encrypted by default if your cluster has encryption enabled. Verify:VPC Configuration
Ensure your EKS cluster nodes have:Troubleshooting
Pod Not Starting
Pod Not Starting
- Image pull errors: Check network connectivity to the container registry provided by Genesis
- PVC issues: Verify EBS CSI driver is installed and storage class exists
- Resource constraints: Check node resources
Persistent Volume Issues
Persistent Volume Issues
Ingress Not Working
Ingress Not Working
For AWS Load Balancer Controller:For NGINX Ingress:
Database Issues
Database Issues
Viewing Logs
Viewing Logs
Upgrading Genesis
When a new Genesis version is released:The StatefulSet will perform a rolling update, and the persistent volume will be reattached to the new pod.
For values-driven upgrades, update
image.tag in the relevant values-eks-*.yaml file and rerun the matching aws-eks/deploy-eks-*.sh helper. If you use aws-eks/update-eks-image.sh or aws-eks/update-eks-helm.sh, both helpers require ECR_REGISTRY in aws-eks/eks-config.env, default to v1.18.2 when no tag argument is provided, refresh the ECR token on every run, and perform a kubectl rollout restart after helm upgrade so EKS nodes pull the updated image and pull secret.Uninstalling
Additional Resources
AWS EKS Documentation
Official Amazon EKS documentation
EBS CSI Driver
EBS CSI Driver setup guide
AWS Load Balancer Controller
AWS Load Balancer Controller documentation

