> ## Documentation Index > Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt > Use this file to discover all available pages before exploring further. # Backup and Recovery > Protect Genesis state on AWS EKS with volume snapshots and standard Kubernetes backup tooling. ## Overview Genesis on EKS stores all persistent state on a single PersistentVolumeClaim (`genesis-data`) backed by an EBS volume. Backing up Genesis means taking periodic snapshots of that volume and preserving the Kubernetes Secret that holds the master encryption key. Genesis is a well-behaved StatefulSet tenant: one namespace, one PVC, standard Kubernetes primitives. Any namespace-aware backup tool — Velero, AWS Backup for EKS, Kasten K10, or manual EBS snapshots — works without application-level changes. The one wrinkle is **SQLite consistency**. Genesis uses multiple SQLite databases in WAL mode, so a raw block-level snapshot of an active volume can capture a mid-transaction state. A short pre-backup hook checkpoints the WAL files to make the snapshot consistent. ### Incremental object-store snapshots For a portable snapshot of the Helm chart's PVC layout, run the bundled `export-state` command while the Genesis server is stopped: ```bash theme={null} uv run export-state s3://my-bucket/genesis-snapshots/ \ --region us-west-2 \ --kms-key-id ``` The destination can also be a `gs://` bucket or a local directory, for example `/mnt/backups/`. For an S3 destination, Genesis uses the ambient AWS credential chain (pod identity, instance profile, `AWS_PROFILE`/SSO, or `AWS_*` environment variables). For GCS, it uses the ambient Google identity. The S3 region defaults to `AWS_REGION`, then `AWS_DEFAULT_REGION`, then `us-east-1`; omit `--kms-key-id` to use the AWS-managed S3 key. GCS uses the bucket's default encryption and does not accept `--kms-key-id`. To request the same snapshot from a running instance, use the authenticated admin API instead of stopping the server manually. See the [state snapshot admin endpoints](/docs/Genesis-API/API-Reference#post-apiadminstate-snapshots) for the request body, required permissions and example. The API validates the S3 or GCS destination, returns `202` with a job ID, then pauses the instance while the exporter copies its state and restarts the server. Poll the job after the server is back to read the resulting snapshot ID. See [State Snapshots and Migration](/docs/deployments/State-Snapshots-And-Migration) for GCS bucket permissions, Workload Identity, and the API connection fields. Each completed snapshot is stored below the destination in a new snapshot ID: ```text theme={null} / ├── db/ ├── bot_storage/ ├── bot_git/ └── manifest.json ``` The layout matches the PVC subpaths above. The manifest is written last, so an incomplete export is ignored. When a previous complete snapshot exists under the same S3 destination, unchanged files are copied server-side and only changed files are uploaded. The export excludes `.microapp_venv/`, `.microapp_home/`, and `__pycache__/`; repository `.git` directories keep their real name. Genesis refuses to export while its server is running: the SQLite database is copied consistently, and stopping the server keeps the directory trees consistent with that database. ### Restoring an object-store snapshot The Genesis Helm chart can restore a completed object-store snapshot into the `genesis-data` PVC before the first Genesis container starts. Add the following to your values file: ```yaml theme={null} restore: from: s3://my-bucket/genesis-snapshots/ region: us-west-2 resources: {} ``` `restore.from` accepts either an `s3://` or `gs://` snapshot path, such as `s3://my-bucket/genesis-snapshots//`, or an instance prefix, which selects its newest complete snapshot. The default is `""`, which disables restore. `restore.region` defaults to `""` and specifies the AWS region for an S3 bucket; it is unused for GCS. `restore.resources` defaults to `{}` and optionally sets the restore init container's Kubernetes resource requests and limits. For GCS, the init container reads with the pod's Google identity; see [State Snapshots and Migration](/docs/deployments/State-Snapshots-And-Migration) for Workload Identity setup. The restore init container uses the pod's identity for object-store access, including IRSA on EKS. It receives the master key from the same configuration as the Genesis container: `secrets.genesisMasterKey`, `secrets.existingSecretGenesisMasterKey`, or `config.extraEnv.GENESIS_MASTER_KEY_REF`. The snapshot must use the same master key as the target instance. The restore refuses a different key, an unsupported or newer release snapshot, an invalid signature, any missing, extra, or corrupted file, and a database that fails SQLite's integrity check. The chart requires an image that includes the restore runner. Release image tags at or below `v1.19.93` are rejected during chart rendering when `restore.from` is set; use a newer image. A tag that is not a plain semantic version is not ordered by the chart, so verify that it includes the restore runner before deploying. The init container only restores an empty volume. If the target already holds Genesis state, it skips the restore on pod startup, including after a restart or upgrade. Failures during discovery, verification, or staging leave the existing volume unchanged, and the init container retries on the next pod start. If a restart interrupts the final tree swap, the next init-container run finishes that verified swap before Genesis starts. To restore again, use an empty PVC, or stop Genesis and run the import runner with `--force` in a pod that mounts the volume: ```bash theme={null} python3 -m scripts.uv.import_state_runner \ --from s3://my-bucket/genesis-snapshots// \ --target-root /genesis-data \ --runtime-root /app \ --force \ --yes ``` Use `--runtime-root /app` when the snapshot's source paths were recorded under a different runtime layout, such as a Snowflake SPCS migration. The restore rewrites matching absolute paths as part of staging; see [State Snapshots and Migration](/docs/deployments/State-Snapshots-And-Migration) for the path rewrite behavior and migration workflow. ## What lives on the PVC The `genesis-data` PVC is mounted at four subpaths: | Mount path | SubPath | Contents | | - | - | - | | `/app/.genesis/db` | `db` | Primary SQLite database (`genesis.db`) | | `/app/bot_storage` | `bot_storage` | Threads, conversations, RBAC DB, flows DB, debriefs DB, scheduler DBs, uploaded files, agent skills, blueprints | | `/app/bot_git` | `bot_git` | Cloned git repositories | | `/app/tmp` | `tmp` | Runtime temp files (not critical to back up) | Everything stateful is on this single volume. The container image, ConfigMaps, and environment variables are stateless and reconstructed from the Helm chart. **The master encryption key is not on the PVC.** It lives outside the volume — in a Kubernetes `Secret` as `GENESIS_MASTER_KEY`, or resolved at startup from AWS/GCP when `GENESIS_MASTER_KEY_REF` is set — and is required to decrypt connection credentials stored in `genesis.db`. Restoring the volume without the matching master key renders encrypted fields unusable. Always back up the master key separately — see [Backing up the master key](#backing-up-the-master-key). ## Backing up the master key If you deploy with `config.extraEnv.GENESIS_MASTER_KEY_REF`, the master key already lives in AWS Secrets Manager or GCP Secret Manager. Ensure that secret is included in your organization's vault backup and rotation policies — you do not need a separate Kubernetes `Secret` export. Export and store the master key in a secure location outside the cluster (when it is stored in the chart-managed Kubernetes `Secret`): ```bash theme={null} kubectl get secret -n genesis genesis \ -o jsonpath='{.data.GENESIS_MASTER_KEY}' | base64 -d > genesis-master-key.txt chmod 600 genesis-master-key.txt ``` Import the file into your organization's secrets vault (AWS Secrets Manager, HashiCorp Vault, etc.) and delete the local copy immediately: ```bash theme={null} # Example: store in AWS Secrets Manager aws secretsmanager create-secret \ --name genesis/master-key \ --secret-string file://genesis-master-key.txt rm genesis-master-key.txt ``` If you rotate the master key, update the backup immediately. If you use External Secrets (AWS Secrets Manager or GCP Secret Manager) for connection credentials, those secrets survive independently of Genesis backups. See [External Secrets](/docs/Setup/External-Secrets) for details. ## SQLite pre-backup hook Genesis ships a checkpoint script at `scripts/sqlite-checkpoint-hook.sh` that flushes all SQLite WAL files on the PVC before a snapshot. The script finds every `.db`, `.sqlite`, and `.sqlite3` file under `/app/.genesis/db` and `/app/bot_storage` and runs `PRAGMA wal_checkpoint(TRUNCATE)` on each. You can run it manually: ```bash theme={null} kubectl exec -n genesis genesis-0 -- /app/scripts/sqlite-checkpoint-hook.sh ``` Or wire it into your backup tool as a pre-snapshot hook (see examples below). The checkpoint takes a few seconds at most. It does not stop the application or interrupt running conversations. It flushes pending writes from WAL files into the main database files so the on-disk state is self-consistent. ## Backup tool setup ### Prerequisites * [Velero](https://velero.io/) installed in the cluster with the [AWS plugin](https://github.com/vmware-tanzu/velero-plugin-for-aws) * An S3 bucket for backup storage * The EBS CSI driver installed (required for CSI snapshots) ### Configure pod annotations Add Velero backup hook annotations to your Helm `values.yaml`: ```yaml theme={null} podAnnotations: pre.hook.backup.velero.io/command: '["/app/scripts/sqlite-checkpoint-hook.sh"]' pre.hook.backup.velero.io/timeout: "60s" pre.hook.backup.velero.io/on-error: Fail ``` Upgrade the release: ```bash theme={null} helm upgrade genesis ./genesis -f values.yaml ``` ### Create a backup schedule ```bash theme={null} velero schedule create genesis-daily \ --schedule="0 2 * * *" \ --include-namespaces genesis \ --ttl 720h ``` This takes a daily backup at 02:00 UTC and retains backups for 30 days. ### Manual backup ```bash theme={null} velero backup create genesis-manual --include-namespaces genesis --wait ``` ### Restore ```bash theme={null} # List available backups velero backup get # Restore from a specific backup velero restore create --from-backup genesis-daily-20260505020000 ``` After the restore completes, the Genesis pod restarts automatically against the restored PVC. Verify with: ```bash theme={null} kubectl get pods -n genesis kubectl exec -n genesis genesis-0 -- sqlite3 /app/.genesis/db/genesis.db "PRAGMA integrity_check;" ``` ### Prerequisites * [AWS Backup](https://docs.aws.amazon.com/aws-backup/latest/devguide/whatisbackup.html) with EKS support enabled * The EBS CSI driver installed * An IAM role for AWS Backup with permissions to snapshot EBS volumes ### Create a backup plan Use the AWS Console or CLI to create a backup plan that targets the Genesis EKS cluster and namespace: ```bash theme={null} aws backup create-backup-plan --backup-plan '{ "BackupPlanName": "genesis-daily", "Rules": [{ "RuleName": "daily-snapshot", "ScheduleExpression": "cron(0 2 * * ? *)", "TargetBackupVaultName": "Default", "Lifecycle": { "DeleteAfterDays": 30 } }] }' ``` Assign the Genesis EKS cluster resources to the plan using resource selection by tag or ARN. ### Pre-backup checkpoint AWS Backup for EKS does not support pod-level pre-backup hooks natively. To ensure SQLite consistency, run the checkpoint on a schedule slightly before the backup window: ```yaml theme={null} # Add via extraManifests in values.yaml extraManifests: # RBAC: allow the checkpoint CronJob to exec into the Genesis pod - apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: genesis-checkpoint namespace: genesis rules: - apiGroups: [""] resources: ["pods"] verbs: ["get", "list"] - apiGroups: [""] resources: ["pods/exec"] verbs: ["create"] - apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: genesis-checkpoint namespace: genesis roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: genesis-checkpoint subjects: - kind: ServiceAccount name: genesis namespace: genesis - apiVersion: batch/v1 kind: CronJob metadata: name: genesis-sqlite-checkpoint namespace: genesis spec: schedule: "55 1 * * *" jobTemplate: spec: template: spec: containers: - name: checkpoint image: bitnami/kubectl:1.31 command: - kubectl - exec - -n - genesis - genesis-0 - -- - /app/scripts/sqlite-checkpoint-hook.sh restartPolicy: OnFailure serviceAccountName: genesis ``` The CronJob runs 5 minutes before the backup window. This is a best-effort consistency window — for guaranteed consistency, use Velero with its native pre-backup hooks. ### Restore Use the AWS Console or CLI to restore from a recovery point: ```bash theme={null} aws backup start-restore-job \ --recovery-point-arn \ --iam-role-arn \ --metadata '{"clusterName":"","newCluster":"false"}' \ --resource-type EKS ``` For environments without Velero or AWS Backup, you can take manual EBS snapshots. ### Identify the EBS volume ```bash theme={null} # Get the PV backing the Genesis PVC PV=$(kubectl get pvc -n genesis genesis-data-genesis-0 \ -o jsonpath='{.spec.volumeName}') # Get the EBS volume ID VOL_ID=$(kubectl get pv "$PV" \ -o jsonpath='{.spec.csi.volumeHandle}') echo "EBS Volume: $VOL_ID" ``` ### Checkpoint and snapshot ```bash theme={null} # Flush SQLite WAL files kubectl exec -n genesis genesis-0 -- /app/scripts/sqlite-checkpoint-hook.sh # Take the snapshot aws ec2 create-snapshot \ --volume-id "$VOL_ID" \ --description "Genesis backup $(date -u +%Y-%m-%dT%H:%M:%SZ)" \ --tag-specifications "ResourceType=snapshot,Tags=[{Key=Name,Value=genesis-backup}]" ``` ### Restore from snapshot 1. Create a new EBS volume from the snapshot 2. Create a PersistentVolume pointing to the new volume 3. Update the PVC to bind to the new PV (or delete and recreate the StatefulSet with the new PVC) 4. Verify the pod starts and the database is intact ```bash theme={null} kubectl exec -n genesis genesis-0 -- sqlite3 /app/.genesis/db/genesis.db "PRAGMA integrity_check;" ``` ## Recovery checklist After restoring from any backup method: 1. **Verify the pod is running:** ```bash theme={null} kubectl get pods -n genesis -w ``` 2. **Check database integrity:** ```bash theme={null} kubectl exec -n genesis genesis-0 -- sqlite3 /app/.genesis/db/genesis.db "PRAGMA integrity_check;" ``` 3. **Verify the master key matches:** If the Kubernetes Secret was also restored, confirm it contains the correct `GENESIS_MASTER_KEY`. If you use `GENESIS_MASTER_KEY_REF`, confirm the referenced AWS/GCP secret still exists and the pod's IRSA/ADC identity can read it. If restoring into a new cluster, recreate the Secret or `GENESIS_MASTER_KEY_REF` configuration from your vault backup. 4. **Check the application health endpoint:** ```bash theme={null} kubectl exec -n genesis genesis-0 -- curl -s http://localhost:8080/api/health ``` 5. **Verify data:** Log into the Genesis UI and confirm agents, threads, and connections are present. Genesis includes automatic SQLite corruption detection and recovery (`sqlite_recovery_core.py`). If the database is slightly inconsistent after a restore, Genesis will attempt to repair it on startup. ## RPO and RTO guidance | | Daily backups | More frequent backups | | - | - | - | | **RPO (data loss)** | Up to 24 hours of threads, conversations, and configuration changes | Proportional to backup interval | | **RTO (downtime)** | Minutes — restore PVC, pod restarts automatically | Same | To reduce RPO impact: * **Blueprints and missions** can be re-deployed from source control * **Connections** require the master key; with it, they restore fully from the PVC backup * **External secrets** (AWS Secrets Manager / GCP Secret Manager) survive independently ## Supplemental: configuration export `genesis config export` exports Genesis configuration to a YAML or JSON file that can be version-controlled independently of PVC backups: ```bash theme={null} kubectl exec -n genesis genesis-0 -- genesis config export > genesis-config-backup.yaml ``` This captures system configuration but not thread history, uploaded files, or connection credentials. ## Related pages * [Deploying Genesis on AWS](/docs/deployments/AWS/Genesis-On-AWS) * [External Secrets](/docs/Setup/External-Secrets)