Backup and Recovery

Overview

Genesis on EKS stores all persistent state on a single PersistentVolumeClaim (genesis-data) backed by an EBS volume. Backing up Genesis means taking periodic snapshots of that volume and preserving the Kubernetes Secret that holds the master encryption key. Genesis is a well-behaved StatefulSet tenant: one namespace, one PVC, standard Kubernetes primitives. Any namespace-aware backup tool — Velero, AWS Backup for EKS, Kasten K10, or manual EBS snapshots — works without application-level changes. The one wrinkle is SQLite consistency. Genesis uses multiple SQLite databases in WAL mode, so a raw block-level snapshot of an active volume can capture a mid-transaction state. A short pre-backup hook checkpoints the WAL files to make the snapshot consistent.

Incremental object-store snapshots

For a portable snapshot of the Helm chart’s PVC layout, run the bundled export-state command while the Genesis server is stopped:

uv run export-state s3://my-bucket/genesis-snapshots/<instance> \
  --region us-west-2 \
  --kms-key-id <customer-managed-kms-key-id>

The destination can also be a gs:// bucket or a local directory, for example /mnt/backups/<instance>. For an S3 destination, Genesis uses the ambient AWS credential chain (pod identity, instance profile, AWS_PROFILE/SSO, or AWS_* environment variables). For GCS, it uses the ambient Google identity. The S3 region defaults to AWS_REGION, then AWS_DEFAULT_REGION, then us-east-1; omit --kms-key-id to use the AWS-managed S3 key. GCS uses the bucket’s default encryption and does not accept --kms-key-id. To request the same snapshot from a running instance, use the authenticated admin API instead of stopping the server manually. See the state snapshot admin endpoints for the request body, required permissions and example. The API validates the S3 or GCS destination, returns 202 with a job ID, then pauses the instance while the exporter copies its state and restarts the server. Poll the job after the server is back to read the resulting snapshot ID. See State Snapshots and Migration for GCS bucket permissions, Workload Identity, and the API connection fields. Each completed snapshot is stored below the destination in a new snapshot ID:

<snapshot-id>/
├── db/
├── bot_storage/
├── bot_git/
└── manifest.json

The layout matches the PVC subpaths above. The manifest is written last, so an incomplete export is ignored. When a previous complete snapshot exists under the same S3 destination, unchanged files are copied server-side and only changed files are uploaded. The export excludes .microapp_venv/, .microapp_home/, and __pycache__/; repository .git directories keep their real name. Genesis refuses to export while its server is running: the SQLite database is copied consistently, and stopping the server keeps the directory trees consistent with that database.

Restoring an object-store snapshot

The Genesis Helm chart can restore a completed object-store snapshot into the genesis-data PVC before the first Genesis container starts. Add the following to your values file:

restore:
  from: s3://my-bucket/genesis-snapshots/<instance>
  region: us-west-2
  resources: {}

restore.from accepts either an s3:// or gs:// snapshot path, such as s3://my-bucket/genesis-snapshots/<instance>/<snapshot-id>, or an instance prefix, which selects its newest complete snapshot. The default is "", which disables restore. restore.region defaults to "" and specifies the AWS region for an S3 bucket; it is unused for GCS. restore.resources defaults to {} and optionally sets the restore init container’s Kubernetes resource requests and limits. For GCS, the init container reads with the pod’s Google identity; see State Snapshots and Migration for Workload Identity setup. The restore init container uses the pod’s identity for object-store access, including IRSA on EKS. It receives the master key from the same configuration as the Genesis container: secrets.genesisMasterKey, secrets.existingSecretGenesisMasterKey, or config.extraEnv.GENESIS_MASTER_KEY_REF. The snapshot must use the same master key as the target instance. The restore refuses a different key, an unsupported or newer release snapshot, an invalid signature, any missing, extra, or corrupted file, and a database that fails SQLite’s integrity check. The chart requires an image that includes the restore runner. Release image tags at or below v1.19.93 are rejected during chart rendering when restore.from is set; use a newer image. A tag that is not a plain semantic version is not ordered by the chart, so verify that it includes the restore runner before deploying. The init container only restores an empty volume. If the target already holds Genesis state, it skips the restore on pod startup, including after a restart or upgrade. Failures during discovery, verification, or staging leave the existing volume unchanged, and the init container retries on the next pod start. If a restart interrupts the final tree swap, the next init-container run finishes that verified swap before Genesis starts. To restore again, use an empty PVC, or stop Genesis and run the import runner with --force in a pod that mounts the volume:

python3 -m scripts.uv.import_state_runner \
  --from s3://my-bucket/genesis-snapshots/<instance>/<snapshot-id> \
  --target-root /genesis-data \
  --runtime-root /app \
  --force \
  --yes

Use --runtime-root /app when the snapshot’s source paths were recorded under a different runtime layout, such as a Snowflake SPCS migration. The restore rewrites matching absolute paths as part of staging; see State Snapshots and Migration for the path rewrite behavior and migration workflow.

What lives on the PVC

The genesis-data PVC is mounted at four subpaths:

Mount path	SubPath	Contents
`/app/.genesis/db`	`db`	Primary SQLite database (`genesis.db`)
`/app/bot_storage`	`bot_storage`	Threads, conversations, RBAC DB, flows DB, debriefs DB, scheduler DBs, uploaded files, agent skills, blueprints
`/app/bot_git`	`bot_git`	Cloned git repositories
`/app/tmp`	`tmp`	Runtime temp files (not critical to back up)

Everything stateful is on this single volume. The container image, ConfigMaps, and environment variables are stateless and reconstructed from the Helm chart.

The master encryption key is not on the PVC. It lives outside the volume — in a Kubernetes Secret as GENESIS_MASTER_KEY, or resolved at startup from AWS/GCP when GENESIS_MASTER_KEY_REF is set — and is required to decrypt connection credentials stored in genesis.db. Restoring the volume without the matching master key renders encrypted fields unusable. Always back up the master key separately — see Backing up the master key.

Backing up the master key

If you deploy with config.extraEnv.GENESIS_MASTER_KEY_REF, the master key already lives in AWS Secrets Manager or GCP Secret Manager. Ensure that secret is included in your organization’s vault backup and rotation policies — you do not need a separate Kubernetes Secret export.

Export and store the master key in a secure location outside the cluster (when it is stored in the chart-managed Kubernetes Secret):

kubectl get secret -n genesis genesis \
  -o jsonpath='{.data.GENESIS_MASTER_KEY}' | base64 -d > genesis-master-key.txt
chmod 600 genesis-master-key.txt

Import the file into your organization’s secrets vault (AWS Secrets Manager, HashiCorp Vault, etc.) and delete the local copy immediately:

# Example: store in AWS Secrets Manager
aws secretsmanager create-secret \
  --name genesis/master-key \
  --secret-string file://genesis-master-key.txt

rm genesis-master-key.txt

If you rotate the master key, update the backup immediately.

If you use External Secrets (AWS Secrets Manager or GCP Secret Manager) for connection credentials, those secrets survive independently of Genesis backups. See External Secrets for details.

SQLite pre-backup hook

Genesis ships a checkpoint script at scripts/sqlite-checkpoint-hook.sh that flushes all SQLite WAL files on the PVC before a snapshot. The script finds every .db, .sqlite, and .sqlite3 file under /app/.genesis/db and /app/bot_storage and runs PRAGMA wal_checkpoint(TRUNCATE) on each. You can run it manually:

kubectl exec -n genesis genesis-0 -- /app/scripts/sqlite-checkpoint-hook.sh

Or wire it into your backup tool as a pre-snapshot hook (see examples below).

The checkpoint takes a few seconds at most. It does not stop the application or interrupt running conversations. It flushes pending writes from WAL files into the main database files so the on-disk state is self-consistent.

Backup tool setup

Velero
AWS Backup for EKS
Manual EBS Snapshot

Prerequisites

Velero installed in the cluster with the AWS plugin
An S3 bucket for backup storage
The EBS CSI driver installed (required for CSI snapshots)

Configure pod annotations

Add Velero backup hook annotations to your Helm values.yaml:

podAnnotations:
  pre.hook.backup.velero.io/command: '["/app/scripts/sqlite-checkpoint-hook.sh"]'
  pre.hook.backup.velero.io/timeout: "60s"
  pre.hook.backup.velero.io/on-error: Fail

Upgrade the release:

helm upgrade genesis ./genesis -f values.yaml

Create a backup schedule

velero schedule create genesis-daily \
  --schedule="0 2 * * *" \
  --include-namespaces genesis \
  --ttl 720h

This takes a daily backup at 02:00 UTC and retains backups for 30 days.

Manual backup

velero backup create genesis-manual --include-namespaces genesis --wait

Restore

# List available backups
velero backup get

# Restore from a specific backup
velero restore create --from-backup genesis-daily-20260505020000

After the restore completes, the Genesis pod restarts automatically against the restored PVC. Verify with:

kubectl get pods -n genesis
kubectl exec -n genesis genesis-0 -- sqlite3 /app/.genesis/db/genesis.db "PRAGMA integrity_check;"

Prerequisites

AWS Backup with EKS support enabled
The EBS CSI driver installed
An IAM role for AWS Backup with permissions to snapshot EBS volumes

Create a backup plan

Use the AWS Console or CLI to create a backup plan that targets the Genesis EKS cluster and namespace:

aws backup create-backup-plan --backup-plan '{
  "BackupPlanName": "genesis-daily",
  "Rules": [{
    "RuleName": "daily-snapshot",
    "ScheduleExpression": "cron(0 2 * * ? *)",
    "TargetBackupVaultName": "Default",
    "Lifecycle": { "DeleteAfterDays": 30 }
  }]
}'

Assign the Genesis EKS cluster resources to the plan using resource selection by tag or ARN.

Pre-backup checkpoint

AWS Backup for EKS does not support pod-level pre-backup hooks natively. To ensure SQLite consistency, run the checkpoint on a schedule slightly before the backup window:

# Add via extraManifests in values.yaml
extraManifests:
  # RBAC: allow the checkpoint CronJob to exec into the Genesis pod
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: Role
    metadata:
      name: genesis-checkpoint
      namespace: genesis
    rules:
      - apiGroups: [""]
        resources: ["pods"]
        verbs: ["get", "list"]
      - apiGroups: [""]
        resources: ["pods/exec"]
        verbs: ["create"]
  - apiVersion: rbac.authorization.k8s.io/v1
    kind: RoleBinding
    metadata:
      name: genesis-checkpoint
      namespace: genesis
    roleRef:
      apiGroup: rbac.authorization.k8s.io
      kind: Role
      name: genesis-checkpoint
    subjects:
      - kind: ServiceAccount
        name: genesis
        namespace: genesis
  - apiVersion: batch/v1
    kind: CronJob
    metadata:
      name: genesis-sqlite-checkpoint
      namespace: genesis
    spec:
      schedule: "55 1 * * *"
      jobTemplate:
        spec:
          template:
            spec:
              containers:
                - name: checkpoint
                  image: bitnami/kubectl:1.31
                  command:
                    - kubectl
                    - exec
                    - -n
                    - genesis
                    - genesis-0
                    - --
                    - /app/scripts/sqlite-checkpoint-hook.sh
              restartPolicy: OnFailure
              serviceAccountName: genesis

The CronJob runs 5 minutes before the backup window. This is a best-effort consistency window — for guaranteed consistency, use Velero with its native pre-backup hooks.

Restore

Use the AWS Console or CLI to restore from a recovery point:

aws backup start-restore-job \
  --recovery-point-arn <recovery-point-arn> \
  --iam-role-arn <backup-role-arn> \
  --metadata '{"clusterName":"<eks-cluster-name>","newCluster":"false"}' \
  --resource-type EKS

For environments without Velero or AWS Backup, you can take manual EBS snapshots.

Identify the EBS volume

# Get the PV backing the Genesis PVC
PV=$(kubectl get pvc -n genesis genesis-data-genesis-0 \
  -o jsonpath='{.spec.volumeName}')

# Get the EBS volume ID
VOL_ID=$(kubectl get pv "$PV" \
  -o jsonpath='{.spec.csi.volumeHandle}')

echo "EBS Volume: $VOL_ID"

Checkpoint and snapshot

# Flush SQLite WAL files
kubectl exec -n genesis genesis-0 -- /app/scripts/sqlite-checkpoint-hook.sh

# Take the snapshot
aws ec2 create-snapshot \
  --volume-id "$VOL_ID" \
  --description "Genesis backup $(date -u +%Y-%m-%dT%H:%M:%SZ)" \
  --tag-specifications "ResourceType=snapshot,Tags=[{Key=Name,Value=genesis-backup}]"

Restore from snapshot

Create a new EBS volume from the snapshot
Create a PersistentVolume pointing to the new volume
Update the PVC to bind to the new PV (or delete and recreate the StatefulSet with the new PVC)
Verify the pod starts and the database is intact

kubectl exec -n genesis genesis-0 -- sqlite3 /app/.genesis/db/genesis.db "PRAGMA integrity_check;"

Recovery checklist

After restoring from any backup method:

Verify the pod is running:
kubectl get pods -n genesis -w

Check database integrity:

kubectl exec -n genesis genesis-0 -- sqlite3 /app/.genesis/db/genesis.db "PRAGMA integrity_check;"

Verify the master key matches: If the Kubernetes Secret was also restored, confirm it contains the correct GENESIS_MASTER_KEY. If you use GENESIS_MASTER_KEY_REF, confirm the referenced AWS/GCP secret still exists and the pod’s IRSA/ADC identity can read it. If restoring into a new cluster, recreate the Secret or GENESIS_MASTER_KEY_REF configuration from your vault backup.

Check the application health endpoint:

kubectl exec -n genesis genesis-0 -- curl -s http://localhost:8080/api/health

Verify data: Log into the Genesis UI and confirm agents, threads, and connections are present.

Genesis includes automatic SQLite corruption detection and recovery (sqlite_recovery_core.py). If the database is slightly inconsistent after a restore, Genesis will attempt to repair it on startup.

RPO and RTO guidance

	Daily backups	More frequent backups
RPO (data loss)	Up to 24 hours of threads, conversations, and configuration changes	Proportional to backup interval
RTO (downtime)	Minutes — restore PVC, pod restarts automatically	Same

To reduce RPO impact:

Blueprints and missions can be re-deployed from source control
Connections require the master key; with it, they restore fully from the PVC backup
External secrets (AWS Secrets Manager / GCP Secret Manager) survive independently

Supplemental: configuration export

genesis config export exports Genesis configuration to a YAML or JSON file that can be version-controlled independently of PVC backups:

kubectl exec -n genesis genesis-0 -- genesis config export > genesis-config-backup.yaml

This captures system configuration but not thread history, uploaded files, or connection credentials.

Getting Started

Getting to Know Genesis

Creating A Mission

User Guide

Blueprint

Genesis Data Agents

Genesis Prover

Genesis Apps

Genesis REST API

Genesis Agent Security Controls

Genesis Data Agent's Toolkit

Setup

Slack and Teams

Data Connections

Deployment Options

Overview

Incremental object-store snapshots

Restoring an object-store snapshot

What lives on the PVC

Backing up the master key

SQLite pre-backup hook

Backup tool setup

Prerequisites

Configure pod annotations

Create a backup schedule

Manual backup

Restore

Prerequisites

Create a backup plan

Pre-backup checkpoint

Restore

Identify the EBS volume

Checkpoint and snapshot

Restore from snapshot

Recovery checklist

RPO and RTO guidance

Supplemental: configuration export

​Overview

​Incremental object-store snapshots

​Restoring an object-store snapshot

​What lives on the PVC

​Backing up the master key

​SQLite pre-backup hook

​Backup tool setup

​Prerequisites

​Configure pod annotations

​Create a backup schedule

​Manual backup

​Restore

​Prerequisites

​Create a backup plan

​Pre-backup checkpoint

​Restore

​Identify the EBS volume

​Checkpoint and snapshot

​Restore from snapshot

​Recovery checklist

​RPO and RTO guidance

​Supplemental: configuration export

​Related pages

Overview

Incremental object-store snapshots

Restoring an object-store snapshot

What lives on the PVC

Backing up the master key

SQLite pre-backup hook

Backup tool setup

Prerequisites

Configure pod annotations

Create a backup schedule

Manual backup

Restore

Prerequisites

Create a backup plan

Pre-backup checkpoint

Restore

Identify the EBS volume

Checkpoint and snapshot

Restore from snapshot

Recovery checklist

RPO and RTO guidance

Supplemental: configuration export

Related pages