> ## Documentation Index
> Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Datasets
> Manage reusable fast-data datasets used by Genesis apps, dashboards, and recurring analysis workflows.
## Overview
The **Datasets** page is where customers manage reusable fast-data layers that power apps, dashboards, and analytical workflows.
## What you can do
* review available datasets,
* filter datasets by type or status,
* refresh datasets,
* materialize or update datasets where supported,
* delete user-created datasets,
* inspect details and preview data,
* manage sharing rules,
* connect refresh behavior to scheduling workflows,
* jump to the owning chat thread or dashboard when a dataset is scoped to one.
When a dataset is scoped to a thread, the details panel shows **Open thread** only when the scope id includes both the agent and thread in `{bot_id}--{thread_id}` form (for example `Eve--chat_abc123`). Genesis opens the canonical chat URL `/chat/{bot_id}/thread/{thread_id}`. Thread scopes that omit a bot identifier do not show a link because the conversation is not addressable without one.
## When to use datasets
Use a dataset when you want a stable, reusable data layer for an app or repeatable workflow without rebuilding the full upstream query every time.
## Refresh policies
Each dataset definition stores a `refresh_policy` object. Genesis validates and
normalizes that policy when you create or update the dataset, then
automatically syncs scheduled policies into the scheduler.
| Mode | Required fields | Notes |
| - | - | - |
| `manual` | none | Default. Refresh only when a user, API call, or agent tool requests it. |
| `scheduled` | `schedule` (optional, defaults to `@daily`) | Accepted formats: `@daily`, `@hourly`, bare 5-field cron strings such as `*/15 * * * *`, `cron:<5 fields>`, `interval:`, or a JSON trigger object. |
| `ttl` | `ttl_seconds` | `ttl_seconds` must be an integer between `60` and `31536000`. |
| `on_demand` | none | Use when an external workflow decides when to call refresh and you do not want a stored schedule or TTL. |
Scheduled refreshes default to `@daily` when `schedule` is omitted. `cron:`
schedules use five fields in the order minute, hour, day, month, and
day\_of\_week.
## Definition guardrails
Genesis validates dataset definitions before it saves or refreshes them:
* unknown keys in the request root, `sources[*]`, or `views[*]` fail fast
instead of being ignored,
* SQL sources must be exactly one read-only `SELECT` or `WITH` query, even
when the SQL is loaded through `sql_path`,
* `sources[*].max_rows` is optional, but when you set it the value must be at
least `1`,
* source IDs and view IDs must stay unique case-insensitively, view dependency
cycles are rejected, and index targets and indexed columns must exist before
Genesis will persist the definition.
When `sql_path` points to a recorded upload, an authorized dataset refresh can retrieve its
missing SQL bytes from the Databricks App volume before parsing the definition; the initial
refresh after a restart waits for this retrieval. See [volume-backed upload behavior](/docs/deployments/Databricks/Genesis-On-Databricks#durable-chat-uploads).
## Partial refreshes and snapshot history
A refresh normally rebuilds all sources in the dataset, but API and agent-tool
callers can pass `only_source_ids` to refresh only a subset of sources.
* Genesis validates `only_source_ids` before it starts the refresh.
* If a requested source ID does not exist in the dataset definition, the
request fails with a `400` and includes both `unknown_source_ids` and
`available_source_ids`.
* A failed partial refresh never replaces the active snapshot.
Datasets keep immutable snapshot history behind the active snapshot pointer:
* refreshing creates a new snapshot and switches `active_snapshot_id` only
after the new snapshot is ready,
* you can list snapshots, roll back to a prior `ready` snapshot, or delete
inactive snapshots,
* Genesis never deletes the active snapshot through the inactive-snapshot
cleanup path, even if it is requested explicitly.
On Databricks Apps with durable SQLite enabled, materialized dataset snapshots
survive host replacement; query-result cache datasets do not. See [Durable
SQLite stores on Databricks](/docs/deployments/Databricks/Genesis-On-Databricks#durable-sqlite-stores)
for restore timing and cache behavior. The snapshot size limit is configured
with `sqlite_durability.user_db_max_bytes` in [Advanced Settings](/docs/Setup/Advanced-Settings#durable-dataset-snapshot-size-limit).
## Deletion safeguards
Genesis lets you delete user-created datasets from the **Datasets** page, but it adds guardrails around destructive actions:
* built-in system datasets such as the Cost & Usage cube and Tool Observability cube cannot be deleted,
* if a dataset is still bound to one or more apps, Genesis shows the affected apps and requires a second confirmation before the delete proceeds,
* the confirmation flow is there to prevent breaking apps that still depend on the dataset binding.
## Automatic retention and cleanup
Genesis also applies background retention to dataset storage. The active
snapshot is never deleted by this cleanup flow.
| Item | Default behavior | How to change it |
| - | - | - |
| Query-cache datasets | If `refresh_policy.ttl_seconds` is not set, query-cache datasets expire after 7 days. Genesis deletes the dataset record and its snapshot files together. | Set `refresh_policy.ttl_seconds` on the dataset. `jobs.dataset_cleanup.enabled` defaults to `true`; `GENESIS_DATASET_CLEANUP_INTERVAL_S` controls the scheduled cleanup interval in seconds (default `21600` on block-backed storage and `0` on stage-backed storage). Set the environment variable to a positive number to enable or change the interval; set it to `0` to disable cleanup. |
| Built-in protected system dataset snapshots | Inactive snapshots older than `GENESIS_SYSTEM_DATASET_SNAPSHOT_TTL_DAYS` are eligible for cleanup. Default: `7` days. | Set `GENESIS_SYSTEM_DATASET_SNAPSHOT_TTL_DAYS` to a positive integer. |
| Built-in protected system dataset snapshot history | Genesis also caps inactive snapshot history for protected system datasets at `GENESIS_SYSTEM_DATASET_MAX_SNAPSHOTS`. Default: `20`. | Set `GENESIS_SYSTEM_DATASET_MAX_SNAPSHOTS` to a positive integer. |
When scheduled cleanup is enabled, its first pass runs `120` seconds after
server start, then runs at the configured interval. Configure
`jobs.dataset_cleanup.startup_delay_seconds` (default
`120`) to change the initial delay. Each pass drains eligible datasets and
snapshots in batches of up to `jobs.dataset_cleanup.batch_size` (default `100`
of each), until no candidates remain, a batch deletes nothing, or the
`jobs.dataset_cleanup.pass_budget_seconds` (default `600`) is reached. The batch
that crosses the time budget finishes; remaining candidates wait for the next
pass. These settings are read at the start of each pass, except the startup
delay, which is read when the scheduler registers the job. After a successful
dataset refresh, Genesis also applies the snapshot policy to superseded
snapshots; the active snapshot remains protected.
## Related pages
* [Data Explorer](/docs/User-Guide/Data-Explorer)
* [Genesis Apps](/docs/genesis-apps)
* [Scheduler](/docs/User-Guide/Scheduler)