Skip to main content

​
Overview

The Datasets page is where customers manage reusable fast-data layers that power apps, dashboards, and analytical workflows. Actual Genesis Datasets page showing dataset records and refresh status.

​
What you can do

  • review available datasets,
  • filter datasets by type or status,
  • refresh datasets,
  • materialize or update datasets where supported,
  • delete user-created datasets,
  • inspect details and preview data,
  • manage sharing rules,
  • connect refresh behavior to scheduling workflows,
  • jump to the owning chat thread or dashboard when a dataset is scoped to one.
When a dataset is scoped to a thread, the details panel shows Open thread only when the scope id includes both the agent and thread in {bot_id}--{thread_id} form (for example Eve--chat_abc123). Genesis opens the canonical chat URL /chat/{bot_id}/thread/{thread_id}. Thread scopes that omit a bot identifier do not show a link because the conversation is not addressable without one.

​
When to use datasets

Use a dataset when you want a stable, reusable data layer for an app or repeatable workflow without rebuilding the full upstream query every time.

​
Refresh policies

Each dataset definition stores a refresh_policy object. Genesis validates and normalizes that policy when you create or update the dataset, then automatically syncs scheduled policies into the scheduler.
Scheduled refreshes default to @daily when schedule is omitted. cron: schedules use five fields in the order minute, hour, day, month, and day_of_week.

​
Definition guardrails

Genesis validates dataset definitions before it saves or refreshes them:
  • unknown keys in the request root, sources[*], or views[*] fail fast instead of being ignored,
  • SQL sources must be exactly one read-only SELECT or WITH query, even when the SQL is loaded through sql_path,
  • sources[*].max_rows is optional, but when you set it the value must be at least 1,
  • source IDs and view IDs must stay unique case-insensitively, view dependency cycles are rejected, and index targets and indexed columns must exist before Genesis will persist the definition.
When sql_path points to a recorded upload, an authorized dataset refresh can retrieve its missing SQL bytes from the Databricks App volume before parsing the definition; the initial refresh after a restart waits for this retrieval. See volume-backed upload behavior.

​
Partial refreshes and snapshot history

A refresh normally rebuilds all sources in the dataset, but API and agent-tool callers can pass only_source_ids to refresh only a subset of sources.
  • Genesis validates only_source_ids before it starts the refresh.
  • If a requested source ID does not exist in the dataset definition, the request fails with a 400 and includes both unknown_source_ids and available_source_ids.
  • A failed partial refresh never replaces the active snapshot.
Datasets keep immutable snapshot history behind the active snapshot pointer:
  • refreshing creates a new snapshot and switches active_snapshot_id only after the new snapshot is ready,
  • you can list snapshots, roll back to a prior ready snapshot, or delete inactive snapshots,
  • Genesis never deletes the active snapshot through the inactive-snapshot cleanup path, even if it is requested explicitly.
On Databricks Apps with durable SQLite enabled, materialized dataset snapshots survive host replacement; query-result cache datasets do not. See Durable SQLite stores on Databricks for restore timing and cache behavior. The snapshot size limit is configured with sqlite_durability.user_db_max_bytes in Advanced Settings.

​
Deletion safeguards

Genesis lets you delete user-created datasets from the Datasets page, but it adds guardrails around destructive actions:
  • built-in system datasets such as the Cost & Usage cube and Tool Observability cube cannot be deleted,
  • if a dataset is still bound to one or more apps, Genesis shows the affected apps and requires a second confirmation before the delete proceeds,
  • the confirmation flow is there to prevent breaking apps that still depend on the dataset binding.

​
Automatic retention and cleanup

Genesis also applies background retention to dataset storage. The active snapshot is never deleted by this cleanup flow. When scheduled cleanup is enabled, its first pass runs 120 seconds after server start, then runs at the configured interval. Configure jobs.dataset_cleanup.startup_delay_seconds (default 120) to change the initial delay. Each pass drains eligible datasets and snapshots in batches of up to jobs.dataset_cleanup.batch_size (default 100 of each), until no candidates remain, a batch deletes nothing, or the jobs.dataset_cleanup.pass_budget_seconds (default 600) is reached. The batch that crosses the time budget finishes; remaining candidates wait for the next pass. These settings are read at the start of each pass, except the startup delay, which is read when the scheduler registers the job. After a successful dataset refresh, Genesis also applies the snapshot policy to superseded snapshots; the active snapshot remains protected.