> ## Documentation Index
> Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Document Index Tools
> Create, manage, and query full text document indices for intelligent search, question answering, and organized retrieval.
## **Overview**
### **Tool Name**
```json theme={null}
document_index_tools
```
### **Purpose**
The **document\_index\_tools** enable you to create, manage, and query document indices for high quality search, question answering, and document organization. Ideal for building reusable source collections by ingesting files from Git, local storage, or other repositories, then performing keyword and semantic retrieval at scale.
Use `document_index_tools` for custom or uploaded collections that you
manage, such as specs, runbooks, PDFs, and policies. Genesis product
documentation is searched separately with `search_genesis_docs(...)` and
`read_genesis_doc(...)` in [Context Tools](/docs/Genbot-Toolkit/Context-Tools).
When indexes are bundled into knowledge bases assigned to an agent, runtime
retrieval uses `search_knowledge_bases(...)` and `read_knowledge_base_doc(...)`
instead. Document indexes are not
the same as knowledge bases: indexes store the underlying searchable source
material, while knowledge bases are reusable bundles of one or more index
names that you assign to agents. Use
[Knowledge Base Admin Tools](/docs/Genbot-Toolkit/Knowledge-Base-Admin-Tools)
to register or delete those bundles, then use
[Bot Management Tools](/docs/Genbot-Toolkit/Bot-Management-Tools) to inspect
or replace agent assignments.
Index create, rename, and delete operations refresh capability discovery
immediately. After a mutation,
`manage_context(action="SEARCH_CONTEXT", search_scope="document_indexes")`
can discover the updated index names without restarting Genesis.
On Databricks Apps, document-index tools follow the snapshot and restore
behavior described in [Durable document indexes on Databricks](/docs/deployments/Databricks/Genesis-On-Databricks#durable-document-indexes).
#### **Functions Available**
1. **`_document_index`**: *`Single multifunction entry point controlled by the action parameter that performs index and document operations.`*
2. **`CREATE_INDEX`**: *Create a new index for future ingestion and search.*
3. **`RENAME_INDEX`**: *Rename an existing index without reingesting content.*
4. **`DELETE_INDEX`**: *Delete an index and all of its files from every embedding-model collection that holds it.*
5. **`LIST_INDICES`**: *List indices with basic metadata.*
6. **`ADD_DOCUMENTS`**: *Add one file or an entire directory to an index. Supports recursive ingest.*
7. **`LIST_DOCUMENTS`**: *List files inside an index with optional path and text filters. Page with `start`/`count`; responses include `total_count`, `actual_count`, `has_more`, and `next_start`.*
8. **`GET_DOCUMENT`**: *Read file content with pagination by lines, characters, or chunks.*
9. **`DELETE_DOCUMENT`**: *Remove a specific file from an index.*
10. **`RENAME_DOCUMENT`**: *Rename a file within an index.*
11. **`SEARCH`**: *Run keyword or semantic search across one or many indices.*
12. **`ASK`**: *Answer natural language questions using composable retrieval across indices.*
13. **`GET_INDICES_STATS`**: *Return per index statistics such as counts, timestamps, and chunking defaults.*
14. **`RECHUNK`**: *Re-chunk an existing index in place with new chunk size, overlap, or strategy.*
15. **`propose_kb_edit`**: *Stage a Git-backed knowledge-base text or metadata file change for operator approval. Also the recovery path when `managed_repo_guard` blocks `git_action`, `file`, or `connection_api_action` writes to a documents/KB repo.*
16. **`open_kb_pull_request`**: *Open or update a pull request or merge request for staged managed KB document changes.*
### Key Features
Create, rename, list, and delete indices to keep document collections organized.
Add single files or whole directories recursively, then manage documents over time.
Search across one or multiple indices and receive ranked results with context.
Ask natural language questions and retrieve sourced answers from indexed content.
Read documents with line or character pagination or navigate chunk based previews.
Delete or rename documents and indices to maintain hygiene without full reindexing.
## **Input Parameters for Each Function**
### **`Common to _document_index`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **action** | One of the supported actions. See list above. | String (enum) |
| **index\_name** | Target index. Required for most actions except global `SEARCH` or `ASK` and some listing actions. | String |
| **top\_n** | Number of results to return where applicable. Default 10 for searches. | Integer |
### **Chunking parameters**
Optional on `CREATE_INDEX`, `ADD_DOCUMENTS`, `ADD_TEXT`, and `RECHUNK`. Sizes are in **tokens** (approx. 4 characters per token).
| **Name** | **Definition** | **Format** | **Default** |
| - | - | :-: | :-: |
| **chunk\_size** | Target chunk size in tokens for `sentence` splitting. Ignored when `chunk_strategy` is `one_chunk_per_doc`. | Integer | `1024` when sentence splitting is active |
| **chunk\_overlap** | Overlap in tokens between consecutive chunks for `sentence` splitting. Must be `>= 0` and less than `chunk_size`. | Integer | `200` when sentence splitting is active |
| **chunk\_strategy** | How each source document is split before embedding: `sentence` (sentence-aware chunks) or `one_chunk_per_doc` (one chunk per source document). | String (`sentence` or `one_chunk_per_doc`) | `sentence` when any chunk parameter is provided |
`CREATE_INDEX` stores provided chunk parameters as **per-index defaults** in index
metadata. `ADD_DOCUMENTS` and `ADD_TEXT` accept per-call overrides; when omitted,
they inherit the index defaults. When neither call nor index defaults are set,
ingestion uses the prior behavior (no explicit sentence splitter). With
`one_chunk_per_doc`, `chunk_size` is ignored and a warning may be returned.
### **`CREATE_INDEX`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **index\_name** | Name of the new index to create. | String |
| **chunk\_size** | Optional per-index default chunk size in tokens. See chunking table above. | Integer |
| **chunk\_overlap** | Optional per-index default chunk overlap in tokens. | Integer |
| **chunk\_strategy** | Optional per-index default chunk strategy. | String |
### **`RENAME_INDEX`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **index\_name** | Current index name. | String |
| **new\_index\_name** | New name for the index. | String |
### **`DELETE_INDEX`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **index\_name** | Name of the index to delete. | String |
Deletion spans the **active** embedding-model collection and any **prior**
model-specific collections that still hold chunks for the index — the same
inventory `GET_INDICES_STATS` (and `GET /api/indexes`) surfaces with
`needs_reindex: true`. Genesis purges stranded copies first, then the active
copy, and detaches indexed files only after every physical copy is confirmed
gone. If a collection could not be read or purged, the action returns
`Success: false` with an `Error` naming the surviving collection name(s), for
example `Index wiki still exists in openai_text_embedding_3_large_index; retry
to finish removing it` — call `DELETE_INDEX` again to finish removal.
### **`LIST_INDICES`**
No additional parameters. Returns the registered index names from memory for the
active document manager, rather than scanning every ChromaDB chunk on each call.
Genesis seeds this name set from collection metadata during initialization and
retries a lazy seed when the initial read fails. The `IndexDetails` field still
contains statistics; use `GET_INDICES_STATS` (or `GET /api/indexes`) when you need
the broader inventory, including indexes stranded in prior model-specific
collections.
### **`ADD_DOCUMENTS`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **index\_name** | Target index. | String |
| **filepath** | Path to a file or directory. Supports `BOT_GIT:` prefix for files in the bot repository. Directories ingest recursively. | String |
| **chunk\_size** | Optional per-call chunk size override in tokens. Inherits index defaults when omitted. | Integer |
| **chunk\_overlap** | Optional per-call chunk overlap override in tokens. | Integer |
| **chunk\_strategy** | Optional per-call chunk strategy override. | String |
Use `BOT_GIT:/path/to/file_or_dir` for reliable, versioned ingestion from your mission repository. This keeps paths stable across runs.
Document ingestion applies built-in exclusions for generated and binary content,
including `**/__pycache__/**`, `**/node_modules/**`, `**/.git/**`, `**/*.o`,
`**/target/**`, `**/build/**`, and `**/dist/**`. Patterns match complete path
segments or filenames, not text within a name: `docs/05-dual-targeting.md` is
not excluded just because it contains `target`. Recursive and pattern-based
ingestion skips excluded files and reports them in its excluded count. If you
explicitly name a file covered by an exclusion pattern, `ADD_DOCUMENTS`
rejects the request instead of indexing it; move the file or use a path outside
the excluded location.
`ADD_DOCUMENTS` now tolerates transient ChromaDB WAL compaction failures more
gracefully. During larger ingests, Genesis may pause briefly at checkpoints to
let the compactor flush, then automatically retry insert operations that fail
with transient compaction errors before returning an error to the caller.
Vector inserts during ingestion are split into batches of 64 vectors. This
limits how long an individual insert can block other server work during a
large upload.
### **`LIST_DOCUMENTS`**
| **Name** | **Definition** | **Format** | **Default** |
| - | - | :-: | :-: |
| **index\_name** | Target index. | String | — |
| **path\_filter** | Optional substring to match file paths. | String | — |
| **query** | Optional text filter to match file content. | String | — |
| **show\_files\_only** | If true, list files only at the current level. | Boolean | `true` |
| **start** | Zero-based document offset for this page. | Integer | `0` |
| **count** | Requested page size. Capped at 50 documents. | Integer | `50` |
`LIST_DOCUMENTS` returns one page at a time. Read `TotalCount`, `has_more`, and
`next_start` from the response, then call again with `start=next_start` until
`has_more` is `false`. When more pages exist, `Message` summarizes the range
(for example, "Showing first 50 of 142 items. 92 more items exist."). Pagination
metadata appears before the `Documents` array so output previews retain the
signal when response size is capped.
To build a listing, Genesis first collects the index's chunk IDs, then reads
chunk records in pages of up to 2,000. This avoids materializing every chunk's
metadata and content at once; this internal read size is separate from the
50-document response page limit.
### **`GET_DOCUMENT`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **index\_name** | Target index. | String |
| **file\_id** | Identifier of the file if known. | String |
| **filepath** | Path to the file. Provide `file_id` or `filepath`. | String |
| **pagination\_mode** | One of `lines`, `chars`, or `None` for chunk based reading. | String or null |
| **start** | Zero-based starting position for the selected mode (line, character, or chunk number — not a document offset). | Integer |
| **count** | Number of lines, characters, or chunks to return (defaults: 100 lines, 10 000 chars, or 20 chunks). | Integer |
For code and structured text prefer `pagination_mode="lines"`. For prose prefer `pagination_mode="chars"`. Use chunk mode when you want the tool’s native segmentation.
### **`DELETE_DOCUMENT`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **index\_name** | Target index. | String |
| **file\_id** | Identifier of the indexed file if known. Provide `file_id` or `filepath`. | String |
| **filepath** | Path of the file to remove from the index. Provide `file_id` or `filepath`. | String |
Prefer `file_id` when the document came from Genesis file storage. If the
backing file has already been removed but its index entry remains,
`DELETE_DOCUMENT` can still remove that orphaned entry by `file_id`. When
you delete by `filepath`, Genesis also tolerates leading or trailing `/`, so
equivalent path forms still resolve to the same indexed document entry.
### **`RENAME_DOCUMENT`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **index\_name** | Target index. | String |
| **filepath** | Current path of the file. | String |
| **new\_filename** | New filename to assign. | String |
### **`SEARCH`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **index\_name** | Optional. If omitted, search across all indices. | String |
| **query** | Search text. Keyword or semantic query depending on implementation. | String |
| **top\_n** | Number of results to return. Default 10. | Integer |
### **`ASK`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **query** | Natural language question. Retrieval runs across indices using a composable graph. | String |
| **top\_n** | Optional fan out control for underlying retrieval. | Integer |
### **`ADD_TEXT`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **index\_name** | Target index. | String |
| **text** | Raw text to add directly (no file required). | String |
| **metadata** | Optional JSON metadata dict attached to the indexed text. | Object |
| **chunk\_size** | Optional per-call chunk size override in tokens. Inherits index defaults when omitted. | Integer |
| **chunk\_overlap** | Optional per-call chunk overlap override in tokens. | Integer |
| **chunk\_strategy** | Optional per-call chunk strategy override. | String |
### **`RECHUNK`**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **index\_name** | Index to re-chunk. Required. | String |
| **chunk\_size** | New chunk size in tokens. Falls back to index defaults, then `1024` for `sentence`. | Integer |
| **chunk\_overlap** | New chunk overlap in tokens. Falls back to index defaults, then `200` for `sentence`. | Integer |
| **chunk\_strategy** | New chunk strategy. Falls back to index defaults, then `sentence`. | String |
`RECHUNK` rebuilds the index in a temporary index and swaps it into place
(delete-then-rename). The index is briefly unavailable during the swap, and
concurrent writes to that index are rejected with a conflict error until
`RECHUNK` completes. Text entries without a source file path are reassembled
from stored chunks before re-chunking; that reassembly is approximate.
### **`GET_INDICES_STATS`**
No additional parameters. Returns per-index statistics: `name`, `file_count`,
`created_at`, `needs_reindex` (boolean, default `false`), and `chunk_config`
(`chunk_size`, `chunk_overlap`, `chunk_strategy` — `null` when unset). When the
active embedding identity's resolved collection is missing indexes that still
exist in another model-specific `*_index` collection on disk (any provider
prefix), those stranded indexes are included with `needs_reindex: true` and
`chunk_config` set to all `null`. Genesis normalizes embedding identity before
collection resolution — provider and model case are folded, and a genuine
`provider/` prefix on the model name is stripped when it matches the normalized
provider — so alternate spellings of the same embedding space (for example bare
`openai` + `text-embedding-3-large` versus LiteLLM-routed `litellm_openai` +
`openai/text-embedding-3-large`) can share one populated collection instead of
leaving indexes falsely stranded after a routing change. When the derived
collection name for the active identity holds no user documents but an alternate
historical spelling already does, Genesis adopts that populated collection.
`file_count` reflects user-indexed documents only: startup demo seed rows and
metadata-backend index markers are discounted. System metadata-backend indexes
`twin` and `harvester` are excluded from `needs_reindex` surfacing. This matches
the **Config → Documents** list and `GET /api/indexes`. Prefer this action over
`LIST_INDICES` when you need the full inventory after an embedding-model change.
### **`propose_kb_edit`**
| **Name** | **Definition** | **Format** | **Default** |
| - | - | :-: | :-: |
| **source\_id\_or\_index** | Managed document source id or its target document index name. | String | none |
| **relative\_path** | Repository-relative path of a supported KB text or metadata file (`.md`, `.yml`, `.yaml`, `.json`, `.txt`, `.csv`) under a first-level folder in the source's `documents_root` (for example `docs/edw/guide.md` or `docs/edw/kb-manifest.json`). | String | none |
| **content** | Full replacement file content. Leave empty when using search/replace. | String | `""` |
| **search** | Exact text to replace. Must occur exactly once. Leave empty when writing full content. | String | `""` |
| **replace** | Replacement text for search/replace edits. May be empty to delete the matched text. | String | `""` |
| **note** | Short operator-facing note explaining why this KB change is proposed. | String | `""` |
Provide either non-empty `content` or non-empty `search`, but not both. Supported
files must be under a first-level folder in `documents_root`; sidecars such as
`kb-manifest.json` and `genesis-kb.yml` use the same workflow.
Genesis owns the proposal branch worktree. Do not commit to that branch through
`git_action`, `commit_files_via_api`, or provider file APIs; the next
`open_kb_pull_request` force-pushes the staged worktree state and overwrites
commits made through those routes.
`open_kb_pull_request` sends the managed branch push through Action Review. A
pending or denied review leaves the branch and pull request unchanged; do not
retry while the proposal is waiting or when its outcome is uncertain.
When `git_action`, `file(action="write")`, or `connection_api_action` returns
`managed_repo_guard` on a Git-backed documents/KB repo, the response includes
`managed_document_repo: true` and `next_steps` that point here — stage the
supported KB file with `propose_kb_edit`, then call `open_kb_pull_request`.
### **`open_kb_pull_request`**
| **Name** | **Definition** | **Format** | **Default** |
| - | - | :-: | :-: |
| **source\_id\_or\_index** | Managed document source id or target document index name. | String | none |
| **target\_index** | Target document index to push. Optional when `source_id_or_index` is already the target index, or when the managed source has only one document index. | String | `""` |
| **title** | Optional pull request title. | String | `""` |
| **body** | Optional pull request body. | String | `""` |
| **commit\_message** | Optional commit message for staged KB document changes. | String | `""` |
The response includes `changed_files`, the authoritative list of files in the
PR. Each entry contains a repository-relative `path` and a status: `A` (added),
`M` (modified), `D` (deleted), or `R` (renamed). Use this list when describing
the PR rather than relying on earlier staging calls.
Git-backed knowledge-base documents cannot be edited in place with
`ADD_DOCUMENTS`, `ADD_TEXT`, or `DELETE_DOCUMENT`, and Genesis-owned KB
checkouts cannot be written with `git_action`, `file(action="write")`, or
`connection_api_action`. Use `propose_kb_edit` to stage supported KB files,
then `open_kb_pull_request` when the changeset is ready for review. If a
blocked mutation returns `managed_repo_guard` with `managed_document_repo:
true`, follow the tool's `next_steps` and call `propose_kb_edit` instead of
retrying file or git mutations. For ad-hoc indexes or one-off uploads,
continue using the standard `_document_index` actions.
## **Use Cases**
1. **Mission document collection**
Build an internal source collection for requirements, SOPs, and architecture notes.
*Example:* Index a runbooks folder and enable search plus Q\&A for on call engineers.
2. **RAG for Onboarding**
Provide new team members with a question answering interface over onboarding materials.
*Example:* Ask, “How do I rotate credentials” and get a sourced answer.
3. **Git Docs Discovery**
Ingest a docs directory from Git and search by keyword or path.
*Example:* Find all pages mentioning “rate limit” in API docs.
4. **Analyst Document Access**
Search PDFs and specs, then preview with pagination to pull the right excerpt.
*Example:* Return 100 lines starting from line 300 in a long changelog.
5. **Governance Library**
Maintain data policy indices and generate summaries on demand.
*Example:* Ask for retention rules and cite the policy document.
6. **Git-backed knowledge base**
Stage markdown fixes from chat, then open a PR/MR for operator review.
*Example:* `propose_kb_edit(source_id_or_index="team-kb", relative_path="runbooks/oncall.md", search="page the SRE", replace="page the platform on-call")`, then `open_kb_pull_request(source_id_or_index="team-kb")`.
7. **Tune chunking for long documents**
Create an index with larger chunks for narrative docs, or call `RECHUNK` after
ingestion to change strategy without re-uploading files.
*Example:* `CREATE_INDEX` with `chunk_strategy="one_chunk_per_doc"` for short policy PDFs, or `RECHUNK` with `chunk_size=512` and `chunk_overlap=50` for tighter retrieval on code comments.
Deleting an index or document is permanent. Confirm that no downstream workflows rely on the content before removal.
## **Workflow/How It Works**
1. **Step 1: Create or Select an Index**
Use `CREATE_INDEX` or locate an existing one via `LIST_INDICES`.
2. **Step 2: Ingest Content**
Use `ADD_DOCUMENTS` to add single files or entire directories. Prefer `BOT_GIT:` paths for repository sources.
3. **Step 3: Explore Inventory**
Use `LIST_DOCUMENTS` with `path_filter` or `query` to scope the file set. For
indexes with more than 50 files, page with `start` and `count`, following
`next_start` until `has_more` is `false`.
4. **Step 4: Retrieve Content Snippets**
Use `GET_DOCUMENT` with pagination to preview without loading entire files.
5. **Step 5: Discover or Ask**
Use `SEARCH` for precise discovery with ranked results. Use `ASK` for natural language answers with sources.
6. **Step 6: Reuse as Agent Knowledge**
When an agent should reuse one or more indexes across many threads, create a
knowledge-base bundle with `create_knowledge_base(...)`, then assign that
`knowledge_base_id` with `set_bot_knowledge_bases(...)`.
7. **Step 7: Adjust Chunking (optional)**
Use `RECHUNK` to apply new chunk defaults to an existing index, or pass
`chunk_size` / `chunk_overlap` / `chunk_strategy` on `ADD_DOCUMENTS` or
`ADD_TEXT` to override per ingest.
8. **Step 8: Maintain Hygiene**
Use `RENAME_DOCUMENT` or `DELETE_DOCUMENT` for files and `RENAME_INDEX` or `DELETE_INDEX` for indices.
## **Integration Relevance**
* **file\_manager\_tools and git\_action**: Register or modify files, then ingest using `ADD_DOCUMENTS` with `BOT_GIT:` paths.
* **project\_manager\_tools**: Link indices to project or mission tasks for traceability.
* **web\_access\_tools**: Scrape pages, save content, then index for unified search.
* **google\_drive\_tools**: Export summaries to Docs or Sheets. Ingest Drive downloads after staging.
* **handoff**: Automate nightly indexing, refreshes, or QA sweeps.
## **Configuration Details**
* **Directory strategy**: Organize repo paths such as `BOT_GIT:docs/specs` and `BOT_GIT:runbooks` for predictable listings.
* **File types**: Prefer text friendly formats like Markdown and plain text for better chunking and recall.
* **Chunking settings**: Set per-index defaults on `CREATE_INDEX` with `chunk_size`, `chunk_overlap`, and `chunk_strategy`. Override per ingest on `ADD_DOCUMENTS` or `ADD_TEXT`, or re-apply to existing content with `RECHUNK`. Sentence splitting defaults are `1024` tokens with `200` overlap; use `one_chunk_per_doc` when each source file should map to a single searchable chunk.
* **Embedding input windows**: When explicit chunking is active, Genesis may warn if chunks exceed the active embedding model's input window (for example, `512` characters for Snowflake Cortex, `8000` for LiteLLM-routed providers). Full chunk text is still stored and returned; semantic similarity uses only the leading portion within that window.
* **Index healing**: See [Document Indexes — embedding recovery](/docs/User-Guide/Document-Indexes#embedding-provider-required) for startup recovery and temporary `SEARCH` warnings.
* **Embedding runtime**: Document ingestion and semantic retrieval use the active embeddings configuration from **Config -> AI Providers** (the same runtime exposed by `GET` and `PUT /api/config/models/embeddings/config`). When no complete embedding identity resolves, Genesis fails closed — `CREATE_INDEX`, `ADD_DOCUMENTS`, `ADD_TEXT`, `SEARCH`, and `ASK` return an error rather than silently routing through an unconfigured OpenAI default. `LIST_INDICES`, `LIST_DOCUMENTS`, and `GET_INDICES_STATS` may return an empty inventory with an `EmbeddingIdentityState` discriminator instead of raising. `EMBEDDING_PROVIDER` / `EMBEDDING_MODEL` seed that identity only while it is unset; a saved identity wins over them. When the saved provider/model pair is incoherent — for example a Cortex-only catalog model such as `e5-base-v2` paired with a non-Cortex provider in a Cortex-capable deployment (Snowflake Native App / SPCS or Cortex with public REST access) — Genesis corrects the provider to `cortex` at config read time instead of routing through LiteLLM/OpenAI. Custom deployment names on `azureopenai` and `litellm_proxy` are never auto-corrected. Genesis also runs a one-time embedding connection smoke test at startup (5-second timeout for LiteLLM-routed providers); failures are logged but do not block boot — check server logs for `failed startup smoke test` when indexing fails after restart. When the active provider is LiteLLM Proxy, Genesis routes embedding calls through the configured proxy even if the selected model is shown as a bare model name. When the active provider is AWS Bedrock (`litellm_bedrock`) with a custom endpoint and a non-bearer `custom_auth_header` (for example `x-api-key`), Genesis applies the same custom-header proxy kwargs as chat — `extra_headers` plus placeholder SigV4 credentials — rather than env-based bearer auth (the chat-only **Fake Stream** setting, `fake_stream`, is not forwarded to embedding calls). Default-endpoint, bearer, and IAM Bedrock embedding calls continue to authenticate through AWS environment credentials only. When the active provider is `Snowflake Cortex`, Genesis uses the embedding method saved on the Cortex provider card (`rest` by default, or `sql` when explicitly selected). In Snowflake Native App / SPCS runtimes, document indexing always uses the Snowflake SQL embedding function via the deployment session. Outside SPCS, `rest` calls the Cortex REST API with the saved PAT or a short-lived JWT from the saved key pair; `sql` opens a direct Snowflake connection (account URL or Snowflake account, PAT or PEM private key, and **Snowflake User** required; optional warehouse and role) and runs `SNOWFLAKE.CORTEX.EMBED_TEXT_768`.
* **Embedding collection resolution**: Document embeddings live in model-specific Chroma collections named `{provider}_{sanitized_model}_index`. Genesis resolves the active collection for the current embedding identity by adopting an already-populated historical spelling when the derived name holds no user documents — for example after OpenAI routing moves to LiteLLM credentials while the prior bare-provider collection still holds indexed files. Background re-embed from prior `*_index` collections copies only when embedding identity is verified from collection metadata (or, for older collections without metadata, when the inferred provider matches and embedding dimensions match). A dimension-only match is not treated as conclusive, so Genesis keeps the source collection on disk instead of archiving it. On every server startup, Genesis also scans for user indexes that still exist only in other model-specific `*_index` collections for the active embedding provider and schedules a background **full** re-embed into the current collection (including the internal harvester metadata index). This stranded-collection sync runs independently of legacy provider-to-model migration when the active collection is empty, and it covers embedding changes applied through config files or deployment templates that do not go through the Embeddings UI save path — the path that already starts reindexing when you activate a new provider or model in **Config → AI Providers**. While sync runs, `GET_INDICES_STATS` may still list affected indexes with `needs_reindex: true`; poll `GET /api/config/models/embeddings/reindexing-status` or watch server logs for messages such as `Re-embedding stranded documents after embedding model change...` for progress. Opening a persisted collection may require an initial disk read; that read no longer pauses other server work, even on slow storage.
* **ChromaDB compaction guardrails**: `ADD_DOCUMENTS` pauses every `50` inserted documents by default (`GENESIS_DOCUMENT_INDEX_COMPACTION_CHECKPOINT_INTERVAL`, integer, default `50`) for `0.5` seconds (`GENESIS_DOCUMENT_INDEX_COMPACTION_CHECKPOINT_SLEEP_S`, float, default `0.5`) and retries transient compaction failures up to `3` times (`GENESIS_DOCUMENT_INDEX_COMPACTION_MAX_RETRIES`, integer, default `3`).
* **Access controls**: Align indices with team access. Do not mix confidential and public content in the same index.
Use `LIST_INDICES` and `GET_INDICES_STATS` together to monitor growth and spot unusually large or stale indices that may need cleanup.
## **Limitations or Notes**
1. **Large files and PDFs**
Text extraction quality depends on parsers. Consider OCR or cleanup for scanned content.
2. **Binary formats**
Non text binaries are not semantically indexed. Store summaries separately if needed.
3. **Path stability**
Renaming or moving files outside the tool requires readding or a `RENAME_DOCUMENT` call to maintain searchability.
4. **ASK scope**
`ASK` typically runs across all indices and ignores `index_name`. Use `SEARCH` to scope to a specific index.
5. **Pagination fidelity**
Line and character counts can vary due to encoding. Rely on `has_more` and `actual_count` for navigation.
6. **Delete conflicts**
If a delete is already in progress for an index, conflicting operations on that same index return a conflict error until cleanup completes.
7. **Partial cross-collection delete**
When `DELETE_INDEX` cannot purge every physical copy (for example a prior
embedding collection is unreadable), Genesis leaves surviving copies in place
and returns `Success: false` with the collection name(s) still holding the
index. Retry `DELETE_INDEX` after fixing storage access; the index stays
listed until all copies are gone.
8. **Compaction recovery**
If a ChromaDB compaction error persists after retries during `ADD_DOCUMENTS`, Genesis attempts automatic recovery. If recovery succeeds, retry the same ingest request. If recovery fails, manual cleanup or a service restart may still be required.
9. **Rechunk conflicts**
`RECHUNK` holds a deletion marker for the target index for the full rebuild-and-swap. Other writes to that index fail with a conflict error until the operation finishes.
10. **Embedding window truncation**
Chunks longer than the active embedding provider's input limit are embedded from their leading characters only. Retrieval still returns the full stored chunk text.
## **Output**
* **Index Management**
Success flags and messages for create, rename, and delete. `CREATE_INDEX` may include `chunk_config` when chunk parameters were set. `DELETE_INDEX` returns `Success: true` with `Message: "Index deleted successfully"` when every physical copy is removed; when copies survive in one or more collections it returns `Success: false` and an `Error` listing those collection names with `retry to finish removing it`. `LIST_INDICES` returns the active document manager's registered names from memory, while its `IndexDetails` field contains statistics. `GET_INDICES_STATS` returns `name`, `file_count`, `created_at`, `needs_reindex`, and `chunk_config` for every index in the resolved active collection plus any stranded indexes from other model-specific `*_index` collections that need re-embedding after an embedding-model change (excluding system metadata-backend indexes `twin` and `harvester`).
* **Chunking and Rechunk**
`RECHUNK` returns `docs_rechunked`, `chunks_before`, `chunks_after`, `chunk_config`, and optional `Warnings` or `approximate_reassembly_notes` when text had to be reassembled from stored chunks. On failure, the original index is preserved and a temporary index name may be returned for inspection.
* **Ingestion and Inventory**
For `ADD_DOCUMENTS` and `ADD_TEXT`, success messages may append chunking warnings (for example, ignored `chunk_size` with `one_chunk_per_doc`, or chunks exceeding the embedding input window). `ADD_DOCUMENTS` also returns a per-file status report with added and skipped items. If a ChromaDB compaction failure exhausts retries, the error message explains whether automatic recovery completed and whether you should retry the ingest or perform manual cleanup. `LIST_DOCUMENTS` returns `TotalCount`, `start`, `count`, `actual_count`, `has_more`, `next_start`, an optional `Message`, and a `Documents` array of paths, sizes, timestamps, and optional metadata (up to 50 items per page).
* **Reading and Retrieval**
`GET_DOCUMENT` returns a content slice with `start`, `count`, `actual_count`, and `has_more`.
`SEARCH` returns ranked results with score, snippet, index name, and path.
`ASK` returns a best answer plus sources and footnotes.
Prefer `SEARCH` when you need multiple hits with scores. Prefer `ASK` when you want a concise answer with citations.