Document Index Tools

​

Overview

​

Tool Name

document_index_tools

​

Purpose

​

Functions Available

1. _document_index: Single multifunction entry point controlled by the action parameter that performs index and document operations.
2. CREATE_INDEX: Create a new index for future ingestion and search.
3. RENAME_INDEX: Rename an existing index without reingesting content.
4. DELETE_INDEX: Delete an index and all of its files from every embedding-model collection that holds it.
5. LIST_INDICES: List indices with basic metadata.
6. ADD_DOCUMENTS: Add one file or an entire directory to an index. Supports recursive ingest.
7. LIST_DOCUMENTS: List files inside an index with optional path and text filters. Page with start/count; responses include total_count, actual_count, has_more, and next_start.
8. GET_DOCUMENT: Read file content with pagination by lines, characters, or chunks.
9. DELETE_DOCUMENT: Remove a specific file from an index.
10. RENAME_DOCUMENT: Rename a file within an index.
11. SEARCH: Run keyword or semantic search across one or many indices.
12. ASK: Answer natural language questions using composable retrieval across indices.
13. GET_INDICES_STATS: Return per index statistics such as counts, timestamps, and chunking defaults.
14. RECHUNK: Re-chunk an existing index in place with new chunk size, overlap, or strategy.
15. propose_kb_edit: Stage a Git-backed knowledge-base text or metadata file change for operator approval. Also the recovery path when managed_repo_guard blocks git_action, file, or connection_api_action writes to a documents/KB repo.
16. open_kb_pull_request: Open or update a pull request or merge request for staged managed KB document changes.

​

Key Features

​

Input Parameters for Each Function

​

Common to _document_index

​

Chunking parameters

​

CREATE_INDEX

​

RENAME_INDEX

​

DELETE_INDEX

​

LIST_INDICES

​

ADD_DOCUMENTS

​

LIST_DOCUMENTS

​

GET_DOCUMENT

​

DELETE_DOCUMENT

​

RENAME_DOCUMENT

​

SEARCH

​

ASK

​

ADD_TEXT

​

RECHUNK

​

GET_INDICES_STATS

​

propose_kb_edit

​

open_kb_pull_request

​

Use Cases

1. Mission document collection Build an internal source collection for requirements, SOPs, and architecture notes. Example: Index a runbooks folder and enable search plus Q&A for on call engineers.
2. RAG for Onboarding Provide new team members with a question answering interface over onboarding materials. Example: Ask, “How do I rotate credentials” and get a sourced answer.
3. Git Docs Discovery Ingest a docs directory from Git and search by keyword or path. Example: Find all pages mentioning “rate limit” in API docs.
4. Analyst Document Access Search PDFs and specs, then preview with pagination to pull the right excerpt. Example: Return 100 lines starting from line 300 in a long changelog.
5. Governance Library Maintain data policy indices and generate summaries on demand. Example: Ask for retention rules and cite the policy document.
6. Git-backed knowledge base Stage markdown fixes from chat, then open a PR/MR for operator review. Example: propose_kb_edit(source_id_or_index="team-kb", relative_path="runbooks/oncall.md", search="page the SRE", replace="page the platform on-call"), then open_kb_pull_request(source_id_or_index="team-kb").
7. Tune chunking for long documents Create an index with larger chunks for narrative docs, or call RECHUNK after ingestion to change strategy without re-uploading files. Example: CREATE_INDEX with chunk_strategy="one_chunk_per_doc" for short policy PDFs, or RECHUNK with chunk_size=512 and chunk_overlap=50 for tighter retrieval on code comments.

​

Workflow/How It Works

1. Step 1: Create or Select an Index Use CREATE_INDEX or locate an existing one via LIST_INDICES.
2. Step 2: Ingest Content Use ADD_DOCUMENTS to add single files or entire directories. Prefer BOT_GIT: paths for repository sources.
3. Step 3: Explore Inventory Use LIST_DOCUMENTS with path_filter or query to scope the file set. For indexes with more than 50 files, page with start and count, following next_start until has_more is false.
4. Step 4: Retrieve Content Snippets Use GET_DOCUMENT with pagination to preview without loading entire files.
5. Step 5: Discover or Ask Use SEARCH for precise discovery with ranked results. Use ASK for natural language answers with sources.
6. Step 6: Reuse as Agent Knowledge When an agent should reuse one or more indexes across many threads, create a knowledge-base bundle with create_knowledge_base(...), then assign that knowledge_base_id with set_bot_knowledge_bases(...).
7. Step 7: Adjust Chunking (optional) Use RECHUNK to apply new chunk defaults to an existing index, or pass chunk_size / chunk_overlap / chunk_strategy on ADD_DOCUMENTS or ADD_TEXT to override per ingest.
8. Step 8: Maintain Hygiene Use RENAME_DOCUMENT or DELETE_DOCUMENT for files and RENAME_INDEX or DELETE_INDEX for indices.

​

Integration Relevance

• file_manager_tools and git_action: Register or modify files, then ingest using ADD_DOCUMENTS with BOT_GIT: paths.
• project_manager_tools: Link indices to project or mission tasks for traceability.
• web_access_tools: Scrape pages, save content, then index for unified search.
• google_drive_tools: Export summaries to Docs or Sheets. Ingest Drive downloads after staging.
• handoff: Automate nightly indexing, refreshes, or QA sweeps.

​

Configuration Details

• Directory strategy: Organize repo paths such as BOT_GIT:docs/specs and BOT_GIT:runbooks for predictable listings.
• File types: Prefer text friendly formats like Markdown and plain text for better chunking and recall.
• Chunking settings: Set per-index defaults on CREATE_INDEX with chunk_size, chunk_overlap, and chunk_strategy. Override per ingest on ADD_DOCUMENTS or ADD_TEXT, or re-apply to existing content with RECHUNK. Sentence splitting defaults are 1024 tokens with 200 overlap; use one_chunk_per_doc when each source file should map to a single searchable chunk.
• Embedding input windows: When explicit chunking is active, Genesis may warn if chunks exceed the active embedding model’s input window (for example, 512 characters for Snowflake Cortex, 8000 for LiteLLM-routed providers). Full chunk text is still stored and returned; semantic similarity uses only the leading portion within that window.
• Index healing: See Document Indexes — embedding recovery for startup recovery and temporary SEARCH warnings.
• Embedding runtime: Document ingestion and semantic retrieval use the active embeddings configuration from Config -> AI Providers (the same runtime exposed by GET and PUT /api/config/models/embeddings/config). When no complete embedding identity resolves, Genesis fails closed — CREATE_INDEX, ADD_DOCUMENTS, ADD_TEXT, SEARCH, and ASK return an error rather than silently routing through an unconfigured OpenAI default. LIST_INDICES, LIST_DOCUMENTS, and GET_INDICES_STATS may return an empty inventory with an EmbeddingIdentityState discriminator instead of raising. EMBEDDING_PROVIDER / EMBEDDING_MODEL seed that identity only while it is unset; a saved identity wins over them. When the saved provider/model pair is incoherent — for example a Cortex-only catalog model such as e5-base-v2 paired with a non-Cortex provider in a Cortex-capable deployment (Snowflake Native App / SPCS or Cortex with public REST access) — Genesis corrects the provider to cortex at config read time instead of routing through LiteLLM/OpenAI. Custom deployment names on azureopenai and litellm_proxy are never auto-corrected. Genesis also runs a one-time embedding connection smoke test at startup (5-second timeout for LiteLLM-routed providers); failures are logged but do not block boot — check server logs for failed startup smoke test when indexing fails after restart. When the active provider is LiteLLM Proxy, Genesis routes embedding calls through the configured proxy even if the selected model is shown as a bare model name. When the active provider is AWS Bedrock (litellm_bedrock) with a custom endpoint and a non-bearer custom_auth_header (for example x-api-key), Genesis applies the same custom-header proxy kwargs as chat — extra_headers plus placeholder SigV4 credentials — rather than env-based bearer auth (the chat-only Fake Stream setting, fake_stream, is not forwarded to embedding calls). Default-endpoint, bearer, and IAM Bedrock embedding calls continue to authenticate through AWS environment credentials only. When the active provider is Snowflake Cortex, Genesis uses the embedding method saved on the Cortex provider card (rest by default, or sql when explicitly selected). In Snowflake Native App / SPCS runtimes, document indexing always uses the Snowflake SQL embedding function via the deployment session. Outside SPCS, rest calls the Cortex REST API with the saved PAT or a short-lived JWT from the saved key pair; sql opens a direct Snowflake connection (account URL or Snowflake account, PAT or PEM private key, and Snowflake User required; optional warehouse and role) and runs SNOWFLAKE.CORTEX.EMBED_TEXT_768.
• Embedding collection resolution: Document embeddings live in model-specific Chroma collections named {provider}_{sanitized_model}_index. Genesis resolves the active collection for the current embedding identity by adopting an already-populated historical spelling when the derived name holds no user documents — for example after OpenAI routing moves to LiteLLM credentials while the prior bare-provider collection still holds indexed files. Background re-embed from prior *_index collections copies only when embedding identity is verified from collection metadata (or, for older collections without metadata, when the inferred provider matches and embedding dimensions match). A dimension-only match is not treated as conclusive, so Genesis keeps the source collection on disk instead of archiving it. On every server startup, Genesis also scans for user indexes that still exist only in other model-specific *_index collections for the active embedding provider and schedules a background full re-embed into the current collection (including the internal harvester metadata index). This stranded-collection sync runs independently of legacy provider-to-model migration when the active collection is empty, and it covers embedding changes applied through config files or deployment templates that do not go through the Embeddings UI save path — the path that already starts reindexing when you activate a new provider or model in Config → AI Providers. While sync runs, GET_INDICES_STATS may still list affected indexes with needs_reindex: true; poll GET /api/config/models/embeddings/reindexing-status or watch server logs for messages such as Re-embedding stranded documents after embedding model change... for progress. Opening a persisted collection may require an initial disk read; that read no longer pauses other server work, even on slow storage.
• ChromaDB compaction guardrails: ADD_DOCUMENTS pauses every 50 inserted documents by default (GENESIS_DOCUMENT_INDEX_COMPACTION_CHECKPOINT_INTERVAL, integer, default 50) for 0.5 seconds (GENESIS_DOCUMENT_INDEX_COMPACTION_CHECKPOINT_SLEEP_S, float, default 0.5) and retries transient compaction failures up to 3 times (GENESIS_DOCUMENT_INDEX_COMPACTION_MAX_RETRIES, integer, default 3).
• Access controls: Align indices with team access. Do not mix confidential and public content in the same index.

​

Limitations or Notes

1. Large files and PDFs Text extraction quality depends on parsers. Consider OCR or cleanup for scanned content.
2. Binary formats Non text binaries are not semantically indexed. Store summaries separately if needed.
3. Path stability Renaming or moving files outside the tool requires readding or a RENAME_DOCUMENT call to maintain searchability.
4. ASK scope ASK typically runs across all indices and ignores index_name. Use SEARCH to scope to a specific index.
5. Pagination fidelity Line and character counts can vary due to encoding. Rely on has_more and actual_count for navigation.
6. Delete conflicts If a delete is already in progress for an index, conflicting operations on that same index return a conflict error until cleanup completes.
7. Partial cross-collection delete When DELETE_INDEX cannot purge every physical copy (for example a prior embedding collection is unreadable), Genesis leaves surviving copies in place and returns Success: false with the collection name(s) still holding the index. Retry DELETE_INDEX after fixing storage access; the index stays listed until all copies are gone.
8. Compaction recovery If a ChromaDB compaction error persists after retries during ADD_DOCUMENTS, Genesis attempts automatic recovery. If recovery succeeds, retry the same ingest request. If recovery fails, manual cleanup or a service restart may still be required.
9. Rechunk conflicts RECHUNK holds a deletion marker for the target index for the full rebuild-and-swap. Other writes to that index fail with a conflict error until the operation finishes.
10. Embedding window truncation Chunks longer than the active embedding provider’s input limit are embedded from their leading characters only. Retrieval still returns the full stored chunk text.

​

Output

• Index Management Success flags and messages for create, rename, and delete. CREATE_INDEX may include chunk_config when chunk parameters were set. DELETE_INDEX returns Success: true with Message: "Index deleted successfully" when every physical copy is removed; when copies survive in one or more collections it returns Success: false and an Error listing those collection names with retry to finish removing it. LIST_INDICES returns the active document manager’s registered names from memory, while its IndexDetails field contains statistics. GET_INDICES_STATS returns name, file_count, created_at, needs_reindex, and chunk_config for every index in the resolved active collection plus any stranded indexes from other model-specific *_index collections that need re-embedding after an embedding-model change (excluding system metadata-backend indexes twin and harvester).
• Chunking and Rechunk RECHUNK returns docs_rechunked, chunks_before, chunks_after, chunk_config, and optional Warnings or approximate_reassembly_notes when text had to be reassembled from stored chunks. On failure, the original index is preserved and a temporary index name may be returned for inspection.
• Ingestion and Inventory For ADD_DOCUMENTS and ADD_TEXT, success messages may append chunking warnings (for example, ignored chunk_size with one_chunk_per_doc, or chunks exceeding the embedding input window). ADD_DOCUMENTS also returns a per-file status report with added and skipped items. If a ChromaDB compaction failure exhausts retries, the error message explains whether automatic recovery completed and whether you should retry the ingest or perform manual cleanup. LIST_DOCUMENTS returns TotalCount, start, count, actual_count, has_more, next_start, an optional Message, and a Documents array of paths, sizes, timestamps, and optional metadata (up to 50 items per page).
• Reading and Retrieval GET_DOCUMENT returns a content slice with start, count, actual_count, and has_more. SEARCH returns ranked results with score, snippet, index name, and path. ASK returns a best answer plus sources and footnotes.