> ## Documentation Index
> Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Harvester Tools

> Configure and control automated metadata harvesting for your databases. Scope by schema, schedule refreshes, monitor status, and clean up when needed.

## **Overview**

### **Tool Name**

```json theme={null}
harvester_tools
```

### **Purpose**

The **harvester\_tools** group manages automated collection of database metadata used by search and discovery. Configure what to harvest, how often to refresh, and where to focus. Track status across sources and remove configurations or metadata when they are no longer required.

### **Key Features & Functions**

<Steps>
  <Step title="Configure Sources" icon="database">
    Register which connections and databases should be harvested and at what cadence.
  </Step>

  <Step title="Schema Scoping" icon="filter">
    Include or exclude schemas to focus on relevant areas and reduce noise.
  </Step>

  <Step title="Scheduling" icon="clock">
    Set refresh intervals in minutes and trigger first crawls on demand.
  </Step>

  <Step title="Monitor Coverage" icon="list-check">
    Review control rows and high-level summaries to see progress and health.
  </Step>

  <Step title="Cleanup" icon="trash">
    Remove control rows or purge stored metadata for decommissioned databases.
  </Step>
</Steps>

## **Input Parameters for Each Function**

### **`_get_harvest_control_data`**

**Parameters**

*(No parameters. Returns all current harvest control rows.)*

### **`_set_harvest_control_data`**

**Parameters**

| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **connection\_id** | Connection id for the source to harvest. | String |
| **database\_name** | Database name to harvest. For BigQuery, use the project id. | String |
| **refresh\_interval** | Refresh cadence in minutes. Default 5. | Integer |
| **initial\_crawl\_complete** | Set to `false` to trigger an immediate initial crawl. | Boolean |
| **status** | Control row status. Use `Include` to enable or `Exclude` to disable. | String |
| **schema\_inclusions** | List of schemas to include. Empty list means include all. | Array |
| **schema\_exclusions** | Schemas to exclude from harvest. | Array |

<Tip>
  To force a fresh crawl after structural changes, call `_set_harvest_control_data` with `initial_crawl_complete: false`.
</Tip>

### **`_remove_harvest_control_data`**

**Parameters**

| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **source\_name** | Source or connection identifier of the control row to remove. | String |
| **database\_name** | Target database for the control row removal. | String |

### **`_remove_metadata_for_database`**

**Parameters**

| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **source\_name** | Source or connection identifier for metadata purge. | String |
| **database\_name** | Database whose harvested metadata will be deleted. | String |

### **`_get_harvest_summary`**

**Parameters**

*(No parameters. Returns coverage, last run times, counts, and errors.)*

## **Use Cases**

1. **Stand up harvesting for a new source**
   * Add a control row for a new Snowflake database with a 5-minute interval.
2. **Tightly scope collections**
   * Include only `ANALYTICS` and `MARTS` while excluding `INFORMATION_SCHEMA`.
3. **Monitor rollout**
   * Track initial crawl completion and spot errors across environments.
4. **Tune cadences**
   * Increase intervals on slow changing warehouses to reduce compute load.
5. **Decommission cleanly**
   * Remove control rows and purge stored metadata for retired systems.

<Warning>
  Purging with `_remove_metadata_for_database` deletes stored metadata for the database. Make sure you truly do not need it before proceeding.
</Warning>

## **Workflow/How It Works**

1. **Inspect current state**
   * Call `_get_harvest_control_data` and `_get_harvest_summary` to see what is active.
2. **Plan scope**
   * Decide on inclusions or exclusions. Default behavior is to harvest all schemas except typical system schemas.
3. **Configure**
   * Use `_set_harvest_control_data` to upsert the row with status, interval, and filters. Set `initial_crawl_complete: false` to kick off the first crawl.
4. **Monitor**
   * Watch `_get_harvest_summary` during rollout. Adjust interval or scope if load is higher than expected.
5. **Cleanup**
   * Use `_remove_harvest_control_data` to stop future crawls. If needed, `_remove_metadata_for_database` to clear stored metadata.

<Note>
  Schema filters are not applicable for some engines like MySQL and SQLite where scoping is at the database level.
</Note>

## **Integration Relevance**

* **data\_connector\_tools** for connection ids and downstream metadata search.
* **genesis\_job\_tools** to observe the background harvester jobs and follow logs.
* **project\_manager\_tools** to track catalog rollout as tasks and milestones.
* **document\_index\_tools** in parallel when you also index full text documents.
* **system\_stats\_tools** for monitoring resource impact during large crawls.

## **Configuration Details**

* Use exact case for database and schema names to match engine reporting.
* Start with `refresh_interval` of 5 minutes for active environments and relax to 15 to 60 minutes where change is rare.
* Exclude noisy system schemas like `INFORMATION_SCHEMA` unless needed.
* Ensure the service role has privileges to run `SHOW` and `DESCRIBE` style operations.
* One control row per source and database pair keeps intent explicit.

## **Limitations or Notes**

1. Harvesting relies on valid connectivity and credentials.
2. Large catalogs may take time to crawl, and summaries can lag until complete.
3. Aggressive refresh cadences can increase compute costs on warehouses.
4. Schema scoping does not apply to all engines.
5. Removing a control row stops future crawls but does not delete stored metadata.
6. Network failures and permission changes can interrupt or reduce coverage.
7. Concurrent large harvests may be throttled by available resources.

## **Output**

* **Harvest Control Data**
  * Current rows with connection, database, status, interval, and filters.
* **Configuration Updates**
  * Upsert confirmation and whether an initial crawl was triggered.
* **Harvest Summary**
  * Coverage counts, last run timestamps, error snapshots, and object totals.
* **Removal Confirmations**
  * Success messages for control row removal and for metadata purge actions.
* **Status Information**
  * Progress indicators and completion timestamps per source and database.