> ## Documentation Index
> Fetch the complete documentation index at: https://docs.genesiscomputing.com/llms.txt
> Use this file to discover all available pages before exploring further.
# Web Access Tools
> Search the web and extract structured content to power research, monitoring, and real-time insights in your workflows.
## **Overview**
### **Tool Name**
```json theme={null}
web_access_tools
```
### **Purpose**
The **web\_access\_tools** group lets you discover and retrieve web content on demand. Run targeted Google searches, scrape specific pages to extract titles, metadata, text, links, tables, and more, and call authenticated APIs or downloads when your workflow needs protected content. Use generic `auth` only for arbitrary APIs. For connection-managed provider APIs such as GitLab `/api/v4/...`, Jira `/rest/api/...` and `/rest/servicedeskapi/...`, or Azure Data Factory management APIs, use `connection_api_action` or provider-specific tooling instead of passing PATs, bearer headers, or service-principal credentials through generic web auth.
Use [Browser Tools](/docs/Genbot-Toolkit/Browser-Tools) when a workflow must interact with a live UI by navigating, clicking, typing, or inspecting current page state. Use **web\_access\_tools** for page reads, REST APIs, and file downloads.
For `get_web_content`, browser-backed actions (`web_page`, `batch_pages`,
`orchestrate`, `start_session`, and `session_actions`) return an unavailable
error when Chromium is not included in a Databricks App deployment. Use
`static_html` for pages that do not need JavaScript, or `api_data` for an API;
these actions do not start a browser.
### **Key Features & Functions**
Query the open web with two providers: **serper** (Google SERP via Serper) for navigational and multi-type searches, or **tavily** (LLM/agent-optimized search) for clean snippets and optional inline page-text extraction.
Fetch a URL and extract structured content such as headings, paragraphs, lists, links, and images.
Parse tables and other page elements for downstream analytics or indexing.
Prompt for credentials securely for arbitrary APIs, call protected endpoints, reuse saved Connections for connection-managed provider APIs, and download files with structured auth settings and clearer diagnostics.
Receive ranked results with snippets and metadata to keep only what matters.
Feed outputs into indexing, storage, and collaboration tools for end-to-end research pipelines.
Start broad, then refine. Use a general search to map the space, follow up with focused queries and selective scraping of the best sources.
## **Input Parameters for Each Function**
### **`_scrape_url`**
On Genesis AI (Genesis Desktop or a hosted Try Me instance using its owner's credit),
`_scrape_url` can fetch a page without a Serper key. Genesis fetches the public page
directly at no charge; it returns static content and does not render JavaScript. A
configured Serper key continues to take precedence.
**Parameters**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **url** | Absolute URL of the webpage to scrape. | String |
### **`search_web`**
Use `search_web` for open-web research. Genesis supports two providers:
* **serper** — Google SERP via Serper ([serper.dev](https://serper.dev)). Best for navigational queries and the full range of `search_type` values.
* **tavily** — LLM/agent-optimized search ([tavily.com](https://tavily.com)). Returns pre-processed snippets and can return extracted page text inline (`include_raw_content`), collapsing search → fetch → parse into one call.
On Genesis AI (Genesis Desktop or a hosted Try Me instance using its owner's credit), `search_web` also works without a search key when the Genesis-issued LiteLLM proxy key is active. With neither a personal Serper nor Tavily key configured, `provider='auto'` (or `'serper'`) uses Genesis AI's built-in Google search, billed to the applicable Genesis AI credit. A configured key for the selected provider takes precedence; `provider='tavily'` requires a Tavily key. Genesis AI search supports general web search only: other `search_type` values fall back to general search, and Tavily-only parameters are ignored; both cases are reported in `meta.notices[]`. Genesis AI searches cap `max_results` at 20.
**Setup actions**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **action** | `search` (default), `set_serper_key`, `set_tavily_key`, or `RTFM` for the full manual. | String |
| **api\_key** | API key when `action='set_serper_key'` or `action='set_tavily_key'`. | String |
Keys can also be configured as Connections (`google_search` and `tavily` types in **Config → Connections**), or via the `SERPER_API_KEY` / `TAVILY_API_KEY` environment variables.
When you paste a key through `set_serper_key` or `set_tavily_key`, Genesis stores it as your personal connection and sets it as your user default. For org-wide setup, create a shared connection in **Config → Connections** and assign it as a role default.
**Search parameters**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **query** | Search query string (required for `action='search'`). | String |
| **provider** | `auto` \| `serper` \| `tavily` (default `auto`). `auto` uses the operator-configured default (`tools.web_access.default_search_provider`, default `serper`), then falls back to a provider with a key. On Genesis AI (Genesis Desktop or hosted Try Me using its owner's credit), it falls back to Genesis AI search if neither personal key is configured. Adding a Tavily key alone does not change `auto` — set the default to `tavily` to opt in. | String |
| **search\_type** | `search` (default) or `news` (Serper and Tavily). Serper also supports `images`, `videos`, `places`, `maps`, `shopping`, `scholar`, `patents`. Genesis AI search supports general search only; other values fall back to it with a notice. | String |
| **max\_results** | Maximum number of results (default `10`). | Integer |
| **engine** | `auto` \| `google` (default `auto`). Serper only; ignored for Tavily (a `meta.notices[]` entry is returned). | String |
**Tavily-only parameters** (ignored for serper, with a `meta.notices[]` entry)
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **search\_depth** | `basic` \| `advanced`. `advanced` extracts page content at the search layer. | String |
| **include\_raw\_content** | When `true`, returns extracted page text per result as `raw_content`. | Boolean |
| **include\_domains** | Restrict results to these domains. | `list[string]` |
| **exclude\_domains** | Exclude results from these domains. | `list[string]` |
**Example**
```python theme={null}
# Default-provider web search
search_web(query="Python tutorials")
# Force Tavily with inline page-content extraction
search_web(
query="Maryland MSRPS investment consultant 2024 ACFR",
provider="tavily",
search_depth="advanced",
include_raw_content=True,
include_domains=["sra.maryland.gov"],
)
# Force Serper image search
search_web(query="sunset", provider="serper", search_type="images", max_results=20)
# On Genesis AI, search without a Serper or Tavily key
search_web(query="latest Genesis AI updates")
```
**Response shape**
Returns `{ success, data?: { engine, provider, query, organic: [{ title, link, snippet, raw_content? }], meta: { notices[] } }, raw?, error? }`. `provider` can be `serper`, `tavily`, or `genesis_ai`; Genesis AI results use `engine: "google"`. `organic[].raw_content` is present only for Tavily results that returned extracted page text.
See [Cost accounting](/docs/Setup/Advanced-Settings#cost-accounting) for web-request metering.
### **`get_web_content` with `action="ask_user_for_secret"`**
**Parameters**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **action** | Must be `ask_user_for_secret`. | String |
| **secret\_hint** | Required secret prompt hint. Preferred format: `name;status;description`. A bare label such as `secret_linear_api_key` or `linear_api_key` is also accepted; Genesis adds the `secret_` prefix when needed, defaults `status` to `not_tried`, and auto-generates the prompt text. Names are normalized to the secret validator's `secret_` plus letters, numbers, and underscores format; URL schemes and other invalid characters become underscores, and the result is limited to 200 characters. | String |
`status="not_tried"` returns an existing secret silently when one is already stored. `status="failed"` always opens an in-chat update card so the user can replace a credential that did not work. When Genesis does need input, it anchors the credential card to the related assistant reply instead of opening a blocking modal. This action returns only the `secret_name`; the credential value stays in the Genesis secret vault. If the user does not respond within `600` seconds, the request times out.
If an arbitrary API call needs a credential, call `get_web_content(action="ask_user_for_secret")` immediately. Do not ask users to paste API keys, passwords, or tokens into chat first. For configured Connections such as GitLab `/api/v4/...` or Jira `/rest/api/...`, do not ask for PATs or bearer tokens here. Use `connection_api_action` or provider-specific GitLab or Jira tooling, and ask the user to configure or reauthorize the Connection if needed.
**Example**
```json theme={null}
{
"action": "ask_user_for_secret",
"secret_hint": "secret_linear_api_key;not_tried;Please provide your Linear API key"
}
```
### **`get_web_content` with `action="api_data"`**
**Auth-related parameters**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **method** | HTTP method for API requests and downloads. Default: `GET`. | String |
| **auth** | Authentication configuration object for arbitrary APIs only. Set `type` to one of the supported literal values: `basic`, `bearer`, `api_key`, `aws_v4`, `oauth1`, `client_cert`, `digest`, `ntlm`, or `custom`; do not use a secret reference for this field. After `ask_user_for_secret` returns a `secret_name`, pass that stored name in the relevant `auth` field on the next `api_data` call so the credential stays out of thread history. For GitLab `/api/v4/...` or Jira `/rest/api/...`, use `connection_api_action` instead of generic `auth`. | Object |
| **json\_body** | Serialize `body` as JSON. Default: `true`. | Boolean |
| **resolve\_body\_secrets** | Resolve `secret_*` references inside structured request bodies before sending the request. Default: `false`. | Boolean |
| **timeout** | Request timeout in seconds. Default: `30`. | Integer |
| **max\_size\_mb** | Maximum saved-response size in MB. Default: `100`. | Number |
On Databricks Apps, `api_data` and `api_data_batch` downloads and `static_html`
files saved with `save_to_file` follow the [durable chat uploads behavior](/docs/deployments/Databricks/Genesis-On-Databricks#durable-chat-uploads).
#### **`auth.type="api_key"` header and query options**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **key** | Stored `secret_*` name returned by `ask_user_for_secret`. Raw API key values are still accepted for compatibility, but prefer the stored secret flow so credentials never appear in chat history. | String |
| **location** | Where to place the key. Use `header` or `query`. Default: `header`. | String |
| **header\_name** | Header name when `location="header"`. Default: `X-API-Key`. | String |
| **param\_name** | Query parameter name when `location="query"`. Default: `api_key`. | String |
| **prefix** | Optional header prefix when `location="header"`. Default: empty string. Example: `prefix="Bearer"` produces `Authorization: Bearer `. | String |
If an API expects the raw token directly in the `Authorization` header, use `auth.type="api_key"` with `location="header"` and `header_name="Authorization"` and leave `prefix` empty. Use `prefix="Bearer"` only when the endpoint explicitly requires `Authorization: Bearer <token>`.
The usual sequence is: first call `ask_user_for_secret`, then pass the returned `secret_name` in the relevant `auth` field on the next `api_data` request.
GitLab REST calls under `/api/v4` still require connection-backed tooling and are blocked before secrets or auth are resolved; GitLab GraphQL queries can be read here. For GitHub, Jira, Salesforce, HubSpot, Linear, ServiceNow, and Databricks, reads remain available, but a request that changes provider data is refused before it is sent with `error_code="RAW_WRITE_REQUIRES_GOVERNED_TOOL"` and a `governed_tool` field naming the replacement. GraphQL queries and Databricks SQL reads sent as `POST` are treated as reads; a body resolved from secrets is treated as a write.
The governed tool does not always submit changes for Action Review. GitLab, GitHub, Jira, Linear, ServiceNow, and Databricks writes use their named governed tools; Salesforce merges and Lead, Contact, or Account deletes, and HubSpot merges, go through Action Review. Salesforce creates, updates, upserts, and other deletes, plus HubSpot creates, updates, and upserts, use the connector under the saved Connection's permissions. HubSpot deletes and archives, and Salesforce anonymous Apex, are not available through Genesis.
The web tool does not follow redirects for non-`GET` requests. A `303` response is reported as a successful delivered write; other write redirects return `WRITE_REDIRECT_NOT_FOLLOWED` with `write_delivered`, `redirect_followed`, and `redirect_location`. The request was already delivered to the original server, so check whether it applied before retrying.
When a blocked provider URL is supplied, the response also includes a `connection_finder` result when discovery can run. That result contains ranked visible connections. The server returns `auto_select` only for one warning-free exact match and `user_must_pick` otherwise, so an agent can use an existing connection instead of falling back to generic web search or asking for provider credentials.
**Example**
```json theme={null}
{
"action": "api_data",
"url": "https://api.linear.app/graphql",
"auth": {
"type": "api_key",
"key": "secret_linear_api_key",
"location": "header",
"header_name": "Authorization"
}
}
```
### **`connection_api_action`**
Use `connection_api_action` for long-tail provider REST API calls that must run through a saved Connection. It supports `gitlab`, `jira`, and `azure_data_factory`. For GitLab and Jira it only reads (`GET`): a `POST`, `PUT`, `PATCH`, or `DELETE` is refused with `error_code` `RAW_WRITE_REQUIRES_GOVERNED_TOOL` and nothing is sent, because GitLab changes go through `git_action` and Jira changes through `jira_connector`, which run them under Action Review. For Azure Data Factory, use it for management-plane operations such as listing or updating pipelines, enumerating triggers, datasets, or linked services, and querying pipeline runs.
**Parameters**
| **Name** | **Definition** | **Format** |
| - | - | :-: |
| **connection\_type** | Provider connection type: `gitlab`, `jira`, or `azure_data_factory`. Use the provider's configured Connection instead of supplying credentials to generic web tooling. | String (required) |
| **connection\_name** | Connection name such as `work`, `gitlab.work`, `ayou`, or `jira.ayou`. If omitted, Genesis uses the requester-aware provider default. Do not pass credentials or auth headers. | String |
| **method** | HTTP method for the provider API call. GitLab and Jira accept `GET` only. Azure Data Factory accepts `GET`, `POST`, `PUT`, `PATCH`, `DELETE`. Default: `GET`. | String |
| **path** | Provider-relative API path. GitLab paths must be under `/api/v4`; Jira paths must be under `/rest/api/2`, `/rest/api/3`, or `/rest/servicedeskapi`. Azure Data Factory accepts a factory-relative path such as `/pipelines` or `/pipelines/LoadUsers`, resolved from the Connection's subscription, resource group, and factory, or a full `/subscriptions//resourceGroups//providers/Microsoft.DataFactory/factories//...` path. Absolute URLs, query strings, fragments, and path traversal are rejected; use `query_params` for query strings. | String (required) |
| **query\_params** | Optional query parameters to send as a JSON object. | Object |
| **body** | Optional request body for an Azure Data Factory `POST`, `PUT`, or `PATCH`. Dict/list bodies are sent as JSON. | Object |
| **save\_to\_file** | Optional file path for saving the response body. Relative paths are saved under `bot_git/repos/uploads/<thread_id>/`; in a Databricks App durable-upload conversation, the actual saved path can use a fresh name if the requested name is already in use. When an eligible file cannot be stored in the volume or attached to its conversation, the response includes `durability_warning`. | String |
| **return\_content** | Whether to include parsed or preview response content in the tool result. Default: `true`. | Boolean |
| **confirmed** | Required as `true` for `DELETE`. Before retrying a destructive delete, present the exact provider path, scope, and expected impact to the user. | Boolean |
| **timeout** | Request timeout in seconds. Default: `30`. | Integer |
`connection_api_action` resolves credentials from the saved Connection and rejects caller-supplied PATs, bearer tokens, auth configs, or provider auth headers. For Azure Data Factory, the saved service principal is exchanged for an Azure Resource Manager bearer token and every request is scoped to the Connection's subscription; `api-version` defaults to `2018-06-01` when omitted. If the Connection is missing, ask the user to configure it. If the Connection needs OAuth renewal, Genesis starts the reauthorization flow.
Successful responses include `success`, `provider` (`gitlab`, `jira`, or `azure_data_factory`), `connection_name`, `method`, `path`, and `status_code`, plus optional `content`. When `save_to_file` is used, Genesis also returns `saved`, `file_path`, `path`, and `timestamp`; the saved path can differ from the requested path. See [durable chat uploads](/docs/deployments/Databricks/Genesis-On-Databricks#durable-chat-uploads) for Databricks behavior and `durability_warning`. Azure Data Factory `POST`, `PUT`, and `PATCH` calls also include `mutation_guidance`.
When `return_content` is `true`, JSON bodies are parsed and returned in full — including GitLab repository *files* metadata responses that carry an entire file as base64 inside JSON. Non-JSON bodies (for example GitLab `/raw` file downloads) are returned in full when they are `4000` characters or fewer. Larger non-JSON bodies are never silently truncated: `content` is only the first `4000` characters (a preview), and Genesis also returns `content_truncated` (`true`), `content_total_chars`, `content_preview_chars` (`4000`), and a `note` explaining that the preview must not be edited or committed. Unless you passed `save_to_file`, Genesis auto-saves the complete body under `bot_git/repos/uploads/<thread_id>/connection_api/<timestamp>_<filename>` and returns `saved`, `file_path`, `path`, `uploads_relative_path`, and `timestamp`; on save failure it returns `full_content_save_error` instead. When `uploads_relative_path` is present, read the full plaintext file back with the file tool — no decoding — using `file(action="read", repo_id="uploads", file_path="<uploads_relative_path>")` and edit or commit only from that full content. The `note` field includes this recovery hint; do not reconstruct the file from the inline preview or decode base64 from a GitLab files metadata response.
**Examples**
```python theme={null}
# GitLab
result = connection_api_action(
connection_type="gitlab",
connection_name="work",
method="GET",
path="/api/v4/user"
)
# Jira Service Management
result = connection_api_action(
connection_type="jira",
connection_name="jira.ayou",
method="GET",
path="/rest/servicedeskapi/request/CUSTOMER-1/comment"
)
# Azure Data Factory
result = connection_api_action(
connection_type="azure_data_factory",
connection_name="azure_data_factory.production",
method="GET",
path="/pipelines"
)
```
## **Use Cases**
1. **Market and industry research**
Identify trends, competitors, and benchmarks from trusted sources.
2. **Real-time news monitoring**
Track breaking updates that affect active missions and roadmaps.
3. **Technical discovery**
Locate docs, guides, and best practices for engineering solutions.
4. **Scholarly evidence gathering**
Pull academic articles or patents to support recommendations.
5. **Vendor and product validation**
Collect specifications and pricing from official sites.
6. **Regulatory checks**
Retrieve policies and compliance notes from government domains.
7. **Visual asset sourcing**
Find images or videos for briefings and presentations.
8. **Connection-backed provider API calls**
Call long-tail GitLab or Jira/JSM REST endpoints through a saved Connection without exposing PATs or bearer tokens in chat history.
## **Workflow/How It Works**
1. **Search** with `search_web` using carefully scoped keywords, a `provider` when you need to force Serper or Tavily, and a `search_type` when needed.
2. **Select sources** by reviewing titles, snippets, and domains for credibility.
3. **Scrape** chosen pages with `_scrape_url` to extract structured content.
4. **Store or index** outputs for retrieval and cross-referencing in later steps.
5. **Cite and share** the most relevant excerpts in reports, docs, or Slack updates.
Respect robots.txt, site terms, and rate limits. Avoid scraping authenticated or prohibited areas and throttle requests to prevent blocking.
## **Integration Relevance**
* **document\_index\_tools** to index scraped pages for semantic search and Q\&A.
* **file\_manager\_tools** to persist captured pages and datasets.
* **project\_manager\_tools** to track research tasks and sources.
* **slack\_tools** to share links and summaries with teams.
* **artifact\_manager\_tools** to preserve snapshots of key findings.
## **Configuration Details**
* Provide fully qualified URLs with `https` when possible.
* Match `search_type` to the content you want to reduce noise. Use Tavily + `include_raw_content` when you need page text back in one call; use serper for image, maps, scholar, or patents result types.
* Configure API keys via Connections (`google_search`, `tavily`), `search_web(action='set_serper_key'|'set_tavily_key')` (creates a personal connection and user default), or `SERPER_API_KEY` / `TAVILY_API_KEY` environment variables. Use shared Connections with role defaults for org-wide keys.
* Set network timeouts conservatively for slower sites.
* Normalize extracted tables before analysis to handle irregular markup.
Some rich pages load data via client-side JavaScript. If a target renders content dynamically, scraping may return partial results.
## **Limitations or Notes**
1. Access can be blocked by anti-bot protections or require sign-in.
2. Dynamic or interactive elements might not render in basic HTML responses.
3. Search relevance depends on the query and the underlying index.
4. International or non-English pages may need language-aware processing.
5. Sponsored results may appear and should be filtered for neutrality.
6. Copyright and usage rights apply to collected content. Assess before reuse.
## **Output**
* **Scraping Results**
Page title, meta description, structured metadata, main content, links, images, table data, and basic load metrics.
* **Search Results**
Ranked items with titles, snippets, URLs, source details, and related-query hints.
* **Saved API and Download Results**
File metadata including `status_code`, `content_type`, `file_size_bytes`, `file_size_mb`, `timestamp`, preview text for JSON/text/CSV/XML where available, and `read_instructions` when the file is saved into a Git-backed repo.
* **Errors and Diagnostics**
Clear messages for network failures, blocked access, invalid URLs, or empty results. Saved API and download failures can also include a masked `response_body`, successful raw-token retries can return an `auth_note` explaining that the request succeeded only after removing the `Bearer ` prefix, and blocked connection-managed provider calls return `error_code="CONNECTION_BACKED_API_REQUIRED"` with the recommended next tool and actions.