Web Access Tools

​

Overview

​

Tool Name

web_access_tools

​

Purpose

​

Key Features & Functions

​

Input Parameters for Each Function

​

_scrape_url

​

search_web

• serper — Google SERP via Serper (serper.dev). Best for navigational queries and the full range of search_type values.
• tavily — LLM/agent-optimized search (tavily.com). Returns pre-processed snippets and can return extracted page text inline (include_raw_content), collapsing search → fetch → parse into one call.

# Default-provider web search
search_web(query="Python tutorials")

# Force Tavily with inline page-content extraction
search_web(
    query="Maryland MSRPS investment consultant 2024 ACFR",
    provider="tavily",
    search_depth="advanced",
    include_raw_content=True,
    include_domains=["sra.maryland.gov"],
)

# Force Serper image search
search_web(query="sunset", provider="serper", search_type="images", max_results=20)

# On Genesis AI, search without a Serper or Tavily key
search_web(query="latest Genesis AI updates")

​

get_web_content with action="ask_user_for_secret"

{
  "action": "ask_user_for_secret",
  "secret_hint": "secret_linear_api_key;not_tried;Please provide your Linear API key"
}

​

get_web_content with action="api_data"

​

auth.type="api_key" header and query options

{
  "action": "api_data",
  "url": "https://api.linear.app/graphql",
  "auth": {
    "type": "api_key",
    "key": "secret_linear_api_key",
    "location": "header",
    "header_name": "Authorization"
  }
}

​

connection_api_action

# GitLab
result = connection_api_action(
    connection_type="gitlab",
    connection_name="work",
    method="GET",
    path="/api/v4/user"
)

# Jira Service Management
result = connection_api_action(
    connection_type="jira",
    connection_name="jira.ayou",
    method="GET",
    path="/rest/servicedeskapi/request/CUSTOMER-1/comment"
)

# Azure Data Factory
result = connection_api_action(
    connection_type="azure_data_factory",
    connection_name="azure_data_factory.production",
    method="GET",
    path="/pipelines"
)

​

Use Cases

1. Market and industry research Identify trends, competitors, and benchmarks from trusted sources.
2. Real-time news monitoring Track breaking updates that affect active missions and roadmaps.
3. Technical discovery Locate docs, guides, and best practices for engineering solutions.
4. Scholarly evidence gathering Pull academic articles or patents to support recommendations.
5. Vendor and product validation Collect specifications and pricing from official sites.
6. Regulatory checks Retrieve policies and compliance notes from government domains.
7. Visual asset sourcing Find images or videos for briefings and presentations.
8. Connection-backed provider API calls Call long-tail GitLab or Jira/JSM REST endpoints through a saved Connection without exposing PATs or bearer tokens in chat history.

​

Workflow/How It Works

1. Search with search_web using carefully scoped keywords, a provider when you need to force Serper or Tavily, and a search_type when needed.
2. Select sources by reviewing titles, snippets, and domains for credibility.
3. Scrape chosen pages with _scrape_url to extract structured content.
4. Store or index outputs for retrieval and cross-referencing in later steps.
5. Cite and share the most relevant excerpts in reports, docs, or Slack updates.

​

Integration Relevance

• document_index_tools to index scraped pages for semantic search and Q&A.
• file_manager_tools to persist captured pages and datasets.
• project_manager_tools to track research tasks and sources.
• slack_tools to share links and summaries with teams.
• artifact_manager_tools to preserve snapshots of key findings.

​

Configuration Details

• Provide fully qualified URLs with https when possible.
• Match search_type to the content you want to reduce noise. Use Tavily + include_raw_content when you need page text back in one call; use serper for image, maps, scholar, or patents result types.
• Configure API keys via Connections (google_search, tavily), search_web(action='set_serper_key'|'set_tavily_key') (creates a personal connection and user default), or SERPER_API_KEY / TAVILY_API_KEY environment variables. Use shared Connections with role defaults for org-wide keys.
• Set network timeouts conservatively for slower sites.
• Normalize extracted tables before analysis to handle irregular markup.

​

Limitations or Notes

1. Access can be blocked by anti-bot protections or require sign-in.
2. Dynamic or interactive elements might not render in basic HTML responses.
3. Search relevance depends on the query and the underlying index.
4. International or non-English pages may need language-aware processing.
5. Sponsored results may appear and should be filtered for neutrality.
6. Copyright and usage rights apply to collected content. Assess before reuse.

​

Output

• Scraping Results Page title, meta description, structured metadata, main content, links, images, table data, and basic load metrics.
• Search Results Ranked items with titles, snippets, URLs, source details, and related-query hints.
• Saved API and Download Results File metadata including status_code, content_type, file_size_bytes, file_size_mb, timestamp, preview text for JSON/text/CSV/XML where available, and read_instructions when the file is saved into a Git-backed repo.
• Errors and Diagnostics Clear messages for network failures, blocked access, invalid URLs, or empty results. Saved API and download failures can also include a masked response_body, successful raw-token retries can return an auth_note explaining that the request succeeded only after removing the Bearer prefix, and blocked connection-managed provider calls return error_code="CONNECTION_BACKED_API_REQUIRED" with the recommended next tool and actions.