A practical guide to web search, extraction, crawling, deep research, structured data, and browser automation.
AI harnesses such as ChatGPT, Codex, Claude , Claude Code, Manus, Cursor, OpenCode, Command Code and similar agent environments usually include web search or page-reading tools. Those built-in capabilities are often enough.
For more demanding work, external research providers can add broader discovery, semantic search, precise filters, site mapping and crawling, structured datasets, batch enrichment, cited synthesis, browser-rendered extraction, or interaction with dynamic pages.
This guide uses Octen, Exa, Perplexity, Parallel, Firecrawl, and TinyFish as practical examples. They are illustrative, not exhaustive or ranked.
The central idea is simple: give each provider a distinct role, use only the providers the task needs, verify important claims against source content the agent has inspected, and report what worked.
Last substantively reviewed: September 2026. Provider tools, schemas, availability, and commercial terms can change.
New to external research tools? Start with Native search or external providers?, then try the short prompt below.
Already have a provider connected? Go to Try it first or Use the reusable routing skill using this page’s table of contents.
| Term | Meaning here |
|---|---|
| AI harness | The application or agent environment that plans the task, calls tools, and presents the result |
| Provider | An external service supplying search, extraction, structured data, synthesis, or platform-native evidence |
| MCP server or connector | A supported way for the harness to call provider tools |
| Skill | Reusable instructions that tell the harness how to route work; a skill does not create provider access by itself |
| Retrieval | Finding or reading sources for the harness-selected model to analyze |
| Provider-generated synthesis | An answer, comparison, research report, or agent result produced by a model or workflow inside the provider |
| Evidence lane | A distinct source type or dataset assigned a clear role in the research |
Start with the search and page-reading tools already available in your AI application. They may be sufficient for a straightforward lookup or reading a known source.
Consider external providers when you need something specific: more precise discovery controls, specialized datasets, repeated research across a list, website mapping or crawling, browser-rendered pages, or provider-generated research.
Use the smallest set that covers the task. External services can add setup, cost, latency, and another service receiving the query. Capabilities vary by application and integration, so verify what is available rather than assuming an external tool is always better.