External Web Search

The External Web Search stage integrates Exa’s neural search API to augment your results with real-time web content. This enables hybrid retrieval combining your indexed documents with fresh web results.

Stage Category: APPLY (Enriches pipeline with web results)Transformation: N documents → N + M documents (web results added)

When to Use

Use Case	Description
Knowledge augmentation	Supplement internal docs with web content
Real-time information	Access current events, news, updates
Research expansion	Broaden search beyond your corpus
Competitive intelligence	Include competitor content in results

When NOT to Use

Scenario	Recommended Alternative
Internal-only search	Skip this stage
Sensitive/confidential queries	Use only indexed content
Low-latency requirements	Web search adds 200-500ms

Parameters

Parameter	Type	Default	Description
`query`	string	`{{INPUT.query}}`	Search query (supports templates)
`num_results`	integer	`10`	Number of web results to retrieve (1-100)
`start_published_date`	string	`null`	Filter by publish date (`YYYY-MM-DD`)
`category`	string	`null`	Content category filter
`use_autoprompt`	boolean	`true`	Let Exa optimize the query
`include_text`	boolean	`true`	Include text snippets in results

Available Categories

Category	Description
`company`	Company websites and profiles
`research_paper`	Academic and research content
`news`	News articles
`pdf`	PDF documents
`github`	GitHub repositories
`tweet`	Twitter/X content
`personal_site`	Personal websites and blogs

Configuration Examples

{
  "stage_name": "external_web_search",
  "stage_type": "apply",
  "config": {
    "stage_id": "external_web_search",
    "parameters": {
      "query": "{{INPUT.query}}",
      "num_results": 10
    }
  }
}

{
  "stage_name": "external_web_search",
  "stage_type": "apply",
  "config": {
    "stage_id": "external_web_search",
    "parameters": {
      "query": "{{INPUT.query}}",
      "num_results": 5,
      "category": "research_paper"
    }
  }
}

{
  "stage_name": "external_web_search",
  "stage_type": "apply",
  "config": {
    "stage_id": "external_web_search",
    "parameters": {
      "query": "{{INPUT.query}}",
      "num_results": 10,
      "category": "news",
      "start_published_date": "2024-01-01"
    }
  }
}

{
  "stage_name": "external_web_search",
  "stage_type": "apply",
  "config": {
    "stage_id": "external_web_search",
    "parameters": {
      "query": "{{INPUT.query}}",
      "num_results": 20,
      "category": "research_paper",
      "include_text": true
    }
  }
}

{
  "stage_name": "external_web_search",
  "stage_type": "apply",
  "config": {
    "stage_id": "external_web_search",
    "parameters": {
      "query": "{{INPUT.query}} implementation",
      "num_results": 10,
      "category": "github",
      "use_autoprompt": false
    }
  }
}

Content Extraction

Set include_text to true (default) to include text snippets in each result. Disable it to reduce API costs and response size.

Output Schema

Web results are added to the document set with a source: "web" marker:

{
  "document_id": "web_abc123",
  "source": "web",
  "url": "https://example.com/article",
  "title": "Article Title",
  "content": "Full extracted text content...",
  "published_date": "2024-03-15T10:30:00Z",
  "author": "John Doe",
  "score": 0.95,
  "metadata": {
    "domain": "example.com",
    "category": "news"
  }
}

Exa Neural Search

Exa uses neural search rather than keyword matching:

Feature	Description
Semantic understanding	Understands query intent
Neural ranking	ML-based relevance scoring
Content extraction	Automatic text extraction
Autoprompt	Query optimization for better results

Enable use_autoprompt (default) for natural language queries. Disable it when you need exact phrase matching or have already optimized your query.

Performance

Metric	Value
Latency	200-500ms
Rate limits	Based on Exa plan
Parallel execution	Concurrent with pipeline
Caching	Results cached for 1 hour

Common Pipeline Patterns

Internal + Web Hybrid Search

[
  {
    "stage_name": "semantic_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 20 }
        ],
        "final_top_k": 20
      }
    }
  },
  {
    "stage_name": "external_web_search",
    "stage_type": "apply",
    "config": {
      "stage_id": "external_web_search",
      "parameters": {
        "query": "{{INPUT.query}}",
        "num_results": 10
      }
    }
  },
  {
    "stage_name": "rerank",
    "stage_type": "sort",
    "config": {
      "stage_id": "rerank",
      "parameters": {
        "inference_name": "BAAI__bge_reranker_v2_m3",
        "top_k": 10
      }
    }
  }
]

Web-Augmented RAG

[
  {
    "stage_name": "hybrid_search",
    "stage_type": "filter",
    "config": {
      "stage_id": "feature_search",
      "parameters": {
        "searches": [
          { "feature_uri": "mixpeek://text_extractor@v1/multilingual_e5_large_instruct_v1", "query": { "input_mode": "text", "value": "{{INPUT.query}}" }, "top_k": 30 }
        ],
        "final_top_k": 30
      }
    }
  },
  {
    "stage_name": "external_web_search",
    "stage_type": "apply",
    "config": {
      "stage_id": "external_web_search",
      "parameters": {
        "query": "{{INPUT.query}}",
        "num_results": 5,
        "category": "news",
        "start_published_date": "{{INPUT.date_filter}}"
      }
    }
  },
  {
    "stage_name": "rag_prepare",
    "stage_type": "apply",
    "config": {
      "stage_id": "rag_prepare",
      "parameters": {
        "max_tokens": 8000,
        "output_mode": "single_context"
      }
    }
  }
]

Error Handling

Error	Behavior
API rate limit	Retry with backoff
Network timeout	Stage fails gracefully, no web results
Invalid domain	Ignored, other domains searched
No results found	Empty web result set

Web search results may include content from untrusted sources. Consider filtering or validating web content before using in sensitive applications.

Web Scrape - Extract content from specific URLs
Feature Search - Search your indexed content
Rerank - Combine and rank mixed results

Get started

Connect your data

Extract features

Build retrievers

Enrich & organize

Integrate & operate

Resources

External Web Search

When to Use

When NOT to Use

Parameters

Available Categories

Configuration Examples

Content Extraction

Output Schema

Exa Neural Search

Performance

Common Pipeline Patterns

Internal + Web Hybrid Search

Web-Augmented RAG

Error Handling

​When to Use

​When NOT to Use

​Parameters

​Available Categories

​Configuration Examples

​Content Extraction

​Output Schema

​Exa Neural Search

​Performance

​Common Pipeline Patterns

​Internal + Web Hybrid Search

​Web-Augmented RAG

​Error Handling

​Related

When to Use

When NOT to Use

Parameters

Available Categories

Configuration Examples

Content Extraction

Output Schema

Exa Neural Search

Performance

Common Pipeline Patterns

Internal + Web Hybrid Search

Web-Augmented RAG

Error Handling

Related