How sreenuraj/web-mcp works
web-mcp is a local, stdio MCP server that lets an agent conduct bounded web research. The workflow starts with a declared research plan, then searches and fetches pages under session budgets and policy checks; fetched content is extracted, ranked and returned as untrusted passages. When limits or policies require consent, the user decides through the approval flow. The README documents these subsystem relationships; sampled source confirms the server-to-engine tool dispatch, provider fallback, and approval persistence. Other internal links are based on README architecture documentation, not unsampled implementation details.
Main entry points: core.py Research engine · policy Policy checks · chunk_rank.py Passage ranking · broker.py Approval broker
Components 24
MCP entry
- Tool server
src/web_mcp/server.py
Research engine
- Research engine
src/web_mcp/core.py - Configuration
src/web_mcp/config.py - Text utilities
src/web_mcp/textutil.py
Guardrails
- Policy checks
src/web_mcp/policy - Leak detection
src/web_mcp/policy/leakage.py - Relevance checks
src/web_mcp/policy/relevance.py - Rate limits
src/web_mcp/policy/budget.py - Domain rules
src/web_mcp/policy/domains.py - Injection defense
src/web_mcp/policy/injection.py - Approval broker
src/web_mcp/approval/broker.py
Web retrieval
- Provider chain
src/web_mcp/providers/chain.py - Provider interface
src/web_mcp/providers/base.py - Search backends
src/web_mcp/providers - Safe HTTP
src/web_mcp/net/safe_http.py - HTML extraction
src/web_mcp/extract/html.py - PDF extraction
src/web_mcp/extract/pdf.py - Passage ranking
src/web_mcp/extract/chunk_rank.py
State and audit
- Sessions and budgets
src/web_mcp/session.py - SQLite state and cache
src/web_mcp/storage.py - Audit log
src/web_mcp/audit.py
Other components
- MCP agent
- User
- Public websites
Connections 27
- MCP agent → Tool server: invokes tools
README.md - Tool server → Research engine: dispatches calls
src/web_mcp/server.py - Research engine → Sessions and budgets: manages sessions
README.md - Research engine → Policy checks: checks requests
README.md - Research engine → Provider chain: searches through
README.md - Research engine → Safe HTTP: fetches through
README.md - Research engine → HTML extraction: extracts pages
README.md - Research engine → PDF extraction: extracts PDFs
README.md - Research engine → Passage ranking: selects passages
README.md - Research engine → Approval broker: requests approval
README.md - Research engine → Audit log: records events
README.md - Research engine → Configuration: uses settings
src/web_mcp/core.py - Policy checks → Leak detection: scans for leaks
README.md - Policy checks → Relevance checks: checks relevance
README.md - Policy checks → Rate limits: checks limits
README.md - Policy checks → Domain rules: checks domains
README.md - Policy checks → Injection defense: checks content
README.md - Approval broker → User: elicits decision
README.md - Approval broker → SQLite state and cache: persists decisions
src/web_mcp/approval/broker.py - Sessions and budgets → SQLite state and cache: persists sessions
README.md - Provider chain → Provider interface: uses interface
src/web_mcp/providers/chain.py - Provider chain → Search backends: dispatches search
src/web_mcp/providers/chain.py - Search backends → Public websites: queries search services
README.md - Safe HTTP → Public websites: fetches pages
README.md - HTML extraction → Passage ranking: provides page text
README.md - PDF extraction → Passage ranking: provides document text
README.md - Passage ranking → Text utilities: uses text helpers
src/web_mcp/extract/chunk_rank.py