How hmshb/scraping-agent-ai works
Scraping Agent AI accepts a seed URL, initializes workflow state, crawls for same-domain links, scrapes those links in batches, formats content with an LLM, and saves structured JSON. LangGraph owns stage registration, sequencing, and retry routing; the stage implementations own calls to Firecrawl, URL filtering, formatting, and file output. The README also describes flexible output and LangSmith monitoring, but sampled source does not show their runtime wiring, so those relationships are omitted or marked unverified.
Main entry points: graph.py LangGraph workflow · nodes.py Crawl links · helpers.py URL filtering and errors · agent.py Agent entry
Components 17
Workflow orchestration
- Agent entry
agent/agent.py - LangGraph workflow
agent/utils/graph.py
Scraping stages
- Initialize state
agent/utils/nodes.py - Crawl links
agent/utils/nodes.py - Batch scrape
agent/utils/nodes.py - Format content
agent/utils/nodes.py - Save results
agent/utils/nodes.py - Retry handler
agent/utils/nodes.py - Saved JSON
agent/utils/nodes.py
State and helpers
- Workflow state
agent/utils/states.py - URL filtering and errors
agent/utils/helpers.py
External services
- Firecrawl client
agent/utils/firecrawl.py - Firecrawl service
- Anthropic model
agent/utils/llm.py - Anthropic API
- LangSmith
Other components
- URL requester
Connections 31
- URL requester → Agent entry: starts run
README.md - Agent entry → LangGraph workflow: builds graph
agent/agent.py - LangGraph workflow → Workflow state: uses schemas
agent/utils/graph.py - LangGraph workflow → Initialize state: registers
agent/utils/graph.py - LangGraph workflow → Crawl links: registers
agent/utils/graph.py - LangGraph workflow → Batch scrape: registers
agent/utils/graph.py - LangGraph workflow → Format content: registers
agent/utils/graph.py - LangGraph workflow → Save results: registers
agent/utils/graph.py - LangGraph workflow → Retry handler: registers
agent/utils/graph.py - LangGraph workflow → Initialize state: starts at
agent/utils/graph.py - LangGraph workflow → Crawl links: routes next
agent/utils/graph.py - LangGraph workflow → Batch scrape: routes on success
agent/utils/graph.py - LangGraph workflow → Format content: routes on success
agent/utils/graph.py - LangGraph workflow → Save results: routes on success
agent/utils/graph.py - LangGraph workflow → Retry handler: routes on failure
agent/utils/graph.py - LangGraph workflow → Crawl links: retries failed stage
agent/utils/graph.py - LangGraph workflow → Batch scrape: retries failed stage
agent/utils/graph.py - LangGraph workflow → Format content: retries failed stage
agent/utils/graph.py - Initialize state → Workflow state: creates state
agent/utils/nodes.py - Crawl links → Firecrawl client: crawls URL
agent/utils/nodes.py - Crawl links → URL filtering and errors: filters links
agent/utils/nodes.py - Batch scrape → Firecrawl client: scrapes batches
agent/utils/nodes.py - Format content → Anthropic model: formats content
agent/utils/nodes.py - Save results → Saved JSON: writes results
agent/utils/nodes.py - Crawl links → URL filtering and errors: uses error wrapper
agent/utils/nodes.py - Batch scrape → URL filtering and errors: uses error wrapper
agent/utils/nodes.py - Format content → URL filtering and errors: uses error wrapper
agent/utils/nodes.py - Save results → URL filtering and errors: uses error wrapper
agent/utils/nodes.py - Firecrawl client → Firecrawl service: requests crawl
agent/utils/firecrawl.py - Anthropic model → Anthropic API: calls model
agent/utils/llm.py - Agent entry → LangSmith: monitoring integration
README.md