How rahulrvu-24/smart-web-scraper works
Smart Web Scraper is currently a frontend and FastAPI skeleton with PostgreSQL models and health checks. The documented end-to-end product would accept a URL and extraction schema, guard the request, render the page, extract and validate JSON, then return it. The README marks the queue, workers, and LLM layer as unfinished; those capabilities are shown as documented, unverified branches, not implemented code. The sampled files support the frontend/API health path, settings, database session, and model relationships; other workflow links rely on README statements.
Main entry points: base.py ORM base · main.py FastAPI app · user.py User accounts · app Sandboxed renderer
Components 23
User interface
- Next.js pages
frontend/app/page.tsx - Status page
frontend/app/status/page.tsx - Typed API client
frontend/lib/api.ts
API service
- FastAPI app
backend/app/main.py - Health endpoints
backend/app/api/health.py - Environment settings
backend/app/core/config.py
Data model
- DB session
backend/app/db/session.py - ORM base
backend/app/db/base.py - User accounts
backend/app/models/user.py - API keys
backend/app/models/api_key.py - Extraction schemas
backend/app/models/extraction_schema.py - Scrape jobs
backend/app/models/scrape_job.py - Scrape results
backend/app/models/scrape_result.py - PostgreSQL
Scraping workflow
- URL and SSRF guard
backend/app - Job queue
backend/app - Sandboxed renderer
backend/app - LLM extraction
backend/app - Schema validation
backend/app - Object storage
Security and operations
- Monitoring
backend/app
Other components
- Scraper user
- Target website
Connections 28
- Scraper user → Next.js pages: uses
README.md - Next.js pages → Typed API client: calls
README.md - Typed API client → FastAPI app: requests
README.md - Status page → FastAPI app: checks API
README.md - FastAPI app → Health endpoints: registers
backend/app/main.py - FastAPI app → Environment settings: reads CORS settings
backend/app/main.py - Health endpoints → DB session: requests session
backend/app/api/health.py - DB session → Environment settings: reads database URL
backend/app/db/session.py - DB session → PostgreSQL: connects
backend/app/db/session.py - ORM base → User accounts: base class
backend/app/models/user.py - ORM base → API keys: base class
backend/app/models/api_key.py - ORM base → Extraction schemas: base class
backend/app/models/extraction_schema.py - ORM base → Scrape jobs: base class
backend/app/models/scrape_job.py - ORM base → Scrape results: base class
backend/app/models/scrape_result.py - User accounts → API keys: relates to
backend/app/models/user.py - User accounts → Extraction schemas: relates to
backend/app/models/user.py - User accounts → Scrape jobs: relates to
backend/app/models/user.py - Scrape jobs → Extraction schemas: references
backend/app/models/scrape_job.py - Scrape jobs → Scrape results: relates to
backend/app/models/scrape_job.py - Scraper user → URL and SSRF guard: submits URL and schema
README.md - URL and SSRF guard → Job queue: enqueues safe request
README.md - Job queue → Sandboxed renderer: dispatches job
README.md - Sandboxed renderer → Target website: renders page
README.md - Sandboxed renderer → LLM extraction: provides page content
README.md - LLM extraction → Schema validation: produces JSON
README.md - Schema validation → Scrape results: stores validated output
README.md - Sandboxed renderer → Object storage: stores snapshots
README.md - Monitoring → FastAPI app: observes service
README.md