# rahulrvu-24/smart-web-scraper architecture

> Architecture diagram of the GitHub repository rahulrvu-24/smart-web-scraper, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rahulrvu-24/smart-web-scraper
- Repository: https://github.com/rahulrvu-24/smart-web-scraper

## Overview

Smart Web Scraper is currently a frontend and FastAPI skeleton with PostgreSQL models and health checks. The documented end-to-end product would accept a URL and extraction schema, guard the request, render the page, extract and validate JSON, then return it. The README marks the queue, workers, and LLM layer as unfinished; those capabilities are shown as documented, unverified branches, not implemented code. The sampled files support the frontend/API health path, settings, database session, and model relationships; other workflow links rely on README statements.

Main entry points: [`backend/app/db/base.py`](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/db/base.py) (ORM base), [`backend/app/main.py`](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/main.py) (FastAPI app), [`backend/app/models/user.py`](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/user.py) (User accounts), [`backend/app`](https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app) (Sandboxed renderer)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["User interface"]
  node_frontend["Next.js pages<br/>[page.tsx]"]
  node_status["Status page<br/>[page.tsx]"]
  node_api_client["Typed API client<br/>[api.ts]"]
end

subgraph group_api["API service"]
  node_fastapi["FastAPI app<br/>[main.py]"]
  node_health["Health endpoints<br/>[health.py]"]
  node_settings["Environment settings<br/>[config.py]"]
end

subgraph group_data["Data model"]
  node_db_session["DB session<br/>[session.py]"]
  node_model_base["ORM base<br/>[base.py]"]
  node_user_model["User accounts<br/>[user.py]"]
  node_api_keys["API keys<br/>[api_key.py]"]
  node_schemas["Extraction schemas"]
  node_jobs["Scrape jobs<br/>[scrape_job.py]"]
  node_results["Scrape results<br/>[scrape_result.py]"]
  node_postgres[("PostgreSQL")]
end

subgraph group_scraping["Scraping workflow"]
  node_url_guard["URL and SSRF guard"]
  node_queue["Job queue"]
  node_worker["Sandboxed renderer"]
  node_extractor["LLM extraction"]
  node_validator["Schema validation"]
  node_object_store[("Object storage")]
end

subgraph group_operations["Security and operations"]
  node_monitoring["Monitoring"]
end

node_user(("Scraper user"))
node_website["Target website"]

node_user -->|"uses"| node_frontend
node_frontend -->|"calls"| node_api_client
node_api_client -->|"requests"| node_fastapi
node_status -->|"checks API"| node_fastapi
node_fastapi -->|"registers"| node_health
node_fastapi -->|"reads CORS settings"| node_settings
node_health -->|"requests session"| node_db_session
node_db_session -->|"reads database URL"| node_settings
node_db_session -->|"connects"| node_postgres
node_model_base -->|"base class"| node_user_model
node_model_base -->|"base class"| node_api_keys
node_model_base -->|"base class"| node_schemas
node_model_base -->|"base class"| node_jobs
node_model_base -->|"base class"| node_results
node_user_model -->|"relates to"| node_api_keys
node_user_model -->|"relates to"| node_schemas
node_user_model -->|"relates to"| node_jobs
node_jobs -->|"references"| node_schemas
node_jobs -->|"relates to"| node_results
node_user -.->|"submits URL and schema"| node_url_guard
node_url_guard -.->|"enqueues safe request"| node_queue
node_queue -.->|"dispatches job"| node_worker
node_worker -.->|"renders page"| node_website
node_worker -.->|"provides page content"| node_extractor
node_extractor -.->|"produces JSON"| node_validator
node_validator -.->|"stores validated output"| node_results
node_worker -.->|"stores snapshots"| node_object_store
node_monitoring -.->|"observes service"| node_fastapi

click node_frontend "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/frontend/app/page.tsx"
click node_status "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/frontend/app/status/page.tsx"
click node_api_client "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/frontend/lib/api.ts"
click node_fastapi "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/main.py"
click node_health "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/api/health.py"
click node_settings "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/core/config.py"
click node_db_session "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/db/session.py"
click node_model_base "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/db/base.py"
click node_user_model "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/user.py"
click node_api_keys "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/api_key.py"
click node_schemas "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/extraction_schema.py"
click node_jobs "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/scrape_job.py"
click node_results "https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/scrape_result.py"
click node_url_guard "https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app"
click node_queue "https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app"
click node_worker "https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app"
click node_extractor "https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app"
click node_validator "https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app"
click node_monitoring "https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend,node_status,node_api_client,node_user toneBlue
class node_fastapi,node_health,node_settings toneAmber
class node_db_session,node_model_base,node_user_model,node_api_keys,node_schemas,node_jobs,node_results,node_postgres toneMint
class node_url_guard,node_queue,node_worker,node_extractor,node_validator,node_object_store toneRose
class node_monitoring,node_website toneIndigo
```

## Components

### User interface

- [Next.js pages](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/frontend/app/page.tsx): `frontend/app/page.tsx`
- [Status page](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/frontend/app/status/page.tsx): `frontend/app/status/page.tsx`
- [Typed API client](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/frontend/lib/api.ts): `frontend/lib/api.ts`

### API service

- [FastAPI app](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/main.py): `backend/app/main.py`
- [Health endpoints](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/api/health.py): `backend/app/api/health.py`
- [Environment settings](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/core/config.py): `backend/app/core/config.py`

### Data model

- [DB session](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/db/session.py): `backend/app/db/session.py`
- [ORM base](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/db/base.py): `backend/app/db/base.py`
- [User accounts](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/user.py): `backend/app/models/user.py`
- [API keys](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/api_key.py): `backend/app/models/api_key.py`
- [Extraction schemas](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/extraction_schema.py): `backend/app/models/extraction_schema.py`
- [Scrape jobs](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/scrape_job.py): `backend/app/models/scrape_job.py`
- [Scrape results](https://github.com/rahulrvu-24/smart-web-scraper/blob/main/backend/app/models/scrape_result.py): `backend/app/models/scrape_result.py`
- **PostgreSQL**

### Scraping workflow

- [URL and SSRF guard](https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app): `backend/app`
- [Job queue](https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app): `backend/app`
- [Sandboxed renderer](https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app): `backend/app`
- [LLM extraction](https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app): `backend/app`
- [Schema validation](https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app): `backend/app`
- **Object storage**

### Security and operations

- [Monitoring](https://github.com/rahulrvu-24/smart-web-scraper/tree/main/backend/app): `backend/app`

### Other components

- **Scraper user**
- **Target website**

## Connections

- Scraper user → Next.js pages: uses (evidence: `README.md`)
- Next.js pages → Typed API client: calls (evidence: `README.md`)
- Typed API client → FastAPI app: requests (evidence: `README.md`)
- Status page → FastAPI app: checks API (evidence: `README.md`)
- FastAPI app → Health endpoints: registers (evidence: `backend/app/main.py`)
- FastAPI app → Environment settings: reads CORS settings (evidence: `backend/app/main.py`)
- Health endpoints → DB session: requests session (evidence: `backend/app/api/health.py`)
- DB session → Environment settings: reads database URL (evidence: `backend/app/db/session.py`)
- DB session → PostgreSQL: connects (evidence: `backend/app/db/session.py`)
- ORM base → User accounts: base class (evidence: `backend/app/models/user.py`)
- ORM base → API keys: base class (evidence: `backend/app/models/api_key.py`)
- ORM base → Extraction schemas: base class (evidence: `backend/app/models/extraction_schema.py`)
- ORM base → Scrape jobs: base class (evidence: `backend/app/models/scrape_job.py`)
- ORM base → Scrape results: base class (evidence: `backend/app/models/scrape_result.py`)
- User accounts → API keys: relates to (evidence: `backend/app/models/user.py`)
- User accounts → Extraction schemas: relates to (evidence: `backend/app/models/user.py`)
- User accounts → Scrape jobs: relates to (evidence: `backend/app/models/user.py`)
- Scrape jobs → Extraction schemas: references (evidence: `backend/app/models/scrape_job.py`)
- Scrape jobs → Scrape results: relates to (evidence: `backend/app/models/scrape_job.py`)
- Scraper user → URL and SSRF guard: submits URL and schema (evidence: `README.md`)
- URL and SSRF guard → Job queue: enqueues safe request (evidence: `README.md`)
- Job queue → Sandboxed renderer: dispatches job (evidence: `README.md`)
- Sandboxed renderer → Target website: renders page (evidence: `README.md`)
- Sandboxed renderer → LLM extraction: provides page content (evidence: `README.md`)
- LLM extraction → Schema validation: produces JSON (evidence: `README.md`)
- Schema validation → Scrape results: stores validated output (evidence: `README.md`)
- Sandboxed renderer → Object storage: stores snapshots (evidence: `README.md`)
- Monitoring → FastAPI app: observes service (evidence: `README.md`)