# utkarsh-wadalkar/data-intelligence architecture

> Architecture diagram of the GitHub repository utkarsh-wadalkar/data-intelligence, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/utkarsh-wadalkar/data-intelligence
- Repository: https://github.com/utkarsh-wadalkar/data-intelligence

## Overview

SourcePilot turns a user’s research request into an approved collection workflow, then runs public-web search, scraping, structured extraction, evidence validation, and identity-based deduplication. Users monitor runs and quotas, inspect source-backed records, and search, filter, or export the cumulative dataset; approved workflows can be scheduled for daily or weekly reruns. The graph reflects the sampled API, worker, provider, and frontend code plus README-documented capabilities; unsampled route wiring is omitted where it cannot be confirmed.

Main entry points: [`backend/app/main.py`](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/main.py) (API routes), [`backend/app/worker.py`](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/worker.py) (Run processor), [`backend/app/db.py`](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/db.py) (Database sessions), [`frontend/src/App.tsx`](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/frontend/src/App.tsx) (Workflow workspace)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_client["Web application"]
  node_frontend["Workflow workspace<br/>[App.tsx]"]
  node_clerk_ui["Clerk sign-in<br/>[main.tsx]"]
end

subgraph group_api["Workflow API"]
  node_api_main["API routes<br/>[main.py]"]
  node_scheduler["Run scheduling<br/>[scheduling.py]"]
end

subgraph group_execution["Collection execution"]
  node_worker["Run processor<br/>[worker.py]"]
  node_quota["Quota accounting<br/>[quotas.py]"]
  node_modal["API and scheduler jobs<br/>[modal_app.py]"]
end

subgraph group_data["State and access"]
  node_auth["Session authorization<br/>[auth.py]"]
  node_database[("Application data<br/>[models.py]")]
  node_db_sessions["Database sessions<br/>[db.py]"]
end

subgraph group_integrations["External services"]
  node_planning["Plan proposal<br/>[providers.py]"]
  node_provider["Search and extraction<br/>[providers.py]"]
  node_clerk["Clerk organization"]
  node_tidb[("TiDB Cloud")]
  node_openrouter["OpenRouter models"]
  node_firecrawl["Firecrawl web data"]
end

node_user(("Research user"))

node_user -->|"submits requests"| node_frontend
node_frontend -->|"uses sign-in"| node_clerk_ui
node_clerk_ui -->|"authenticates"| node_clerk
node_frontend -->|"calls API"| node_api_main
node_api_main -->|"checks sessions"| node_auth
node_auth -->|"verifies JWT"| node_clerk
node_api_main -->|"proposes plan"| node_planning
node_planning -->|"requests model output"| node_openrouter
node_api_main -->|"enqueues runs"| node_scheduler
node_api_main -.->|"dispatches jobs"| node_modal
node_api_main -->|"starts local work"| node_worker
node_modal -->|"serves API"| node_api_main
node_modal -->|"starts or recovers"| node_worker
node_worker -->|"searches and extracts"| node_provider
node_provider -->|"searches and scrapes"| node_firecrawl
node_provider -->|"requests extraction"| node_openrouter
node_worker -->|"consumes allowance"| node_quota
node_api_main -->|"checks usage"| node_quota
node_api_main -->|"reads and writes"| node_db_sessions
node_worker -->|"reads and writes"| node_db_sessions
node_scheduler -->|"stores run state"| node_db_sessions
node_db_sessions -->|"accesses records"| node_database
node_db_sessions -->|"persists data"| node_tidb

click node_frontend "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/frontend/src/App.tsx"
click node_clerk_ui "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/frontend/src/main.tsx"
click node_api_main "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/main.py"
click node_planning "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/providers.py"
click node_scheduler "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/scheduling.py"
click node_worker "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/worker.py"
click node_quota "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/quotas.py"
click node_provider "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/providers.py"
click node_auth "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/auth.py"
click node_database "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/models.py"
click node_db_sessions "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/db.py"
click node_modal "https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/modal_app.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend,node_clerk_ui,node_user toneBlue
class node_api_main,node_scheduler toneAmber
class node_worker,node_quota,node_modal toneMint
class node_auth,node_database,node_db_sessions toneRose
class node_planning,node_provider,node_clerk,node_tidb,node_openrouter,node_firecrawl toneIndigo
```

## Components

### Web application

- [Workflow workspace](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/frontend/src/App.tsx): `frontend/src/App.tsx`
- [Clerk sign-in](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/frontend/src/main.tsx): `frontend/src/main.tsx`

### Workflow API

- [API routes](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/main.py): `backend/app/main.py`
- [Run scheduling](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/scheduling.py): `backend/app/scheduling.py`

### Collection execution

- [Run processor](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/worker.py): `backend/app/worker.py`
- [Quota accounting](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/quotas.py): `backend/app/quotas.py`
- [API and scheduler jobs](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/modal_app.py): `backend/modal_app.py`

### State and access

- [Session authorization](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/auth.py): `backend/app/auth.py`
- [Application data](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/models.py): `backend/app/models.py`
- [Database sessions](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/db.py): `backend/app/db.py`

### External services

- [Plan proposal](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/providers.py): `backend/app/providers.py`
- [Search and extraction](https://github.com/utkarsh-wadalkar/data-intelligence/blob/main/backend/app/providers.py): `backend/app/providers.py`
- **Clerk organization**
- **TiDB Cloud**
- **OpenRouter models**
- **Firecrawl web data**

### Other components

- **Research user**

## Connections

- Research user → Workflow workspace: submits requests
- Workflow workspace → Clerk sign-in: uses sign-in
- Clerk sign-in → Clerk organization: authenticates
- Workflow workspace → API routes: calls API
- API routes → Session authorization: checks sessions
- Session authorization → Clerk organization: verifies JWT
- API routes → Plan proposal: proposes plan
- Plan proposal → OpenRouter models: requests model output
- API routes → Run scheduling: enqueues runs
- API routes → API and scheduler jobs: dispatches jobs
- API routes → Run processor: starts local work
- API and scheduler jobs → API routes: serves API
- API and scheduler jobs → Run processor: starts or recovers
- Run processor → Search and extraction: searches and extracts
- Search and extraction → Firecrawl web data: searches and scrapes
- Search and extraction → OpenRouter models: requests extraction
- Run processor → Quota accounting: consumes allowance
- API routes → Quota accounting: checks usage
- API routes → Database sessions: reads and writes
- Run processor → Database sessions: reads and writes
- Run scheduling → Database sessions: stores run state
- Database sessions → Application data: accesses records
- Database sessions → TiDB Cloud: persists data