How utkarsh-wadalkar/data-intelligence works
SourcePilot turns a user’s research request into an approved collection workflow, then runs public-web search, scraping, structured extraction, evidence validation, and identity-based deduplication. Users monitor runs and quotas, inspect source-backed records, and search, filter, or export the cumulative dataset; approved workflows can be scheduled for daily or weekly reruns. The graph reflects the sampled API, worker, provider, and frontend code plus README-documented capabilities; unsampled route wiring is omitted where it cannot be confirmed.
Main entry points: main.py API routes · worker.py Run processor · db.py Database sessions · App.tsx Workflow workspace
Components 17
Web application
- Workflow workspace
frontend/src/App.tsx - Clerk sign-in
frontend/src/main.tsx
Workflow API
- API routes
backend/app/main.py - Run scheduling
backend/app/scheduling.py
Collection execution
- Run processor
backend/app/worker.py - Quota accounting
backend/app/quotas.py - API and scheduler jobs
backend/modal_app.py
State and access
- Session authorization
backend/app/auth.py - Application data
backend/app/models.py - Database sessions
backend/app/db.py
External services
- Plan proposal
backend/app/providers.py - Search and extraction
backend/app/providers.py - Clerk organization
- TiDB Cloud
- OpenRouter models
- Firecrawl web data
Other components
- Research user
Connections 23
- Research user → Workflow workspace: submits requests
- Workflow workspace → Clerk sign-in: uses sign-in
- Clerk sign-in → Clerk organization: authenticates
- Workflow workspace → API routes: calls API
- API routes → Session authorization: checks sessions
- Session authorization → Clerk organization: verifies JWT
- API routes → Plan proposal: proposes plan
- Plan proposal → OpenRouter models: requests model output
- API routes → Run scheduling: enqueues runs
- API routes → API and scheduler jobs: dispatches jobs
- API routes → Run processor: starts local work
- API and scheduler jobs → API routes: serves API
- API and scheduler jobs → Run processor: starts or recovers
- Run processor → Search and extraction: searches and extracts
- Search and extraction → Firecrawl web data: searches and scrapes
- Search and extraction → OpenRouter models: requests extraction
- Run processor → Quota accounting: consumes allowance
- API routes → Quota accounting: checks usage
- API routes → Database sessions: reads and writes
- Run processor → Database sessions: reads and writes
- Run scheduling → Database sessions: stores run state
- Database sessions → Application data: accesses records
- Database sessions → TiDB Cloud: persists data