How dilip-sheesh/bap works
1. Repository shape
- This is a two-tier application: a Next.js 15 frontend in frontend/ and a FastAPI backend in backend/.
- The backend does the data acquisition, cleaning, analytics, AI calls, and persistence. The frontend is a thin interactive shell for search, dataset audit, correlation, and maps.
2. Frontend surface area
- The App Router is the main UI boundary:
- frontend/app/page.tsx is the landing page and dataset discovery/search entry point.
- frontend/app/dataset/[id]/page.tsx is the primary audit dashboard for one dataset.
- frontend/app/correlate/page.tsx is the cross-dataset correlation view.
- frontend/app/components/StopsMap.tsx is the geo-visualization component used when latitude/longitude data exists.
- Styling and global layout live in globals.css and layout.tsx, with the rest of the UI driven from these route pages.
3. Backend entry points
- backend/app/main.py is the central FastAPI application and likely contains the API routes described in the README.
- backend/api/index.py exists as an alternate entry module, which usually indicates a serverless or platform-specific wrapper for deployment.
- backend/app/database.py isolates database/client initialization so the API layer can access Supabase without hard-coding connection setup everywhere.
4. Primary data sources
- The system is built around live civic datasets rather than a preloaded warehouse.
- Source data is fetched on demand from open government portals and direct file links, then parsed as CSV/XLSX/PDF.
- A Supabase catalog stores dataset metadata such as titles, descriptions, source URLs, tags, and headers; it is not the analytical store for the raw datasets.
5. Core request flow for a dataset audit
- The browser opens an SSE/EventSource stream to the backend for a specific dataset.
- The backend fetches the raw file, parses it into a dataframe, cleans and normalizes it, and runs analytics.
- Progress and intermediate states are streamed back to the UI, then the final audit payload is delivered when processing completes.
- The frontend uses that live stream to drive the dashboard, progress indicators, and terminal-style analysis display.
6. Analytics boundary
- The analytical engine is dataframe-based, centered on Pandas/NumPy in the backend.
- It chooses a useful numeric column, excludes noise columns such as IDs and coordinates, and runs anomaly detection.
- That output feeds both charts and explanations, so the same backend analysis powers visualization, audit text, and report generation.
7. AI subsystem
- Groq-hosted Llama 3.1 is used for several distinct tasks:
- natural-language dataset analysis,
- natural-language-to-dataframe query translation,
- cross-dataset correlation synthesis.
- The AI is not the source of truth; it consumes the structured stats/anomalies produced by the analytics layer and turns them into human-readable insight.
8. Cross-dataset correlation
- The correlation route takes two datasets, audits both, and compares their anomaly sets.
- The meaningful output is the overlap between entities/locations/departments flagged as unusual in both sources.
- This is a separate interaction path from single-dataset auditing, but it uses the same backend data-fetch and analytics primitives.
9. Persistence boundary
- Supabase serves two roles:
- a catalog store for dataset metadata,
- a report store for saved audit outputs.
- Saved reports include stats, anomaly flags, AI analysis, chart data, and query history, allowing a shareable audit result independent of the live fetch.
- The architecture keeps raw source data transient while persisting only curated metadata and report artifacts.
10. Geo and visualization layer
- When the backend or frontend detects coordinate fields, the UI can render a map using the Leaflet-based StopsMap component.
- Charts are rendered in the frontend from backend-derived metric data, so visual selection follows the analytics model rather than arbitrary frontend logic.
- The map and charts are presentation layers only; they do not own the data logic.
11. Deployment and runtime boundaries
- The backend includes Dockerfile, Procfile, railway.json, and vercel.json, which suggests it is designed to run on multiple hosting targets, including containerized and serverless-style deployments.
- The frontend is a standard Next.js app with its own build/runtime configuration in next.config.ts, tsconfig.json, and package tooling.
- Runtime communication is HTTP-based, with SSE used where progressive updates matter.
12. What to draw in an architecture diagram
- Show a client tier with three main views: discovery, dataset audit, and correlation.
- Show a FastAPI service in the middle with endpoints for streaming audits, AI analysis, semantic search, correlation, seeding, and report storage.
- Show two external data domains feeding the backend: Supabase for metadata/reports and government open-data portals for live raw datasets.
- Emphasize that the raw dataset is fetched just-in-time, processed server-side, and then converted into streamed UI state plus AI-generated narrative.
Main entry points: main.py API app · [id]/page.tsx Dataset audit · database.py Supabase client · app/page.tsx Discover
Components 15
Frontend
Next.js client tier with discovery, audit, and correlation views
- App shell · Global layout and route boundary for the client
frontend/app/layout.tsx - Discover · Landing page and dataset search entry point
frontend/app/page.tsx - Dataset audit · Primary per-dataset audit dashboard
frontend/app/dataset/[id]/page.tsx - Correlate · Cross-dataset comparison view
frontend/app/correlate/page.tsx - Stops map · Geographic visualization for coordinate-based datasets
frontend/app/components/StopsMap.tsx
Backend
FastAPI service that fetches, processes, analyzes, and stores dataset reports
- API app · Central backend application and route surface
backend/app/main.py - Edge entry · Alternate API entrypoint for platform-specific hosting
backend/api/index.py - Supabase client · Backend database and service initialization
backend/app/database.py - Audit stream · Dataset fetch, parse, clean, and progress streaming flow
backend/app/main.py - Analytics · Dataframe profiling, numeric selection, and anomaly detection
backend/app/main.py - AI synthesis · Narrative analysis and dataframe query translation
backend/app/main.py - Correlation · Audits two datasets and compares anomaly overlap
backend/app/main.py - Report store · Saved audit outputs with stats, anomalies, charts, and history
backend/app/main.py
External systems
Live data sources and persistence services
- Gov portals · Live civic datasets fetched on demand
- Supabase · Metadata catalog and persisted report storage
Connections 17
- Discover → API app: search / browse · Loads dataset catalog and discovery results
- Dataset audit → API app: stream audit · Opens SSE audit stream for a selected dataset
- Correlate → API app: compare datasets · Requests cross-dataset correlation analysis
- Dataset audit → Stops map: render map · Uses map visualization when coordinates exist
- API app → Audit stream: orchestrates · Routes audit requests into the processing pipeline
- Audit stream → Gov portals: fetch raw data · Retrieves live CSV, XLSX, or PDF sources
- Audit stream → Analytics: clean and profile · Transforms raw files into dataframe-backed metrics
- Analytics → AI synthesis: structured stats · Feeds anomalies and summaries into AI generation
- API app → AI synthesis: nl insights · Exposes dataset analysis and query translation
- API app → Correlation: correlate · Coordinates two audits and compares anomaly sets
- Correlation → Audit stream: reuse audit flow · Reuses the same fetch and analytics primitives
- API app → Supabase client: init access · Uses shared Supabase initialization
- Supabase client → Supabase: connect · Backend access to catalog and report data
- API app → Report store: save reports · Persists curated audit outputs
- Report store → Supabase: store artifacts · Saved reports live in Supabase
- API app → Edge entry: alternate entry · Supports deployment-specific API startup
- API app → Supabase: catalog lookup · Reads dataset metadata for discovery and search