# dilip-sheesh/bap architecture

> Architecture diagram of the GitHub repository dilip-sheesh/bap, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-05-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dilip-sheesh/bap
- Repository: https://github.com/dilip-sheesh/bap

## Overview

1. **Repository shape**
   - This is a two-tier application: a **Next.js 15 frontend** in `frontend/` and a **FastAPI backend** in `backend/`.
   - The backend does the data acquisition, cleaning, analytics, AI calls, and persistence. The frontend is a thin interactive shell for search, dataset audit, correlation, and maps.

2. **Frontend surface area**
   - The App Router is the main UI boundary:
     - `frontend/app/page.tsx` is the landing page and dataset discovery/search entry point.
     - `frontend/app/dataset/[id]/page.tsx` is the primary audit dashboard for one dataset.
     - `frontend/app/correlate/page.tsx` is the cross-dataset correlation view.
   - `frontend/app/components/StopsMap.tsx` is the geo-visualization component used when latitude/longitude data exists.
   - Styling and global layout live in `globals.css` and `layout.tsx`, with the rest of the UI driven from these route pages.

3. **Backend entry points**
   - `backend/app/main.py` is the central FastAPI application and likely contains the API routes described in the README.
   - `backend/api/index.py` exists as an alternate entry module, which usually indicates a serverless or platform-specific wrapper for deployment.
   - `backend/app/database.py` isolates database/client initialization so the API layer can access Supabase without hard-coding connection setup everywhere.

4. **Primary data sources**
   - The system is built around **live civic datasets** rather than a preloaded warehouse.
   - Source data is fetched on demand from open government portals and direct file links, then parsed as CSV/XLSX/PDF.
   - A Supabase catalog stores dataset metadata such as titles, descriptions, source URLs, tags, and headers; it is not the analytical store for the raw datasets.

5. **Core request flow for a dataset audit**
   - The browser opens an **SSE/EventSource** stream to the backend for a specific dataset.
   - The backend fetches the raw file, parses it into a dataframe, cleans and normalizes it, and runs analytics.
   - Progress and intermediate states are streamed back to the UI, then the final audit payload is delivered when processing completes.
   - The frontend uses that live stream to drive the dashboard, progress indicators, and terminal-style analysis display.

6. **Analytics boundary**
   - The analytical engine is dataframe-based, centered on **Pandas/NumPy** in the backend.
   - It chooses a useful numeric column, excludes noise columns such as IDs and coordinates, and runs anomaly detection.
   - That output feeds both charts and explanations, so the same backend analysis powers visualization, audit text, and report generation.

7. **AI subsystem**
   - Groq-hosted **Llama 3.1** is used for several distinct tasks:
     - natural-language dataset analysis,
     - natural-language-to-dataframe query translation,
     - cross-dataset correlation synthesis.
   - The AI is not the source of truth; it consumes the structured stats/anomalies produced by the analytics layer and turns them into human-readable insight.

8. **Cross-dataset correlation**
   - The correlation route takes two datasets, audits both, and compares their anomaly sets.
   - The meaningful output is the overlap between entities/locations/departments flagged as unusual in both sources.
   - This is a separate interaction path from single-dataset auditing, but it uses the same backend data-fetch and analytics primitives.

9. **Persistence boundary**
   - Supabase serves two roles:
     - a **catalog store** for dataset metadata,
     - a **report store** for saved audit outputs.
   - Saved reports include stats, anomaly flags, AI analysis, chart data, and query history, allowing a shareable audit result independent of the live fetch.
   - The architecture keeps raw source data transient while persisting only curated metadata and report artifacts.

10. **Geo and visualization layer**
   - When the backend or frontend detects coordinate fields, the UI can render a map using the Leaflet-based `StopsMap` component.
   - Charts are rendered in the frontend from backend-derived metric data, so visual selection follows the analytics model rather than arbitrary frontend logic.
   - The map and charts are presentation layers only; they do not own the data logic.

11. **Deployment and runtime boundaries**
   - The backend includes `Dockerfile`, `Procfile`, `railway.json`, and `vercel.json`, which suggests it is designed to run on multiple hosting targets, including containerized and serverless-style deployments.
   - The frontend is a standard Next.js app with its own build/runtime configuration in `next.config.ts`, `tsconfig.json`, and package tooling.
   - Runtime communication is HTTP-based, with SSE used where progressive updates matter.

12. **What to draw in an architecture diagram**
   - Show a **client tier** with three main views: discovery, dataset audit, and correlation.
   - Show a **FastAPI service** in the middle with endpoints for streaming audits, AI analysis, semantic search, correlation, seeding, and report storage.
   - Show two external data domains feeding the backend: **Supabase** for metadata/reports and **government open-data portals** for live raw datasets.
   - Emphasize that the raw dataset is fetched just-in-time, processed server-side, and then converted into streamed UI state plus AI-generated narrative.

Main entry points: [`backend/app/main.py`](https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py) (API app), [`frontend/app/dataset/[id]/page.tsx`](https://github.com/dilip-sheesh/bap/blob/main/frontend/app/dataset/[id]/page.tsx) (Dataset audit), [`backend/app/database.py`](https://github.com/dilip-sheesh/bap/blob/main/backend/app/database.py) (Supabase client), [`frontend/app/page.tsx`](https://github.com/dilip-sheesh/bap/blob/main/frontend/app/page.tsx) (Discover)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Frontend"]
  node_frontend_shell["App shell<br/>next.js app<br/>[layout.tsx]"]
  node_discovery_page["Discover<br/>route page<br/>[page.tsx]"]
  node_dataset_page["Dataset audit<br/>route page<br/>[page.tsx]"]
  node_correlate_page["Correlate<br/>route page<br/>[page.tsx]"]
  node_stops_map["Stops map<br/>leaflet map<br/>[StopsMap.tsx]"]
end

subgraph group_backend["Backend"]
  node_fastapi_main["API app<br/>fastapi service<br/>[main.py]"]
  node_api_wrapper["Edge entry<br/>deployment wrapper<br/>[index.py]"]
  node_db_client[("Supabase client<br/>client init<br/>[database.py]")]
  node_audit_stream["Audit stream<br/>sse pipeline<br/>[main.py]"]
  node_analytics_engine["Analytics<br/>pandas engine<br/>[main.py]"]
  node_ai_synthesis{{"AI synthesis<br/>groq llama<br/>[main.py]"}}
  node_correlation_service["Correlation<br/>compare pipeline<br/>[main.py]"]
  node_report_store[("Report store<br/>persistence layer<br/>[main.py]")]
end

subgraph group_external["External systems"]
  node_gov_sources["Gov portals<br/>open-data sources"]
  node_supabase[("Supabase<br/>catalog database")]
end

node_discovery_page -->|"search / browse"| node_fastapi_main
node_dataset_page -->|"stream audit"| node_fastapi_main
node_correlate_page -->|"compare datasets"| node_fastapi_main
node_dataset_page -->|"render map"| node_stops_map
node_fastapi_main -->|"orchestrates"| node_audit_stream
node_audit_stream -->|"fetch raw data"| node_gov_sources
node_audit_stream -->|"clean and profile"| node_analytics_engine
node_analytics_engine -->|"structured stats"| node_ai_synthesis
node_fastapi_main -->|"nl insights"| node_ai_synthesis
node_fastapi_main -->|"correlate"| node_correlation_service
node_correlation_service -->|"reuse audit flow"| node_audit_stream
node_fastapi_main -->|"init access"| node_db_client
node_db_client -->|"connect"| node_supabase
node_fastapi_main -->|"save reports"| node_report_store
node_report_store -->|"store artifacts"| node_supabase
node_fastapi_main -->|"alternate entry"| node_api_wrapper
node_fastapi_main -->|"catalog lookup"| node_supabase

click node_frontend_shell "https://github.com/dilip-sheesh/bap/blob/main/frontend/app/layout.tsx"
click node_discovery_page "https://github.com/dilip-sheesh/bap/blob/main/frontend/app/page.tsx"
click node_dataset_page "https://github.com/dilip-sheesh/bap/blob/main/frontend/app/dataset/[id]/page.tsx"
click node_correlate_page "https://github.com/dilip-sheesh/bap/blob/main/frontend/app/correlate/page.tsx"
click node_stops_map "https://github.com/dilip-sheesh/bap/blob/main/frontend/app/components/StopsMap.tsx"
click node_fastapi_main "https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py"
click node_api_wrapper "https://github.com/dilip-sheesh/bap/blob/main/backend/api/index.py"
click node_db_client "https://github.com/dilip-sheesh/bap/blob/main/backend/app/database.py"
click node_audit_stream "https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py"
click node_analytics_engine "https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py"
click node_ai_synthesis "https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py"
click node_correlation_service "https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py"
click node_report_store "https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend_shell,node_discovery_page,node_dataset_page,node_correlate_page,node_stops_map toneBlue
class node_fastapi_main,node_api_wrapper,node_db_client,node_audit_stream,node_analytics_engine,node_ai_synthesis,node_correlation_service,node_report_store toneAmber
class node_gov_sources,node_supabase toneMint
```

## Components

### Frontend

Next.js client tier with discovery, audit, and correlation views

- [App shell](https://github.com/dilip-sheesh/bap/blob/main/frontend/app/layout.tsx): next.js app · Global layout and route boundary for the client · `frontend/app/layout.tsx`
- [Discover](https://github.com/dilip-sheesh/bap/blob/main/frontend/app/page.tsx): route page · Landing page and dataset search entry point · `frontend/app/page.tsx`
- [Dataset audit](https://github.com/dilip-sheesh/bap/blob/main/frontend/app/dataset/[id]/page.tsx): route page · Primary per-dataset audit dashboard · `frontend/app/dataset/[id]/page.tsx`
- [Correlate](https://github.com/dilip-sheesh/bap/blob/main/frontend/app/correlate/page.tsx): route page · Cross-dataset comparison view · `frontend/app/correlate/page.tsx`
- [Stops map](https://github.com/dilip-sheesh/bap/blob/main/frontend/app/components/StopsMap.tsx): leaflet map · Geographic visualization for coordinate-based datasets · `frontend/app/components/StopsMap.tsx`

### Backend

FastAPI service that fetches, processes, analyzes, and stores dataset reports

- [API app](https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py): fastapi service · Central backend application and route surface · `backend/app/main.py`
- [Edge entry](https://github.com/dilip-sheesh/bap/blob/main/backend/api/index.py): deployment wrapper · Alternate API entrypoint for platform-specific hosting · `backend/api/index.py`
- [Supabase client](https://github.com/dilip-sheesh/bap/blob/main/backend/app/database.py): client init · Backend database and service initialization · `backend/app/database.py`
- [Audit stream](https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py): sse pipeline · Dataset fetch, parse, clean, and progress streaming flow · `backend/app/main.py`
- [Analytics](https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py): pandas engine · Dataframe profiling, numeric selection, and anomaly detection · `backend/app/main.py`
- [AI synthesis](https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py): groq llama · Narrative analysis and dataframe query translation · `backend/app/main.py`
- [Correlation](https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py): compare pipeline · Audits two datasets and compares anomaly overlap · `backend/app/main.py`
- [Report store](https://github.com/dilip-sheesh/bap/blob/main/backend/app/main.py): persistence layer · Saved audit outputs with stats, anomalies, charts, and history · `backend/app/main.py`

### External systems

Live data sources and persistence services

- **Gov portals**: open-data sources · Live civic datasets fetched on demand
- **Supabase**: catalog database · Metadata catalog and persisted report storage

## Connections

- Discover → API app: search / browse · Loads dataset catalog and discovery results
- Dataset audit → API app: stream audit · Opens SSE audit stream for a selected dataset
- Correlate → API app: compare datasets · Requests cross-dataset correlation analysis
- Dataset audit → Stops map: render map · Uses map visualization when coordinates exist
- API app → Audit stream: orchestrates · Routes audit requests into the processing pipeline
- Audit stream → Gov portals: fetch raw data · Retrieves live CSV, XLSX, or PDF sources
- Audit stream → Analytics: clean and profile · Transforms raw files into dataframe-backed metrics
- Analytics → AI synthesis: structured stats · Feeds anomalies and summaries into AI generation
- API app → AI synthesis: nl insights · Exposes dataset analysis and query translation
- API app → Correlation: correlate · Coordinates two audits and compares anomaly sets
- Correlation → Audit stream: reuse audit flow · Reuses the same fetch and analytics primitives
- API app → Supabase client: init access · Uses shared Supabase initialization
- Supabase client → Supabase: connect · Backend access to catalog and report data
- API app → Report store: save reports · Persists curated audit outputs
- Report store → Supabase: store artifacts · Saved reports live in Supabase
- API app → Edge entry: alternate entry · Supports deployment-specific API startup
- API app → Supabase: catalog lookup · Reads dataset metadata for discovery and search
