# sparshkapoor-coder/insight-engine architecture

> Architecture diagram of the GitHub repository sparshkapoor-coder/insight-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sparshkapoor-coder/insight-engine
- Repository: https://github.com/sparshkapoor-coder/insight-engine

## Overview

1. **Overall shape**
   - This is a single Flask application that turns a tabular file upload into a generated PDF report.
   - The architecture is a linear document-generation pipeline rather than a distributed system: request in, file analysis, LLM planning, chart generation, narration, PDF assembly, report download.

2. **Entry point and app bootstrap**
   - `main.py` starts the Flask server.
   - `config.py` centralizes runtime configuration such as environment variables, file paths, row/chart limits, and model settings.
   - The presence of `Procfile` and `runtime.txt` indicates it is designed to run on a PaaS environment like Render.

3. **Presentation layer**
   - The `frontend/` folder contains static HTML/CSS pages for upload/login/dashboard flows.
   - This is a thin client: the browser handles file upload and report download interactions, while all meaningful processing happens server-side.
   - There is no separate frontend build system; it is plain HTML, CSS, and likely a small amount of vanilla JavaScript embedded in the pages.

4. **HTTP/API boundary**
   - `api/routes.py` is the main backend surface area, likely exposing endpoints for file upload and report retrieval.
   - `api/auth.py` suggests an authentication or access-control seam exists, even though the README describes the app as effectively single-user/local in practice.
   - This API layer is the boundary between the browser and the generation pipeline.

5. **File ingestion and validation**
   - `utils/file_handler.py` handles upload validation and reads `.csv` / `.xlsx` inputs into pandas DataFrames.
   - This is the first data boundary: raw user files are normalized into in-memory tabular data before any analysis begins.
   - `utils/data_cleaner.py` then standardizes missing values, dtypes, and date parsing so downstream profiling and charting can work reliably.

6. **Profiling and analysis planning**
   - `core/profiler.py` computes dataset statistics and structure summaries from pandas data.
   - `core/llm_analyst.py` sends that profile to Groq-hosted LLaMA and receives a structured chart plan, rather than giving the model raw rows.
   - This is a key architectural decision: the LLM is used for reasoning over metadata and summaries, not over the full dataset.

7. **Chart execution layer**
   - `core/chart_engine.py` takes the LLM’s chart plan and generates actual chart artifacts with matplotlib and seaborn.
   - The output is persisted as PNG files, up to the configured chart limit.
   - This layer translates abstract LLM intent into deterministic analytics visuals.

8. **Narration generation**
   - `core/narrator.py` makes a second LLM call per chart, using small aggregated tables or chart-specific data.
   - Its job is to turn chart outputs into business-language insight text, typically 10–15 sentences per chart.
   - The LLM still does not see raw source files or rendered images, which keeps token usage and leakage risk low.

9. **Report assembly**
   - `core/report_builder.py` combines the cover page, executive summary, chart pages, and closing client summary into a PDF using ReportLab.
   - The report is the final product boundary: charts and narrations are composed into a static document suitable for download/share.
   - The README indicates a page-per-chart layout, so the report structure is strongly standardized.

10. **Persistence, storage, and observability**
   - `models/database.py` suggests there is a lightweight persistence layer for report metadata or job state, though the repo is not organized as a heavy ORM-backed app.
   - `utils/logger.py` generates per-report logs, and the README shows logs capturing each pipeline stage from upload through completion.
   - The app likely uses local storage directories for uploads, generated charts, PDFs, and logs, so file system storage is part of the runtime architecture.

11. **Execution model**
   - The whole pipeline appears to run synchronously within the request lifecycle or a single server-side job flow.
   - There is no evidence of a queue/worker system, which means report generation latency is user-visible and depends on file size, chart count, and LLM response time.

12. **Testing and reliability boundaries**
   - `tests/test_pipeline.py`, `tests/test_profiler.py`, and `tests/test_llm_validator.py` show the core pipeline is unit/integration tested around data profiling, pipeline behavior, and LLM output validation.
   - That implies a strong emphasis on validating structured LLM responses before chart generation, which is important because downstream steps depend on the model producing usable JSON-like plans.

13. **External dependencies that materially shape the design**
   - pandas/NumPy drive profiling and data manipulation.
   - matplotlib/seaborn generate visual artifacts.
   - Groq API provides the reasoning model.
   - ReportLab produces the final PDF.
   - Flask is the web/runtime shell around the whole workflow.

14. **Architectural diagram takeaway**
   - The clean diagram should show: browser/UI → Flask API → file validation/cleaning → profiling → LLM chart planning → chart engine → per-chart narration LLM → PDF builder → downloadable report, with logging/storage alongside the pipeline and optional auth/database at the API boundary.

Main entry points: [`api/routes.py`](https://github.com/sparshkapoor-coder/insight-engine/blob/main/api/routes.py) (Routes), [`core/report_builder.py`](https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/report_builder.py) (PDF builder), [`config.py`](https://github.com/sparshkapoor-coder/insight-engine/blob/main/config.py) (Config), [`core/chart_engine.py`](https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/chart_engine.py) (Chart engine)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_group_ui["UI"]
  node_frontend_pages["Frontend<br/>static HTML<br/>[index.html]"]
end

subgraph group_group_api["API"]
  node_main_py["main.py<br/>flask entrypoint<br/>[main.py]"]
  node_api_routes["Routes<br/>HTTP API<br/>[routes.py]"]
  node_api_auth["Auth<br/>access control<br/>[auth.py]"]
end

subgraph group_group_pipeline["Pipeline"]
  node_file_handler["File handler<br/>ingestion<br/>[file_handler.py]"]
  node_data_cleaner["Cleaner<br/>prep<br/>[data_cleaner.py]"]
  node_profiler["Profiler<br/>dataset profiling<br/>[profiler.py]"]
  node_llm_analyst["LLM planner<br/>chart planning<br/>[llm_analyst.py]"]
  node_chart_engine["Chart engine<br/>visual generation<br/>[chart_engine.py]"]
  node_narrator["Narrator<br/>insight generation<br/>[narrator.py]"]
  node_report_builder["PDF builder<br/>report assembly<br/>[report_builder.py]"]
end

subgraph group_group_platform["Platform"]
  node_config_py["Config<br/>[config.py]"]
  node_database[("Database<br/>persistence<br/>[database.py]")]
  node_logger["Logger<br/>observability<br/>[logger.py]"]
  node_tests_pipeline["Tests<br/>validation<br/>[test_pipeline.py]"]
  node_deployment_files["Deploy files<br/>platform config"]
end

node_frontend_pages -->|"submit file"| node_api_routes
node_main_py -->|"serve"| node_api_routes
node_api_routes -->|"check access"| node_api_auth
node_api_routes -->|"ingest"| node_file_handler
node_file_handler -->|"normalize"| node_data_cleaner
node_data_cleaner -->|"profile input"| node_profiler
node_profiler -->|"summaries"| node_llm_analyst
node_llm_analyst -->|"chart plan"| node_chart_engine
node_chart_engine -->|"chart data"| node_narrator
node_narrator -->|"insights"| node_report_builder
node_chart_engine -->|"images"| node_report_builder
node_report_builder -->|"download"| node_api_routes
node_config_py -->|"settings"| node_api_routes
node_config_py -->|"model config"| node_llm_analyst
node_config_py -->|"limits"| node_chart_engine
node_config_py -->|"paths"| node_report_builder
node_logger -->|"trace"| node_api_routes
node_logger -->|"trace"| node_report_builder
node_database -->|"state"| node_api_routes
node_tests_pipeline -.->|"verify"| node_profiler
node_tests_pipeline -.->|"validate"| node_llm_analyst
node_tests_pipeline -.->|"verify"| node_report_builder
node_deployment_files -->|"run app"| node_main_py

click node_main_py "https://github.com/sparshkapoor-coder/insight-engine/blob/main/main.py"
click node_frontend_pages "https://github.com/sparshkapoor-coder/insight-engine/blob/main/frontend/index.html"
click node_api_routes "https://github.com/sparshkapoor-coder/insight-engine/blob/main/api/routes.py"
click node_api_auth "https://github.com/sparshkapoor-coder/insight-engine/blob/main/api/auth.py"
click node_file_handler "https://github.com/sparshkapoor-coder/insight-engine/blob/main/utils/file_handler.py"
click node_data_cleaner "https://github.com/sparshkapoor-coder/insight-engine/blob/main/utils/data_cleaner.py"
click node_profiler "https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/profiler.py"
click node_llm_analyst "https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/llm_analyst.py"
click node_chart_engine "https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/chart_engine.py"
click node_narrator "https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/narrator.py"
click node_report_builder "https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/report_builder.py"
click node_config_py "https://github.com/sparshkapoor-coder/insight-engine/blob/main/config.py"
click node_database "https://github.com/sparshkapoor-coder/insight-engine/blob/main/models/database.py"
click node_logger "https://github.com/sparshkapoor-coder/insight-engine/blob/main/utils/logger.py"
click node_tests_pipeline "https://github.com/sparshkapoor-coder/insight-engine/blob/main/tests/test_pipeline.py"
click node_deployment_files "https://github.com/sparshkapoor-coder/insight-engine/tree/main/Procfile"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_frontend_pages toneBlue
class node_main_py,node_api_routes,node_api_auth toneAmber
class node_file_handler,node_data_cleaner,node_profiler,node_llm_analyst,node_chart_engine,node_narrator,node_report_builder toneMint
class node_config_py,node_database,node_logger,node_tests_pipeline,node_deployment_files toneRose
```

## Components

### UI

Browser-facing pages and entrypoint

- [Frontend](https://github.com/sparshkapoor-coder/insight-engine/blob/main/frontend/index.html): static HTML · Upload, login, and dashboard pages for user interaction · `frontend/index.html`

### API

Flask boundary and request handling

- [main.py](https://github.com/sparshkapoor-coder/insight-engine/blob/main/main.py): flask entrypoint · Starts the Flask server and boots the app · `main.py`
- [Routes](https://github.com/sparshkapoor-coder/insight-engine/blob/main/api/routes.py): HTTP API · Main request surface for upload and report actions · `api/routes.py`
- [Auth](https://github.com/sparshkapoor-coder/insight-engine/blob/main/api/auth.py): access control · Authentication and access-control seam at the API boundary · `api/auth.py`

### Pipeline

Tabular analysis, LLM planning, charting, narration, and PDF assembly

- [File handler](https://github.com/sparshkapoor-coder/insight-engine/blob/main/utils/file_handler.py): ingestion · Validates uploads and reads CSV/XLSX into DataFrames · `utils/file_handler.py`
- [Cleaner](https://github.com/sparshkapoor-coder/insight-engine/blob/main/utils/data_cleaner.py): prep · Normalizes missing values, dtypes, and date parsing · `utils/data_cleaner.py`
- [Profiler](https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/profiler.py): dataset profiling · Computes structural summaries and statistics from tabular data · `core/profiler.py`
- [LLM planner](https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/llm_analyst.py): chart planning · Sends dataset profiles to Groq LLaMA and returns a structured chart plan · `core/llm_analyst.py`
- [Chart engine](https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/chart_engine.py): visual generation · Turns chart plans into matplotlib/seaborn PNG charts · `core/chart_engine.py`
- [Narrator](https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/narrator.py): insight generation · Makes per-chart LLM calls to generate business-language commentary · `core/narrator.py`
- [PDF builder](https://github.com/sparshkapoor-coder/insight-engine/blob/main/core/report_builder.py): report assembly · Composes cover, summary, chart pages, and closing content into a PDF · `core/report_builder.py`

### Platform

Runtime configuration, storage, logging, and persistence

- [Config](https://github.com/sparshkapoor-coder/insight-engine/blob/main/config.py): runtime config · Centralizes environment, limits, file paths, and model settings · `config.py`
- [Database](https://github.com/sparshkapoor-coder/insight-engine/blob/main/models/database.py): persistence · Lightweight storage for report metadata or job state · `models/database.py`
- [Logger](https://github.com/sparshkapoor-coder/insight-engine/blob/main/utils/logger.py): observability · Per-report logs spanning upload through completion · `utils/logger.py`
- [Tests](https://github.com/sparshkapoor-coder/insight-engine/blob/main/tests/test_pipeline.py): validation · Pipeline and profiler tests with LLM output validation · `tests/test_pipeline.py`
- [Deploy files](https://github.com/sparshkapoor-coder/insight-engine/tree/main/Procfile): platform config · PaaS runtime and process settings for deployment · `Procfile`

## Connections

- Frontend → Routes: submit file · Browser uploads and report requests flow into the Flask API
- main.py → Routes: serve · Server startup wires routes into the Flask app
- Routes → Auth: check access · Requests may pass through authentication or authorization checks
- Routes → File handler: ingest · Uploaded files are validated and loaded into tabular form
- File handler → Cleaner: normalize · Raw tables are cleaned before analysis
- Cleaner → Profiler: profile input · Cleaned data is summarized for downstream reasoning
- Profiler → LLM planner: summaries · Dataset profile is sent to the LLM for chart planning
- LLM planner → Chart engine: chart plan · Structured plan drives deterministic chart generation
- Chart engine → Narrator: chart data · Generated chart outputs feed per-chart narration
- Narrator → PDF builder: insights · Narratives are assembled into the final report
- Chart engine → PDF builder: images · Chart artifacts are embedded into the PDF
- PDF builder → Routes: download · Completed PDF is returned or exposed for retrieval
- Config → Routes: settings · API behavior depends on configured paths and limits
- Config → LLM planner: model config · LLM provider and prompt settings come from configuration
- Config → Chart engine: limits · Chart count and output settings constrain generation
- Config → PDF builder: paths · Report output locations and runtime settings are centralized
- Logger → Routes: trace · Request and pipeline stages are logged
- Logger → PDF builder: trace · Report assembly is logged as part of the job flow
- Database → Routes: state · Optional persistence for report metadata or job state
- Tests → Profiler: verify · Tests cover profiling behavior and data handling
- Tests → LLM planner: validate · Tests validate structured LLM output before charting
- Tests → PDF builder: verify · Tests exercise pipeline assembly and report generation
- Deploy files → main.py: run app · PaaS process configuration launches the Flask entrypoint
