How sparshkapoor-coder/insight-engine works
1. Overall shape
- This is a single Flask application that turns a tabular file upload into a generated PDF report.
- The architecture is a linear document-generation pipeline rather than a distributed system: request in, file analysis, LLM planning, chart generation, narration, PDF assembly, report download.
2. Entry point and app bootstrap
- main.py starts the Flask server.
- config.py centralizes runtime configuration such as environment variables, file paths, row/chart limits, and model settings.
- The presence of Procfile and runtime.txt indicates it is designed to run on a PaaS environment like Render.
3. Presentation layer
- The frontend/ folder contains static HTML/CSS pages for upload/login/dashboard flows.
- This is a thin client: the browser handles file upload and report download interactions, while all meaningful processing happens server-side.
- There is no separate frontend build system; it is plain HTML, CSS, and likely a small amount of vanilla JavaScript embedded in the pages.
4. HTTP/API boundary
- api/routes.py is the main backend surface area, likely exposing endpoints for file upload and report retrieval.
- api/auth.py suggests an authentication or access-control seam exists, even though the README describes the app as effectively single-user/local in practice.
- This API layer is the boundary between the browser and the generation pipeline.
5. File ingestion and validation
- utils/file_handler.py handles upload validation and reads .csv / .xlsx inputs into pandas DataFrames.
- This is the first data boundary: raw user files are normalized into in-memory tabular data before any analysis begins.
- utils/data_cleaner.py then standardizes missing values, dtypes, and date parsing so downstream profiling and charting can work reliably.
6. Profiling and analysis planning
- core/profiler.py computes dataset statistics and structure summaries from pandas data.
- core/llm_analyst.py sends that profile to Groq-hosted LLaMA and receives a structured chart plan, rather than giving the model raw rows.
- This is a key architectural decision: the LLM is used for reasoning over metadata and summaries, not over the full dataset.
7. Chart execution layer
- core/chart_engine.py takes the LLM’s chart plan and generates actual chart artifacts with matplotlib and seaborn.
- The output is persisted as PNG files, up to the configured chart limit.
- This layer translates abstract LLM intent into deterministic analytics visuals.
8. Narration generation
- core/narrator.py makes a second LLM call per chart, using small aggregated tables or chart-specific data.
- Its job is to turn chart outputs into business-language insight text, typically 10–15 sentences per chart.
- The LLM still does not see raw source files or rendered images, which keeps token usage and leakage risk low.
9. Report assembly
- core/report_builder.py combines the cover page, executive summary, chart pages, and closing client summary into a PDF using ReportLab.
- The report is the final product boundary: charts and narrations are composed into a static document suitable for download/share.
- The README indicates a page-per-chart layout, so the report structure is strongly standardized.
10. Persistence, storage, and observability
- models/database.py suggests there is a lightweight persistence layer for report metadata or job state, though the repo is not organized as a heavy ORM-backed app.
- utils/logger.py generates per-report logs, and the README shows logs capturing each pipeline stage from upload through completion.
- The app likely uses local storage directories for uploads, generated charts, PDFs, and logs, so file system storage is part of the runtime architecture.
11. Execution model
- The whole pipeline appears to run synchronously within the request lifecycle or a single server-side job flow.
- There is no evidence of a queue/worker system, which means report generation latency is user-visible and depends on file size, chart count, and LLM response time.
12. Testing and reliability boundaries
- tests/test_pipeline.py, tests/test_profiler.py, and tests/test_llm_validator.py show the core pipeline is unit/integration tested around data profiling, pipeline behavior, and LLM output validation.
- That implies a strong emphasis on validating structured LLM responses before chart generation, which is important because downstream steps depend on the model producing usable JSON-like plans.
13. External dependencies that materially shape the design
- pandas/NumPy drive profiling and data manipulation.
- matplotlib/seaborn generate visual artifacts.
- Groq API provides the reasoning model.
- ReportLab produces the final PDF.
- Flask is the web/runtime shell around the whole workflow.
14. Architectural diagram takeaway
- The clean diagram should show: browser/UI → Flask API → file validation/cleaning → profiling → LLM chart planning → chart engine → per-chart narration LLM → PDF builder → downloadable report, with logging/storage alongside the pipeline and optional auth/database at the API boundary.
Main entry points: routes.py Routes · report_builder.py PDF builder · config.py Config · chart_engine.py Chart engine
Components 16
UI
Browser-facing pages and entrypoint
- Frontend · Upload, login, and dashboard pages for user interaction
frontend/index.html
API
Flask boundary and request handling
Pipeline
Tabular analysis, LLM planning, charting, narration, and PDF assembly
- File handler · Validates uploads and reads CSV/XLSX into DataFrames
utils/file_handler.py - Cleaner · Normalizes missing values, dtypes, and date parsing
utils/data_cleaner.py - Profiler · Computes structural summaries and statistics from tabular data
core/profiler.py - LLM planner · Sends dataset profiles to Groq LLaMA and returns a structured chart plan
core/llm_analyst.py - Chart engine · Turns chart plans into matplotlib/seaborn PNG charts
core/chart_engine.py - Narrator · Makes per-chart LLM calls to generate business-language commentary
core/narrator.py - PDF builder · Composes cover, summary, chart pages, and closing content into a PDF
core/report_builder.py
Platform
Runtime configuration, storage, logging, and persistence
- Config · Centralizes environment, limits, file paths, and model settings
config.py - Database · Lightweight storage for report metadata or job state
models/database.py - Logger · Per-report logs spanning upload through completion
utils/logger.py - Tests · Pipeline and profiler tests with LLM output validation
tests/test_pipeline.py - Deploy files · PaaS runtime and process settings for deployment
Procfile
Connections 23
- Frontend → Routes: submit file · Browser uploads and report requests flow into the Flask API
- main.py → Routes: serve · Server startup wires routes into the Flask app
- Routes → Auth: check access · Requests may pass through authentication or authorization checks
- Routes → File handler: ingest · Uploaded files are validated and loaded into tabular form
- File handler → Cleaner: normalize · Raw tables are cleaned before analysis
- Cleaner → Profiler: profile input · Cleaned data is summarized for downstream reasoning
- Profiler → LLM planner: summaries · Dataset profile is sent to the LLM for chart planning
- LLM planner → Chart engine: chart plan · Structured plan drives deterministic chart generation
- Chart engine → Narrator: chart data · Generated chart outputs feed per-chart narration
- Narrator → PDF builder: insights · Narratives are assembled into the final report
- Chart engine → PDF builder: images · Chart artifacts are embedded into the PDF
- PDF builder → Routes: download · Completed PDF is returned or exposed for retrieval
- Config → Routes: settings · API behavior depends on configured paths and limits
- Config → LLM planner: model config · LLM provider and prompt settings come from configuration
- Config → Chart engine: limits · Chart count and output settings constrain generation
- Config → PDF builder: paths · Report output locations and runtime settings are centralized
- Logger → Routes: trace · Request and pipeline stages are logged
- Logger → PDF builder: trace · Report assembly is logged as part of the job flow
- Database → Routes: state · Optional persistence for report metadata or job state
- Tests → Profiler: verify · Tests cover profiling behavior and data handling
- Tests → LLM planner: validate · Tests validate structured LLM output before charting
- Tests → PDF builder: verify · Tests exercise pipeline assembly and report generation
- Deploy files → main.py: run app · PaaS process configuration launches the Flask entrypoint