# nisheshjain12/report-data-extraction-evaluation architecture
> Architecture diagram of the GitHub repository nisheshjain12/report-data-extraction-evaluation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-21). It is an AI-made overview: check details against the source.
- Interactive diagram: https://gitdiagram.com/nisheshjain12/report-data-extraction-evaluation
- Repository: https://github.com/nisheshjain12/report-data-extraction-evaluation
## Overview
1. **This is a file-based data pipeline, not a service backend.**
The repository centers on extracting a small set of financial values from a fixed corpus of 10 company 10-K PDFs, scoring the outputs against a hand-labeled CSV ground truth, and visualizing the results. There is no database or persistent server component; the key integration point is the filesystem.
2. **Primary data inputs are local and curated.**
The source documents live in `data/pdfs/` as 10 annual report PDFs, and the answer key lives in `data/ground_truth.csv`. These are the authoritative inputs for both extraction and evaluation.
3. **`src/pdf_utils.py` is the document ingestion boundary.**
This module isolates PDF text extraction from the rest of the pipeline. The README indicates the PDFs are flattened into text before being sent to the model, so this step is where the document-format boundary is crossed.
4. **`src/llm.py` is the only external AI dependency.**
All model calls go through one Gemini client module. The extractor does not talk to the API directly elsewhere, which makes the LLM interaction a single architectural boundary. The project depends on Google Gemini, with model choice controlled centrally in config.
5. **`src/extract.py` is the main orchestration layer for extraction.**
It ties together PDF text extraction, prompt construction, Gemini inference, and output serialization. It supports two extractor versions:
- **v1:** naive prompt over flattened text
- **v2:** refined prompt with explicit units and field definitions
This module is the place where prompt design changes drive different extraction outcomes.
6. **The extracted data is normalized into CSV artifacts under `results/`.**
Extraction runs write versioned outputs such as `results/extraction_v1.csv` and `results/extraction_v2.csv`. These CSVs are the handoff between model inference and scoring, and they are also the data source for the dashboard.
7. **`src/evaluate.py` is the scoring and error-analysis boundary.**
It compares predicted values against `data/ground_truth.csv`, applies tolerance-based accuracy, and produces `results/metrics.csv` plus `results/scored.csv`. The presence of an error-category breakdown means evaluation is not just pass/fail; it also classifies failure modes for iteration.
8. **The architecture has a clear feedback loop from evaluation back to extraction design.**
The README describes an error-analysis-driven iteration: v1 exposed unit mismatches and field-definition ambiguity, and v2 fixed those issues in the prompt. So evaluation informs prompt refinement, but that loop is part of development rather than runtime execution.
9. **`src/config.py` centralizes project-wide policy.**
Model ID, company list, field definitions, accuracy tolerance, and paths are all defined here. That keeps the extraction and evaluation modules aligned on the same schema and operating parameters.
10. **`dashboard/app.py` is a read-only presentation layer.**
The Streamlit dashboard consumes the generated CSV outputs and visualizes extracted values plus metrics. It does not perform extraction or scoring itself, so it sits entirely downstream of the pipeline.
11. **Operationally, the pipeline is constrained by API quota and model choice.**
Gemini free-tier limits affect execution cadence, and the README notes model switching between v1 and v2 due to quota. This matters architecturally because the model is an external dependency with per-model request limits, influencing how extraction runs are scheduled.
12. **The repo’s overall shape is: documents in, model inference, scored outputs, visual inspection.**
A good diagram would show four main layers: local PDF/ground-truth assets, extraction orchestration with Gemini, evaluation/scoring, and Streamlit visualization. The results directory is the durable exchange point between those layers.
Main entry points: [`src/extract.py`](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/extract.py) (Extractor), [`src/evaluate.py`](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/evaluate.py) (Evaluator), [`dashboard/app.py`](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/dashboard/app.py) (Dashboard), [`results/extraction_v1.csv`](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/results/extraction_v1.csv) (Extraction v1)
## Diagram (Mermaid)
```mermaid
flowchart TD
subgraph group_assets["Local Assets"]
node_data_pdfs["10-K PDFs
corpus"]
node_ground_truth["Ground Truth
labels
[ground_truth.csv]"]
end
subgraph group_core["Pipeline Core"]
node_config["Config
policy
[config.py]"]
node_pdf_utils["PDF Utils
ingestion
[pdf_utils.py]"]
node_llm["Gemini Client
model adapter
[llm.py]"]
node_extract["Extractor
orchestrator
[extract.py]"]
node_evaluate["Evaluator
scoring
[evaluate.py]"]
node_gemini_api(("Gemini API
external service"))
node_readme["README
docs
[README.md]"]
end
subgraph group_outputs["Results"]
node_results_v1["Extraction v1
csv artifact
[extraction_v1.csv]"]
node_results_v2["Extraction v2
csv artifact
[extraction_v2.csv]"]
node_metrics["Metrics
csv artifact
[metrics.csv]"]
node_scored["Scored Rows
csv artifact
[scored.csv]"]
end
subgraph group_ui["Dashboard"]
node_dashboard["Dashboard
ui
[app.py]"]
end
node_data_pdfs -->|"source docs"| node_pdf_utils
node_config -.->|"paths"| node_pdf_utils
node_pdf_utils -->|"text"| node_extract
node_config -.->|"schema"| node_extract
node_extract -->|"prompt"| node_llm
node_llm -->|"request"| node_gemini_api
node_extract -->|"write"| node_results_v1
node_extract -->|"write"| node_results_v2
node_ground_truth -->|"reference"| node_evaluate
node_results_v1 -->|"predict"| node_evaluate
node_results_v2 -->|"predict"| node_evaluate
node_evaluate -->|"write"| node_metrics
node_evaluate -->|"write"| node_scored
node_results_v1 -->|"visualize"| node_dashboard
node_results_v2 -->|"visualize"| node_dashboard
node_metrics -->|"visualize"| node_dashboard
node_scored -->|"visualize"| node_dashboard
node_readme -.->|"describes"| node_extract
node_readme -.->|"describes"| node_evaluate
click node_data_pdfs "https://github.com/nisheshjain12/report-data-extraction-evaluation/tree/main/data/pdfs"
click node_ground_truth "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/data/ground_truth.csv"
click node_config "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/config.py"
click node_pdf_utils "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/pdf_utils.py"
click node_llm "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/llm.py"
click node_extract "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/extract.py"
click node_evaluate "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/evaluate.py"
click node_results_v1 "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/results/extraction_v1.csv"
click node_results_v2 "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/results/extraction_v2.csv"
click node_metrics "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/results/metrics.csv"
click node_scored "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/results/scored.csv"
click node_dashboard "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/dashboard/app.py"
click node_readme "https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/README.md"
classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_data_pdfs,node_ground_truth toneBlue
class node_config,node_pdf_utils,node_llm,node_extract,node_evaluate,node_gemini_api,node_readme toneAmber
class node_results_v1,node_results_v2,node_metrics,node_scored toneMint
class node_dashboard toneRose
```
## Components
### Local Assets
Curated input documents and ground truth used by the pipeline.
- [10-K PDFs](https://github.com/nisheshjain12/report-data-extraction-evaluation/tree/main/data/pdfs): corpus · Fixed set of company annual report PDFs used as source documents. · `data/pdfs`
- [Ground Truth](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/data/ground_truth.csv): labels · Hand-labeled CSV answer key for evaluation. · `data/ground_truth.csv`
### Pipeline Core
Extraction, model access, configuration, and evaluation logic.
- [Config](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/config.py): policy · Central settings for model choice, fields, company list, tolerance, and paths. · `src/config.py`
- [PDF Utils](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/pdf_utils.py): ingestion · Boundary for extracting text from PDFs before model input. · `src/pdf_utils.py`
- [Gemini Client](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/llm.py): model adapter · Single external AI integration for all model calls. · `src/llm.py`
- [Extractor](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/extract.py): orchestrator · Coordinates PDF text loading, prompt construction, Gemini inference, and CSV output. · `src/extract.py`
- [Evaluator](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/src/evaluate.py): scoring · Compares predictions to ground truth, applies tolerance, and classifies errors. · `src/evaluate.py`
- **Gemini API**: external service · External model endpoint used for extraction.
- [README](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/README.md): docs · Project narrative describing the pipeline, iteration loop, and usage. · `README.md`
### Results
Versioned CSV artifacts produced by extraction and scoring.
- [Extraction v1](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/results/extraction_v1.csv): csv artifact · Versioned extraction output from the initial prompt. · `results/extraction_v1.csv`
- [Extraction v2](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/results/extraction_v2.csv): csv artifact · Versioned extraction output from the refined prompt. · `results/extraction_v2.csv`
- [Metrics](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/results/metrics.csv): csv artifact · Aggregate scoring summary produced by evaluation. · `results/metrics.csv`
- [Scored Rows](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/results/scored.csv): csv artifact · Row-level evaluation output with errors and classifications. · `results/scored.csv`
### Dashboard
Read-only visualization layer over generated outputs.
- [Dashboard](https://github.com/nisheshjain12/report-data-extraction-evaluation/blob/main/dashboard/app.py): ui · Streamlit presentation layer for inspecting extracted values and metrics. · `dashboard/app.py`
## Connections
- 10-K PDFs → PDF Utils: source docs · PDFs are ingested and flattened to text for downstream extraction.
- Config → PDF Utils: paths · Shared file locations and operating parameters influence ingestion.
- PDF Utils → Extractor: text · Extracted text becomes the input payload for prompting.
- Config → Extractor: schema · Extraction uses centralized fields, company list, and model settings.
- Extractor → Gemini Client: prompt · Extractor sends the prompt and flattened text to the model adapter.
- Gemini Client → Gemini API: request · All model calls flow through the Gemini client boundary.
- Extractor → Extraction v1: write · v1 extraction run writes versioned CSV output.
- Extractor → Extraction v2: write · v2 extraction run writes versioned CSV output.
- Ground Truth → Evaluator: reference · Ground truth provides the comparison baseline for scoring.
- Extraction v1 → Evaluator: predict · Evaluation consumes extracted rows for scoring.
- Extraction v2 → Evaluator: predict · Evaluation consumes extracted rows for scoring.
- Evaluator → Metrics: write · Aggregate accuracy and summary metrics are persisted.
- Evaluator → Scored Rows: write · Row-level scored output captures failures and categories.
- Extraction v1 → Dashboard: visualize · Dashboard can display the generated extraction output.
- Extraction v2 → Dashboard: visualize · Dashboard can display the generated extraction output.
- Metrics → Dashboard: visualize · Dashboard reads metrics for reporting and comparison.
- Scored Rows → Dashboard: visualize · Dashboard reads scored rows for inspection.
- README → Extractor: describes · Documentation explains the extraction workflow and iterations.
- README → Evaluator: describes · Documentation explains scoring and error analysis.