nisheshjain12/report-data-extraction-evaluation

1. This is a file-based data pipeline, not a service backend.

The repository centers on extracting a small set of financial values from a fixed corpus of 10 company 10-K PDFs, scoring the outputs against a hand-labeled CSV ground truth, and visualizing the results. There is no database or persistent server component; the key integration point is the filesystem.

2. Primary data inputs are local and curated.

The source documents live in data/pdfs/ as 10 annual report PDFs, and the answer key lives in data/ground_truth.csv. These are the authoritative inputs for both extraction and evaluation.

3. src/pdf_utils.py is the document ingestion boundary.

This module isolates PDF text extraction from the rest of the pipeline. The README indicates the PDFs are flattened into text before being sent to the model, so this step is where the document-format boundary is crossed.

4. src/llm.py is the only external AI dependency.

All model calls go through one Gemini client module. The extractor does not talk to the API directly elsewhere, which makes the LLM interaction a single architectural boundary. The project depends on Google Gemini, with model choice controlled centrally in config.

5. src/extract.py is the main orchestration layer for extraction.

It ties together PDF text extraction, prompt construction, Gemini inference, and output serialization. It supports two extractor versions:

- v1: naive prompt over flattened text

- v2: refined prompt with explicit units and field definitions

This module is the place where prompt design changes drive different extraction outcomes.

6. The extracted data is normalized into CSV artifacts under results/.

Extraction runs write versioned outputs such as results/extraction_v1.csv and results/extraction_v2.csv. These CSVs are the handoff between model inference and scoring, and they are also the data source for the dashboard.

7. src/evaluate.py is the scoring and error-analysis boundary.

It compares predicted values against data/ground_truth.csv, applies tolerance-based accuracy, and produces results/metrics.csv plus results/scored.csv. The presence of an error-category breakdown means evaluation is not just pass/fail; it also classifies failure modes for iteration.

8. The architecture has a clear feedback loop from evaluation back to extraction design.

The README describes an error-analysis-driven iteration: v1 exposed unit mismatches and field-definition ambiguity, and v2 fixed those issues in the prompt. So evaluation informs prompt refinement, but that loop is part of development rather than runtime execution.

9. src/config.py centralizes project-wide policy.

Model ID, company list, field definitions, accuracy tolerance, and paths are all defined here. That keeps the extraction and evaluation modules aligned on the same schema and operating parameters.

10. dashboard/app.py is a read-only presentation layer.

The Streamlit dashboard consumes the generated CSV outputs and visualizes extracted values plus metrics. It does not perform extraction or scoring itself, so it sits entirely downstream of the pipeline.

11. Operationally, the pipeline is constrained by API quota and model choice.

Gemini free-tier limits affect execution cadence, and the README notes model switching between v1 and v2 due to quota. This matters architecturally because the model is an external dependency with per-model request limits, influencing how extraction runs are scheduled.

12. The repo’s overall shape is: documents in, model inference, scored outputs, visual inspection.

A good diagram would show four main layers: local PDF/ground-truth assets, extraction orchestration with Gemini, evaluation/scoring, and Streamlit visualization. The results directory is the durable exchange point between those layers.