# mprabhu26/lni-reference-checker architecture

> Architecture diagram of the GitHub repository mprabhu26/lni-reference-checker, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mprabhu26/lni-reference-checker
- Repository: https://github.com/mprabhu26/lni-reference-checker

## Overview

1. **Overall shape**
   This is a small Python web application with a **Flask backend** (`app.py`) and a **single-page browser UI** (`static/index.html`). The backend does the document processing; the frontend is mainly a thin presentation layer for uploads and results.

2. **User-facing flow**
   The main input boundary is a submitted academic manuscript in **PDF, DOCX, or LaTeX** form, optionally with a separate **`.bib`** file for LaTeX. The user toggles whether to run online verification, then the server returns structured findings for bibliography completeness, citation cross-checking, and reference verification.

3. **Format-specific extraction**
   `extractor.py` is the first processing stage. It normalizes the different source formats into text:
   - PDF and DOCX are text-extracted and split around the bibliography section.
   - LaTeX is cleaned of markup, and the `.bib` sidecar can be parsed directly.
   This is the boundary between raw file handling and document understanding.

4. **Bibliography parsing**
   `parser.py` turns the extracted bibliography into structured entries keyed by citation labels like `[Key]`. This is where the app converts human-readable reference blocks into machine-checkable records with fields such as author, title, year, and venue.

5. **Core checking engine**
   `checker.py` appears to be the main analysis layer. It compares:
   - in-text citations vs. bibliography entries,
   - bibliography completeness vs. expected fields,
   - cited references vs. uncited bibliography items.
   This is the subsystem that produces the “bibliography” and “cross-check” findings shown in the UI.

6. **Online verification boundary**
   `web_search_verifier.py` adds an external-service layer. Per the README, it queries **CrossRef**, **Semantic Scholar**, and **Unpaywall** to assess whether a reference appears real and whether an open-access full text is available. This is an optional enrichment step, not required for offline checking.

7. **Auxiliary AI/scoring path**
   `ai_checker.py` suggests an additional heuristic or AI-assisted verification path alongside the web lookups. Architecturally, this is a secondary confidence/scoring component rather than the primary source of truth.

8. **State, caching, and manual review**
   The repo includes `local_db.py`, `review_queue.py`, `download_db.py`, `.lni_cache/`, and `.lni_db/`. Together these indicate a local persistence layer for:
   - caching verification results,
   - storing verified papers,
   - queuing ambiguous items for review.
   The `.db` files are SQLite databases, so this is a local-first architecture rather than a remote backend service.

9. **Presentation of results**
   The frontend is a static HTML app that likely renders three conceptual result views described in the README:
   - bibliography quality/completeness,
   - citation cross-check status,
   - verification outcome for each reference.
   The UI is not a separate API consumer in another service; it is served by the Flask app and talks to the local backend.

10. **Deployment and runtime**
   `requirements.txt` defines the Python runtime dependencies, and `Procfile` suggests the app can be launched in a platform-friendly process model. The architecture is therefore “single-process web app + local storage + external verification APIs,” not a microservice system.

11. **Testing and regression safety**
   The `tests/` folder, `pytest.ini`, and fixtures indicate an emphasis on deterministic checking. `test_cache.py` and `test_regression_full.py` imply the system is sensitive to parser/checker correctness and cache behavior, which matters because reference validation is rule-heavy and easy to regress.

12. **Diagram-worthy boundary summary**
   A clear architecture diagram should show:
   - **Browser UI**
   - **Flask app/controller**
   - **Document extraction and parsing pipeline**
   - **Reference/citation analysis engine**
   - **External reference-verification services**
   - **Local SQLite/cache persistence**
   - **Manual review queue**
   These are the main data-flow and trust boundaries in the repository.

Main entry points: [`local_db.py`](https://github.com/mprabhu26/lni-reference-checker/blob/main/local_db.py) (Local DB), [`checker.py`](https://github.com/mprabhu26/lni-reference-checker/blob/main/checker.py) (Checker), [`app.py`](https://github.com/mprabhu26/lni-reference-checker/blob/main/app.py) (Flask app), [`extractor.py`](https://github.com/mprabhu26/lni-reference-checker/blob/main/extractor.py) (Extractor)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ui["User interface"]
  node_browser_ui["Browser UI<br/>static page<br/>[index.html]"]
end

subgraph group_core["Document pipeline"]
  node_extractor["Extractor<br/>format normalizer<br/>[extractor.py]"]
  node_parser["Parser<br/>[parser.py]"]
  node_checker["Checker<br/>analysis engine<br/>[checker.py]"]
end

subgraph group_verify["Verification"]
  node_web_verifier["Web verifier<br/>external lookup"]
  node_ai_checker["AI checker<br/>heuristic scorer<br/>[ai_checker.py]"]
end

subgraph group_store["Persistence"]
  node_local_db[("Local DB<br/>db access<br/>[local_db.py]")]
  node_review_queue["Review queue<br/>manual triage<br/>[review_queue.py]"]
  node_download_db[("Verified store<br/>paper store<br/>[download_db.py]")]
  node_cache_db[("Cache DB<br/>sqlite cache")]
  node_review_sqlite[("Review SQLite<br/>sqlite db<br/>[review_queue.db]")]
  node_verified_sqlite[("Verified SQLite<br/>sqlite db<br/>[verified_papers.db]")]
end

subgraph group_qa["Quality"]
  node_tests_all["Test suite<br/>[test_all.py]"]
  node_tests_cache["Cache tests<br/>pytest module<br/>[test_cache.py]"]
  node_tests_regression["Regression tests<br/>pytest module"]
end

node_flask_app["Flask app<br/>web controller<br/>[app.py]"]

node_browser_ui -->|"uploads"| node_flask_app
node_flask_app -->|"extracts"| node_extractor
node_extractor -->|"bibliography"| node_parser
node_parser -->|"entries"| node_checker
node_extractor -->|"text"| node_checker
node_checker -->|"findings"| node_flask_app
node_flask_app -->|"optional lookup"| node_web_verifier
node_web_verifier -->|"scores"| node_ai_checker
node_web_verifier -->|"reads/writes"| node_local_db
node_ai_checker -->|"persists"| node_local_db
node_local_db -->|"cache"| node_cache_db
node_local_db -->|"queue"| node_review_sqlite
node_local_db -->|"store"| node_verified_sqlite
node_review_queue -->|"backing db"| node_review_sqlite
node_download_db -->|"backing db"| node_verified_sqlite
node_tests_all -->|"covers"| node_checker
node_tests_cache -->|"validates"| node_cache_db
node_tests_regression -->|"covers"| node_parser
node_tests_regression -->|"covers"| node_checker

click node_browser_ui "https://github.com/mprabhu26/lni-reference-checker/blob/main/static/index.html"
click node_flask_app "https://github.com/mprabhu26/lni-reference-checker/blob/main/app.py"
click node_extractor "https://github.com/mprabhu26/lni-reference-checker/blob/main/extractor.py"
click node_parser "https://github.com/mprabhu26/lni-reference-checker/blob/main/parser.py"
click node_checker "https://github.com/mprabhu26/lni-reference-checker/blob/main/checker.py"
click node_web_verifier "https://github.com/mprabhu26/lni-reference-checker/blob/main/web_search_verifier.py"
click node_ai_checker "https://github.com/mprabhu26/lni-reference-checker/blob/main/ai_checker.py"
click node_local_db "https://github.com/mprabhu26/lni-reference-checker/blob/main/local_db.py"
click node_review_queue "https://github.com/mprabhu26/lni-reference-checker/blob/main/review_queue.py"
click node_download_db "https://github.com/mprabhu26/lni-reference-checker/blob/main/download_db.py"
click node_cache_db "https://github.com/mprabhu26/lni-reference-checker/blob/main/.lni_cache/2b79c6b40a6cf28f8ab28541.json"
click node_review_sqlite "https://github.com/mprabhu26/lni-reference-checker/blob/main/.lni_db/review_queue.db"
click node_verified_sqlite "https://github.com/mprabhu26/lni-reference-checker/blob/main/.lni_db/verified_papers.db"
click node_tests_all "https://github.com/mprabhu26/lni-reference-checker/blob/main/tests/test_all.py"
click node_tests_cache "https://github.com/mprabhu26/lni-reference-checker/blob/main/tests/test_cache.py"
click node_tests_regression "https://github.com/mprabhu26/lni-reference-checker/blob/main/tests/test_regression_full.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_browser_ui toneBlue
class node_extractor,node_parser,node_checker toneAmber
class node_web_verifier,node_ai_checker toneMint
class node_local_db,node_review_queue,node_download_db,node_cache_db,node_review_sqlite,node_verified_sqlite toneRose
class node_tests_all,node_tests_cache,node_tests_regression toneIndigo
class node_flask_app toneNeutral
```

## Components

### User interface

Browser-facing upload and results presentation

- [Browser UI](https://github.com/mprabhu26/lni-reference-checker/blob/main/static/index.html): static page · Single-page upload and results interface · `static/index.html`

### Document pipeline

Format normalization, parsing, and checking

- [Extractor](https://github.com/mprabhu26/lni-reference-checker/blob/main/extractor.py): format normalizer · Converts PDF, DOCX, and LaTeX inputs into analyzable text · `extractor.py`
- [Parser](https://github.com/mprabhu26/lni-reference-checker/blob/main/parser.py): bib parser · Turns bibliography blocks into structured reference records · `parser.py`
- [Checker](https://github.com/mprabhu26/lni-reference-checker/blob/main/checker.py): analysis engine · Compares citations, bibliography completeness, and cross-reference consistency · `checker.py`

### Verification

Optional external and heuristic reference verification

- [Web verifier](https://github.com/mprabhu26/lni-reference-checker/blob/main/web_search_verifier.py): external lookup · Checks references against CrossRef, Semantic Scholar, and Unpaywall · `web_search_verifier.py`
- [AI checker](https://github.com/mprabhu26/lni-reference-checker/blob/main/ai_checker.py): heuristic scorer · Secondary confidence and verification path · `ai_checker.py`

### Persistence

Local cache and SQLite-backed state

- [Local DB](https://github.com/mprabhu26/lni-reference-checker/blob/main/local_db.py): db access · Shared persistence helpers for local state · `local_db.py`
- [Review queue](https://github.com/mprabhu26/lni-reference-checker/blob/main/review_queue.py): manual triage · Queues ambiguous references for human review · `review_queue.py`
- [Verified store](https://github.com/mprabhu26/lni-reference-checker/blob/main/download_db.py): paper store · Holds verified paper metadata and download state · `download_db.py`
- [Cache DB](https://github.com/mprabhu26/lni-reference-checker/blob/main/.lni_cache/2b79c6b40a6cf28f8ab28541.json): sqlite cache · Cached verification results in the local filesystem · `.lni_cache/2b79c6b40a6cf28f8ab28541.json`
- [Review SQLite](https://github.com/mprabhu26/lni-reference-checker/blob/main/.lni_db/review_queue.db): sqlite db · Persistent review queue state · `.lni_db/review_queue.db`
- [Verified SQLite](https://github.com/mprabhu26/lni-reference-checker/blob/main/.lni_db/verified_papers.db): sqlite db · Persistent verified paper records · `.lni_db/verified_papers.db`

### Quality

Regression and cache behavior coverage

- [Test suite](https://github.com/mprabhu26/lni-reference-checker/blob/main/tests/test_all.py): pytest suite · Repository-wide regression and cache tests · `tests/test_all.py`
- [Cache tests](https://github.com/mprabhu26/lni-reference-checker/blob/main/tests/test_cache.py): pytest module · Validates cache behavior and persistence semantics · `tests/test_cache.py`
- [Regression tests](https://github.com/mprabhu26/lni-reference-checker/blob/main/tests/test_regression_full.py): pytest module · End-to-end regression coverage for the checker pipeline · `tests/test_regression_full.py`

### Other components

- [Flask app](https://github.com/mprabhu26/lni-reference-checker/blob/main/app.py): web controller · Request handling and orchestration entrypoint · `app.py`

## Connections

- Browser UI → Flask app: uploads · User submits manuscripts and receives results
- Flask app → Extractor: extracts · Routes uploaded files into format normalization
- Extractor → Parser: bibliography · Passes normalized reference blocks for structuring
- Parser → Checker: entries · Structured bibliography feeds analysis checks
- Extractor → Checker: text · Extracted manuscript text supports citation analysis
- Checker → Flask app: findings · Produces bibliography and cross-check results for the response
- Flask app → Web verifier: optional lookup · Invoked when online verification is enabled
- Web verifier → AI checker: scores · Heuristic or AI-assisted assessment complements web lookup
- Web verifier → Local DB: reads/writes · Uses local persistence for cached verification state
- AI checker → Local DB: persists · Stores auxiliary verification state or scores
- Local DB → Cache DB: cache · Reads and writes cached verification results
- Local DB → Review SQLite: queue · Persists ambiguous items for manual review
- Local DB → Verified SQLite: store · Stores verified paper records locally
- Review queue → Review SQLite: backing db · Review queue is backed by SQLite state
- Verified store → Verified SQLite: backing db · Verified paper store is backed by SQLite state
- Test suite → Checker: covers · Exercises the main analysis pipeline
- Cache tests → Cache DB: validates · Checks cache semantics and regressions
- Regression tests → Parser: covers · Protects parsing and end-to-end behavior
- Regression tests → Checker: covers · Protects rule-based checking behavior