mprabhu26/lni-reference-checker

1. Overall shape

This is a small Python web application with a Flask backend (app.py) and a single-page browser UI (static/index.html). The backend does the document processing; the frontend is mainly a thin presentation layer for uploads and results.

2. User-facing flow

The main input boundary is a submitted academic manuscript in PDF, DOCX, or LaTeX form, optionally with a separate .bib file for LaTeX. The user toggles whether to run online verification, then the server returns structured findings for bibliography completeness, citation cross-checking, and reference verification.

3. Format-specific extraction

extractor.py is the first processing stage. It normalizes the different source formats into text:

- PDF and DOCX are text-extracted and split around the bibliography section.

- LaTeX is cleaned of markup, and the .bib sidecar can be parsed directly.

This is the boundary between raw file handling and document understanding.

4. Bibliography parsing

parser.py turns the extracted bibliography into structured entries keyed by citation labels like [Key]. This is where the app converts human-readable reference blocks into machine-checkable records with fields such as author, title, year, and venue.

5. Core checking engine

checker.py appears to be the main analysis layer. It compares:

- in-text citations vs. bibliography entries,

- bibliography completeness vs. expected fields,

- cited references vs. uncited bibliography items.

This is the subsystem that produces the “bibliography” and “cross-check” findings shown in the UI.

6. Online verification boundary

web_search_verifier.py adds an external-service layer. Per the README, it queries CrossRef, Semantic Scholar, and Unpaywall to assess whether a reference appears real and whether an open-access full text is available. This is an optional enrichment step, not required for offline checking.

7. Auxiliary AI/scoring path

ai_checker.py suggests an additional heuristic or AI-assisted verification path alongside the web lookups. Architecturally, this is a secondary confidence/scoring component rather than the primary source of truth.

8. State, caching, and manual review

The repo includes local_db.py, review_queue.py, download_db.py, .lni_cache/, and .lni_db/. Together these indicate a local persistence layer for:

- caching verification results,

- storing verified papers,

- queuing ambiguous items for review.

The .db files are SQLite databases, so this is a local-first architecture rather than a remote backend service.

9. Presentation of results

The frontend is a static HTML app that likely renders three conceptual result views described in the README:

- bibliography quality/completeness,

- citation cross-check status,

- verification outcome for each reference.

The UI is not a separate API consumer in another service; it is served by the Flask app and talks to the local backend.

10. Deployment and runtime

requirements.txt defines the Python runtime dependencies, and Procfile suggests the app can be launched in a platform-friendly process model. The architecture is therefore “single-process web app + local storage + external verification APIs,” not a microservice system.

11. Testing and regression safety

The tests/ folder, pytest.ini, and fixtures indicate an emphasis on deterministic checking. test_cache.py and test_regression_full.py imply the system is sensitive to parser/checker correctness and cache behavior, which matters because reference validation is rule-heavy and easy to regress.

12. Diagram-worthy boundary summary

A clear architecture diagram should show:

- Browser UI

- Flask app/controller

- Document extraction and parsing pipeline

- Reference/citation analysis engine

- External reference-verification services

- Local SQLite/cache persistence

- Manual review queue

These are the main data-flow and trust boundaries in the repository.