# liuzhne/paper-grading-system architecture

> Architecture diagram of the GitHub repository liuzhne/paper-grading-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/liuzhne/paper-grading-system
- Repository: https://github.com/liuzhne/paper-grading-system

## Overview

1. **Overall shape**
This repository is a local-first thesis-grading system built around one shared scoring core. The main runtime is a FastAPI backend, with two operator surfaces on top of it: a browser UI served by the backend itself, and a zero-service CLI (`pgs`). A Streamlit app exists as a fallback/admin-style front end.

2. **Primary end-to-end workflow**
The core business flow is: define a rubric, organize a batch, upload papers, parse documents, retrieve evidence, run AI scoring item-by-item, allow human review, then export results to Excel/HTML and optionally write back to an online sheet. That closed loop is the central architecture of the repo.

3. **Backend API boundary**
`backend/app/api/routes/` contains the HTTP surface area split by domain: rubrics, batches, papers, scoring, calibration, exports, auth, and system status. These routes are thin entry points; they depend on shared services and Pydantic schemas rather than embedding business logic.

4. **Domain services layer**
Most of the real behavior lives under `backend/app/services/`. This is where the architecture fans out into scoring, calibration, paper ingestion, document parsing, retrieval, report generation, spreadsheet export, rubric import, caching, and LLM integration. The services layer is the main boundary between transport/UI concerns and the scoring pipeline.

5. **Scoring pipeline**
`backend/app/services/scoring/` is the core engine. It appears to combine rubric rules, deterministic checks, LLM-based evaluation, validation, and post-processing into score items and run results. Related subsystems such as `checkers/`, `coherence/`, and `retrieval/` feed evidence and quality signals into scoring rather than acting as standalone features.

6. **Document ingestion and parsing**
Uploaded papers flow through `services/papers/ingestion.py` and `services/document_parser/`. This subsystem handles file intake, parsing of DOCX/PDF content, chunking, format checks, section summaries, track-changes handling, and format resolution. Parsed artifacts are stored separately from raw uploads, which is important for diagramming the data lifecycle.

7. **Rubric import and calibration**
Rubrics are not just simple records; they can be imported from spreadsheets/docx-based formats through `services/rubric_import/`, then compiled and persisted. Calibration support lives in `services/calibration/`, including anchor management and analytics, and the API exposes dedicated endpoints for that part of the grading model.

8. **LLM integration boundary**
All model calls are abstracted behind `services/llm/`, with adapters for OpenAI-style and OpenAI-compatible providers plus a mock implementation and retry logic. The README makes clear that the system can fall back to local mock behavior when environment variables are not configured, so the architecture must show LLM as an optional external dependency rather than a hard requirement.

9. **Persistence model**
State is stored in PostgreSQL via SQLAlchemy models in `backend/app/db/models.py`, with session management in `db/session.py` and schema evolution via Alembic migrations. In addition to the relational database, the system writes files to `storage/` for uploads, parsed JSON, reports, exports, and LLM cache state. That split between relational metadata and file artifacts is a key architectural boundary.

10. **Batching, runs, and review**
Batches are a top-level operational unit. A batch groups papers, scoring runs, summaries, rankings, drift analysis, and export logs. Scoring produces run-level records and item-level results, which can then be reviewed and patched through dedicated endpoints before final report/export generation.

11. **CLI as a first-class interface**
The `backend/app/cli/` package exposes the same core services without running a web server. It supports initialization, rubric import, scoring, review, reporting, export, and evaluation. This is not a separate implementation path; it is a second front door into the same backend architecture, designed for offline or batch use.

12. **Reporting and export**
`services/report/` and `services/spreadsheet/` handle output generation. The system can generate human-readable reports, JSON exports, Excel exports, and teacher score templates. There is also an online sheet writer integration point, but it is optional and external.

13. **Evaluation and quality gates**
`backend/app/eval/` and the `scripts/run_qwk_eval.py` path implement an offline evaluation pipeline for scoring quality, centered on QWK and related metrics against teacher-labeled data. This subsystem is separate from production scoring but shares the same rubric and paper parsing model, which matters if you are diagramming validation or release gates.

14. **Operational and deployment boundaries**
The repository is designed to run locally with Dockerized PostgreSQL, Alembic migrations, and environment-driven integration toggles. The external services that materially affect architecture are the LLM provider and the optional Google Sheets write-back endpoint. Everything else is intentionally self-contained for local operation and offline fallback.

Main entry points: [`backend/app/main.py`](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/main.py) (FastAPI app), [`backend/app/services/scoring/engine.py`](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/scoring/engine.py) (Scoring engine), [`backend/app/services/papers/ingestion.py`](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/papers/ingestion.py) (Ingestion & parse), [`backend/app/services/rubric_import/compiler.py`](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/rubric_import/compiler.py) (Rubric & calibration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_interfaces["Interfaces"]
  node_main_api["FastAPI app<br/>web server<br/>[main.py]"]
  node_browser_ui["Web UI<br/>static frontend<br/>[index.html]"]
  node_streamlit_ui["Streamlit app<br/>fallback UI<br/>[streamlit_app.py]"]
  node_cli_frontend["CLI<br/>operator shell<br/>[main.py]"]
end

subgraph group_api["HTTP API"]
  node_auth_route["Auth<br/>API routes<br/>[auth.py]"]
  node_batch_route["Batches<br/>API routes<br/>[batches.py]"]
  node_papers_route["Papers<br/>API routes<br/>[papers.py]"]
  node_rubrics_route["Rubrics<br/>API routes<br/>[rubrics.py]"]
  node_scoring_route["Scoring<br/>API routes<br/>[scoring.py]"]
  node_calibration_route["Calibration<br/>API routes<br/>[calibration.py]"]
  node_exports_route["Exports<br/>API routes<br/>[exports.py]"]
  node_system_route["System<br/>API routes<br/>[system.py]"]
end

subgraph group_services["Core Services"]
  node_scoring_engine{{"Scoring engine<br/>core pipeline<br/>[engine.py]"}}
  node_ingestion_parser{{"Ingestion & parse<br/>document pipeline<br/>[ingestion.py]"}}
  node_rubric_calibration{{"Rubric & calibration<br/>domain services<br/>[compiler.py]"}}
  node_report_export{{"Reports & export<br/>output services<br/>[generator.py]"}}
  node_retrieval_checks{{"Retrieval & checks<br/>support services<br/>[keyword.py]"}}
end

subgraph group_infra["Data and Integrations"]
  node_llm_boundary{{"LLM adapters<br/>model integration<br/>[factory.py]"}}
  node_db_layer[("Database layer<br/>persistence<br/>[models.py]")]
  node_file_storage[("File storage<br/>artifact store<br/>[local.py]")]
  node_llm_cache[("LLM cache<br/>[llm_cache.py]")]
  node_eval_suite["Eval suite<br/>offline evaluation<br/>[runner.py]"]
end

node_browser_ui -->|"calls"| node_main_api
node_streamlit_ui -->|"calls"| node_main_api
node_cli_frontend -->|"uses"| node_rubric_calibration
node_cli_frontend -->|"uses"| node_scoring_engine
node_main_api -->|"mounts"| node_auth_route
node_main_api -->|"mounts"| node_batch_route
node_main_api -->|"mounts"| node_papers_route
node_main_api -->|"mounts"| node_rubrics_route
node_main_api -->|"mounts"| node_scoring_route
node_main_api -->|"mounts"| node_calibration_route
node_main_api -->|"mounts"| node_exports_route
node_main_api -->|"mounts"| node_system_route
node_batch_route -->|"orchestrates"| node_scoring_engine
node_papers_route -->|"delegates"| node_ingestion_parser
node_rubrics_route -->|"delegates"| node_rubric_calibration
node_scoring_route -->|"delegates"| node_scoring_engine
node_calibration_route -->|"delegates"| node_rubric_calibration
node_exports_route -->|"delegates"| node_report_export
node_scoring_engine -->|"uses"| node_retrieval_checks
node_scoring_engine -->|"uses"| node_llm_boundary
node_scoring_engine -->|"persists"| node_db_layer
node_ingestion_parser -->|"reads/writes"| node_file_storage
node_ingestion_parser -->|"persists"| node_db_layer
node_rubric_calibration -->|"persists"| node_db_layer
node_report_export -->|"writes"| node_file_storage
node_llm_boundary -->|"caches"| node_llm_cache
node_llm_boundary -->|"writes"| node_file_storage
node_eval_suite -->|"evaluates"| node_scoring_engine
node_eval_suite -->|"reuses"| node_ingestion_parser
node_db_layer -->|"coordinates"| node_file_storage

click node_main_api "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/main.py"
click node_browser_ui "https://github.com/liuzhne/paper-grading-system/blob/main/frontend/web/index.html"
click node_streamlit_ui "https://github.com/liuzhne/paper-grading-system/blob/main/frontend/streamlit_app.py"
click node_cli_frontend "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/cli/main.py"
click node_auth_route "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/auth.py"
click node_batch_route "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/batches.py"
click node_papers_route "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/papers.py"
click node_rubrics_route "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/rubrics.py"
click node_scoring_route "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/scoring.py"
click node_calibration_route "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/calibration.py"
click node_exports_route "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/exports.py"
click node_system_route "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/system.py"
click node_scoring_engine "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/scoring/engine.py"
click node_ingestion_parser "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/papers/ingestion.py"
click node_rubric_calibration "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/rubric_import/compiler.py"
click node_report_export "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/report/generator.py"
click node_retrieval_checks "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/retrieval/keyword.py"
click node_llm_boundary "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/llm/factory.py"
click node_db_layer "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/db/models.py"
click node_file_storage "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/storage/local.py"
click node_llm_cache "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/cache/llm_cache.py"
click node_eval_suite "https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/eval/runner.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_main_api,node_browser_ui,node_streamlit_ui,node_cli_frontend toneBlue
class node_auth_route,node_batch_route,node_papers_route,node_rubrics_route,node_scoring_route,node_calibration_route,node_exports_route,node_system_route toneAmber
class node_scoring_engine,node_ingestion_parser,node_rubric_calibration,node_report_export,node_retrieval_checks toneMint
class node_llm_boundary,node_db_layer,node_file_storage,node_llm_cache,node_eval_suite toneRose
```

## Components

### Interfaces

User-facing entry points and operators

- [FastAPI app](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/main.py): web server · Backend application entry point that assembles the API and runtime · `backend/app/main.py`
- [Web UI](https://github.com/liuzhne/paper-grading-system/blob/main/frontend/web/index.html): static frontend · Browser-based operator interface served alongside the backend · `frontend/web/index.html`
- [Streamlit app](https://github.com/liuzhne/paper-grading-system/blob/main/frontend/streamlit_app.py): fallback UI · Alternative admin-style frontend for local use · `frontend/streamlit_app.py`
- [CLI](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/cli/main.py): operator shell · Zero-service command line entry point for offline and batch workflows · `backend/app/cli/main.py`

### HTTP API

FastAPI routes exposing the domain surface

- [Auth](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/auth.py): API routes · Authentication and user/session surface · `backend/app/api/routes/auth.py`
- [Batches](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/batches.py): API routes · Batch lifecycle, summaries, rankings, and review coordination · `backend/app/api/routes/batches.py`
- [Papers](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/papers.py): API routes · Paper upload and document handling endpoints · `backend/app/api/routes/papers.py`
- [Rubrics](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/rubrics.py): API routes · Rubric definition and import endpoints · `backend/app/api/routes/rubrics.py`
- [Scoring](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/scoring.py): API routes · Scoring, review, and patch endpoints · `backend/app/api/routes/scoring.py`
- [Calibration](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/calibration.py): API routes · Anchor and calibration management endpoints · `backend/app/api/routes/calibration.py`
- [Exports](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/exports.py): API routes · Report and export endpoints · `backend/app/api/routes/exports.py`
- [System](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/api/routes/system.py): API routes · Health and status endpoints · `backend/app/api/routes/system.py`

### Core Services

Scoring pipeline and domain logic

- [Scoring engine](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/scoring/engine.py): core pipeline · Hybrid scoring core combining rules, validation, and model-assisted judgment · `backend/app/services/scoring/engine.py`
- [Ingestion & parse](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/papers/ingestion.py): document pipeline · File intake, parsing, chunking, summaries, and format handling · `backend/app/services/papers/ingestion.py`
- [Rubric & calibration](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/rubric_import/compiler.py): domain services · Rubric import, persistence, anchors, and calibration analytics · `backend/app/services/rubric_import/compiler.py`
- [Reports & export](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/report/generator.py): output services · Human-readable reports, JSON exports, and spreadsheet generation · `backend/app/services/report/generator.py`
- [Retrieval & checks](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/retrieval/keyword.py): support services · Evidence retrieval, deterministic checks, coherence, and validation support · `backend/app/services/retrieval/keyword.py`

### Data and Integrations

Database, files, LLM, and evaluation support

- [LLM adapters](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/llm/factory.py): model integration · Provider abstraction with OpenAI-compatible, mock, and retry behavior · `backend/app/services/llm/factory.py`
- [Database layer](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/db/models.py): persistence · SQLAlchemy models, sessions, and migration-backed relational state · `backend/app/db/models.py`
- [File storage](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/storage/local.py): artifact store · Local storage for uploads, parsed artifacts, reports, and exports · `backend/app/services/storage/local.py`
- [LLM cache](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/services/cache/llm_cache.py): cache · Cached model responses and local cache state · `backend/app/services/cache/llm_cache.py`
- [Eval suite](https://github.com/liuzhne/paper-grading-system/blob/main/backend/app/eval/runner.py): offline evaluation · Quality evaluation pipeline for QWK and labeled datasets · `backend/app/eval/runner.py`

## Connections

- Web UI → FastAPI app: calls · Uses the backend HTTP surface for operator workflows
- Streamlit app → FastAPI app: calls · Optional frontend that reaches the same backend services
- CLI → Rubric & calibration: uses · CLI invokes shared domain services without a web server
- CLI → Scoring engine: uses · Runs scoring and review flows directly
- FastAPI app → Auth: mounts · Registers authentication routes
- FastAPI app → Batches: mounts · Registers batch lifecycle routes
- FastAPI app → Papers: mounts · Registers paper ingestion routes
- FastAPI app → Rubrics: mounts · Registers rubric routes
- FastAPI app → Scoring: mounts · Registers scoring and review routes
- FastAPI app → Calibration: mounts · Registers calibration routes
- FastAPI app → Exports: mounts · Registers export routes
- FastAPI app → System: mounts · Registers system/status routes
- Batches → Scoring engine: orchestrates · Coordinates batch-level scoring and review
- Papers → Ingestion & parse: delegates · Hands uploads to the document pipeline
- Rubrics → Rubric & calibration: delegates · Creates and imports rubric definitions
- Scoring → Scoring engine: delegates · Runs item-level and run-level scoring
- Calibration → Rubric & calibration: delegates · Manages anchors and calibration workflows
- Exports → Reports & export: delegates · Generates reports and export artifacts
- Scoring engine → Retrieval & checks: uses · Consumes evidence retrieval and deterministic quality checks
- Scoring engine → LLM adapters: uses · Optionally calls model providers through an abstraction
- Scoring engine → Database layer: persists · Stores runs, items, scores, and review state
- Ingestion & parse → File storage: reads/writes · Moves raw uploads into parsed artifacts and summaries
- Ingestion & parse → Database layer: persists · Records paper metadata and parsing state
- Rubric & calibration → Database layer: persists · Stores rubrics, anchors, and calibration metadata
- Reports & export → File storage: writes · Creates reports, Excel exports, and JSON outputs
- LLM adapters → LLM cache: caches · Stores and reuses model responses when enabled
- LLM adapters → File storage: writes · May record diagnostics and local fallback artifacts
- Eval suite → Scoring engine: evaluates · Measures scoring quality against labeled data
- Eval suite → Ingestion & parse: reuses · Shares the same parsing and document model
- Database layer → File storage: coordinates · Database metadata points to stored artifacts