liuzhne/paper-grading-system

1. Overall shape

This repository is a local-first thesis-grading system built around one shared scoring core. The main runtime is a FastAPI backend, with two operator surfaces on top of it: a browser UI served by the backend itself, and a zero-service CLI (pgs). A Streamlit app exists as a fallback/admin-style front end.

2. Primary end-to-end workflow

The core business flow is: define a rubric, organize a batch, upload papers, parse documents, retrieve evidence, run AI scoring item-by-item, allow human review, then export results to Excel/HTML and optionally write back to an online sheet. That closed loop is the central architecture of the repo.

3. Backend API boundary

backend/app/api/routes/ contains the HTTP surface area split by domain: rubrics, batches, papers, scoring, calibration, exports, auth, and system status. These routes are thin entry points; they depend on shared services and Pydantic schemas rather than embedding business logic.

4. Domain services layer

Most of the real behavior lives under backend/app/services/. This is where the architecture fans out into scoring, calibration, paper ingestion, document parsing, retrieval, report generation, spreadsheet export, rubric import, caching, and LLM integration. The services layer is the main boundary between transport/UI concerns and the scoring pipeline.

5. Scoring pipeline

backend/app/services/scoring/ is the core engine. It appears to combine rubric rules, deterministic checks, LLM-based evaluation, validation, and post-processing into score items and run results. Related subsystems such as checkers/, coherence/, and retrieval/ feed evidence and quality signals into scoring rather than acting as standalone features.

6. Document ingestion and parsing

Uploaded papers flow through services/papers/ingestion.py and services/document_parser/. This subsystem handles file intake, parsing of DOCX/PDF content, chunking, format checks, section summaries, track-changes handling, and format resolution. Parsed artifacts are stored separately from raw uploads, which is important for diagramming the data lifecycle.

7. Rubric import and calibration

Rubrics are not just simple records; they can be imported from spreadsheets/docx-based formats through services/rubric_import/, then compiled and persisted. Calibration support lives in services/calibration/, including anchor management and analytics, and the API exposes dedicated endpoints for that part of the grading model.

8. LLM integration boundary

All model calls are abstracted behind services/llm/, with adapters for OpenAI-style and OpenAI-compatible providers plus a mock implementation and retry logic. The README makes clear that the system can fall back to local mock behavior when environment variables are not configured, so the architecture must show LLM as an optional external dependency rather than a hard requirement.

9. Persistence model

State is stored in PostgreSQL via SQLAlchemy models in backend/app/db/models.py, with session management in db/session.py and schema evolution via Alembic migrations. In addition to the relational database, the system writes files to storage/ for uploads, parsed JSON, reports, exports, and LLM cache state. That split between relational metadata and file artifacts is a key architectural boundary.

10. Batching, runs, and review

Batches are a top-level operational unit. A batch groups papers, scoring runs, summaries, rankings, drift analysis, and export logs. Scoring produces run-level records and item-level results, which can then be reviewed and patched through dedicated endpoints before final report/export generation.

11. CLI as a first-class interface

The backend/app/cli/ package exposes the same core services without running a web server. It supports initialization, rubric import, scoring, review, reporting, export, and evaluation. This is not a separate implementation path; it is a second front door into the same backend architecture, designed for offline or batch use.

12. Reporting and export

services/report/ and services/spreadsheet/ handle output generation. The system can generate human-readable reports, JSON exports, Excel exports, and teacher score templates. There is also an online sheet writer integration point, but it is optional and external.

13. Evaluation and quality gates

backend/app/eval/ and the scripts/run_qwk_eval.py path implement an offline evaluation pipeline for scoring quality, centered on QWK and related metrics against teacher-labeled data. This subsystem is separate from production scoring but shares the same rubric and paper parsing model, which matters if you are diagramming validation or release gates.

14. Operational and deployment boundaries

The repository is designed to run locally with Dockerized PostgreSQL, Alembic migrations, and environment-driven integration toggles. The external services that materially affect architecture are the LLM provider and the optional Google Sheets write-back endpoint. Everything else is intentionally self-contained for local operation and offline fallback.