# shef-aire/llms_post-ocr_correction architecture

> Architecture diagram of the GitHub repository shef-aire/llms_post-ocr_correction, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-11). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/shef-aire/llms_post-ocr_correction
- Repository: https://github.com/shef-aire/llms_post-ocr_correction

## Overview

##### Project type and purpose
- This is a research‐oriented ML pipeline, not a typical web or backend system. Its goal is to fine‐tune large language models (BART and Llama 2 variants) to perform post‐OCR correction on historical English newspapers, using the BLN600 parallel corpus, and then evaluate and analyze the results.

##### File structure
- Top‐level files:
- `bart.py`, `llama-2.py`: training/inference scripts for two model families.
- `bln600.ipynb`: data loading and preprocessing notebook.
- `results.ipynb`: inference and result‐analysis notebook.
- `config.yaml`: hyperparameter and path configuration.
- `requirements.txt`: dependency list.
- `data/`: input dataset (`train.csv`, `test.csv`, plus zipped sequences).
- `results/`: CSV outputs from inference runs.
- There are no service containers or microservices—this is a linear pipeline with clearly separated steps.

##### README for additional insights
- It describes:
- A prompt‐based approach for post‐OCR correction.
- Scripts accept `--model`, `--config`, and `--data` parameters.
- Dependencies on HuggingFace Transformers and a YAML‐based configuration.
- Two phases: fine-tuning and evaluation.

##### Detail without overthinking
- Config file as a side input to training scripts.
- Both BART and Llama flows in parallel or layered for comparison.
- Branching (e.g., two model families) clearly.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Preparation Stage
    subgraph "Data Preparation" 
        direction TB
        rawDataDir["data/ (Raw Data Dir)"]:::data
        trainCsv["data/train.csv"]:::data
        testCsv["data/test.csv"]:::data
        seqZip["data/Sequences.zip"]:::data
        bln600Notebook["bln600.ipynb (Data Prep Notebook)"]:::compute
        cleanedData["Cleaned Data"]:::data
    end

    rawDataDir --> trainCsv
    rawDataDir --> testCsv
    rawDataDir --> seqZip
    rawDataDir --> bln600Notebook
    bln600Notebook --> cleanedData

    %% Model Training Stage
    subgraph "Model Training" 
        direction TB
        configYaml["config.yaml"]:::data
        bartScript["bart.py (BART Training)"]:::compute
        llamaScript["llama-2.py (Llama 2 Training)"]:::compute
        externalHF["HuggingFace Hub"]:::external
        modelArtifacts["Model Artifacts"]:::data
    end

    cleanedData --> bartScript
    cleanedData --> llamaScript
    configYaml --> bartScript
    configYaml --> llamaScript
    externalHF --> bartScript
    externalHF --> llamaScript
    bartScript --> modelArtifacts
    llamaScript --> modelArtifacts

    %% Inference & Evaluation Stage
    subgraph "Inference & Evaluation" 
        direction TB
        resultsNotebook["results.ipynb (Results Notebook)"]:::compute
        resultsDir["results/ (Results Directory)"]:::data
        bartBaseCsv["results/bart-base.csv"]:::data
        bartLargeCsv["results/bart-large.csv"]:::data
        llama7bCsv["results/llama-2-7b.csv"]:::data
        llama13bCsv["results/llama-2-13b.csv"]:::data
    end

    modelArtifacts --> resultsNotebook
    testCsv --> resultsNotebook
    resultsNotebook --> resultsDir
    resultsDir --> bartBaseCsv
    resultsDir --> bartLargeCsv
    resultsDir --> llama7bCsv
    resultsDir --> llama13bCsv

    %% Project Dependencies
    subgraph "Project Files" 
        direction TB
        req["requirements.txt"]:::external
        readme["README.md"]:::external
        license["LICENSE"]:::external
    end

    %% Click Events
    click rawDataDir "https://github.com/shef-aire/llms_post-ocr_correction/tree/main/data/"
    click trainCsv "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/data/train.csv"
    click testCsv "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/data/test.csv"
    click seqZip "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/data/Sequences.zip"
    click bln600Notebook "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/bln600.ipynb"
    click configYaml "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/config.yaml"
    click bartScript "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/bart.py"
    click llamaScript "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/llama-2.py"
    click resultsNotebook "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/results.ipynb"
    click resultsDir "https://github.com/shef-aire/llms_post-ocr_correction/tree/main/results/"
    click bartBaseCsv "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/results/bart-base.csv"
    click bartLargeCsv "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/results/bart-large.csv"
    click llama7bCsv "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/results/llama-2-7b.csv"
    click llama13bCsv "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/results/llama-2-13b.csv"
    click req "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/requirements.txt"
    click readme "https://github.com/shef-aire/llms_post-ocr_correction/blob/main/README.md"
    click license "https://github.com/shef-aire/llms_post-ocr_correction/tree/main/LICENSE"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef compute fill:#90EE90,stroke:#333,stroke-width:1px
    classDef external fill:#FFE4B5,stroke:#333,stroke-width:1px
```
