How shef-aire/llms_post-ocr_correction works
Project type and purpose
• This is a research‐oriented ML pipeline, not a typical web or backend system. Its goal is to fine‐tune large language models (BART and Llama 2 variants) to perform post‐OCR correction on historical English newspapers, using the BLN600 parallel corpus, and then evaluate and analyze the results.
File structure
• Top‐level files:
• bart.py, llama-2.py: training/inference scripts for two model families.
• bln600.ipynb: data loading and preprocessing notebook.
• results.ipynb: inference and result‐analysis notebook.
• config.yaml: hyperparameter and path configuration.
• requirements.txt: dependency list.
• data/: input dataset (train.csv, test.csv, plus zipped sequences).
• results/: CSV outputs from inference runs.
• There are no service containers or microservices—this is a linear pipeline with clearly separated steps.
README for additional insights
• It describes:
• A prompt‐based approach for post‐OCR correction.
• Scripts accept --model, --config, and --data parameters.
• Dependencies on HuggingFace Transformers and a YAML‐based configuration.
• Two phases: fine-tuning and evaluation.
Detail without overthinking
• Config file as a side input to training scripts.
• Both BART and Llama flows in parallel or layered for comparison.
• Branching (e.g., two model families) clearly.