How vyasylum/lawlinks works
Project type and purpose
• This is a research-oriented NLP pipeline for legal case translation and question-answering.
• It ingests PDF case files in multiple languages, translates them into English using Marian NMT, stores the outputs, then fine-tunes and serves a FLAN-T5 model to answer user queries.
File structure
• Top-level “Case Files” folder holds raw PDFs.
• “Datasets” folder holds JSON datasets for training/QA.
• Notebooks:
• MarianMT.ipynb – translation pipeline
• queryprocess_final.ipynb – QA fine-tuning and inference
• Translated Outputs folder holds generated text/PDFs.
• Config: requirements.txt, environment variable for CUDA.
Key insights from README
• Two main stages: translation (Marian NMT) and question answering (FLAN-T5).
• Data curation (“elder brother” LLM) yields training JSONs.
• No user interface yet—pure batch/interactive notebooks.
Main components
a. Ingestion Layer
• PDF Uploader / File System (Case Files)
• PDF Parser / Text Extractor
b. Translation Service
• MarianMT Translation Module (HuggingFace integration)
c. Storage
• Translated Outputs (text files, PDFs)
• Dataset JSONs for QA (cleaned.json, cleaned_final.json, etc.)
d. Preprocessing & Training
• Preprocessor (tokenization, formatting)
• FLAN-T5 Fine-tuning Pipeline
e. Inference / Query Service
• FLAN-T5 Model Server
• Query Processor (accepts user questions, returns answers)
f. External Dependencies
• HuggingFace Models (MarianMT, T5)
• GPU / CUDA runtime environment
Architectural patterns and principles
• Pipeline/ETL pattern: sequential stages of data transformation.
• Separation of concerns: ingestion, translation, storage, training, inference.
• Modular components: swapping in new translation or QA models without rewriting the pipeline.
• Batch vs. interactive: Notebooks for batch tasks; a potential microservice for real-time queries.
Technologies, frameworks, and libraries
• Python, Jupyter Notebooks
• HuggingFace Transformers (MarianMT, T5)
• PyTorch (CUDA memory config via environment variable)
• JSON for dataset storage
• PDF parsing libraries (e.g., pdfminer, PyPDF2)
Detail level
• Notebook as a separate module in the pipeline.
• File-based storage paths (e.g., “Case Files/”, “Translated Outputs/”, “Datasets/”).