# vyasylum/lawlinks architecture

> Architecture diagram of the GitHub repository vyasylum/lawlinks, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vyasylum/lawlinks
- Repository: https://github.com/vyasylum/lawlinks

## Overview

##### Project type and purpose
- This is a research-oriented NLP pipeline for legal case translation and question-answering.
- It ingests PDF case files in multiple languages, translates them into English using Marian NMT, stores the outputs, then fine-tunes and serves a FLAN-T5 model to answer user queries.

##### File structure
- Top-level “Case Files” folder holds raw PDFs.
- “Datasets” folder holds JSON datasets for training/QA.
- Notebooks:
- MarianMT.ipynb – translation pipeline
- queryprocess_final.ipynb – QA fine-tuning and inference
- Translated Outputs folder holds generated text/PDFs.
- Config: requirements.txt, environment variable for CUDA.

##### Key insights from README
- Two main stages: translation (Marian NMT) and question answering (FLAN-T5).
- Data curation (“elder brother” LLM) yields training JSONs.
- No user interface yet—pure batch/interactive notebooks.

##### Main components
a. Ingestion Layer
- PDF Uploader / File System (Case Files)
- PDF Parser / Text Extractor
b. Translation Service
- MarianMT Translation Module (HuggingFace integration)
c. Storage
- Translated Outputs (text files, PDFs)
- Dataset JSONs for QA (cleaned.json, cleaned_final.json, etc.)
d. Preprocessing & Training
- Preprocessor (tokenization, formatting)
- FLAN-T5 Fine-tuning Pipeline
e. Inference / Query Service
- FLAN-T5 Model Server
- Query Processor (accepts user questions, returns answers)
f. External Dependencies
- HuggingFace Models (MarianMT, T5)
- GPU / CUDA runtime environment

##### Architectural patterns and principles
- Pipeline/ETL pattern: sequential stages of data transformation.
- Separation of concerns: ingestion, translation, storage, training, inference.
- Modular components: swapping in new translation or QA models without rewriting the pipeline.
- Batch vs. interactive: Notebooks for batch tasks; a potential microservice for real-time queries.

##### Technologies, frameworks, and libraries
- Python, Jupyter Notebooks
- HuggingFace Transformers (MarianMT, T5)
- PyTorch (CUDA memory config via environment variable)
- JSON for dataset storage
- PDF parsing libraries (e.g., pdfminer, PyPDF2)

##### Detail level
- Notebook as a separate module in the pipeline.
- File-based storage paths (e.g., “Case Files/”, “Translated Outputs/”, “Datasets/”).

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Ingestion Layer
    User["User / Cron"]:::external
    Ingestion["PDF Uploader / File System"]:::storage
    User -->|"upload PDFs"| Ingestion
    Ingestion -->|"raw PDFs"| Parser

    %% Translation Pipeline
    subgraph "Translation Stage"
        Parser["PDF Parser / Text Extractor"]:::translation
        Translation["MarianMT Translation Service"]:::translation
    end
    Parser -->|"raw text"| Translation
    Translation -->|"translated text"| TranslatedStore

    %% Storage
    TranslatedStore["Translated Outputs"]:::storage
    DatasetStore["Dataset JSON Storage"]:::storage
    Translation --> TranslatedStore
    DatasetStore

    %% QA Pipeline
    subgraph "QA Stage"
        Preprocessor["Preprocessor"]:::qa
        Trainer["FLAN-T5 Fine-tuning Pipeline"]:::qa
        QueryProc["Query Processor / Model Server"]:::qa
    end
    TranslatedStore -->|"translated text"| Preprocessor
    DatasetStore -->|"raw JSON"| Preprocessor
    Preprocessor -->|"training JSON"| Trainer
    Trainer -->|"trained model"| QueryProc

    %% User Queries
    User -->|"question"| QueryProc
    QueryProc -->|"answer JSON"| User

    %% External Dependencies
    HF["HuggingFace Models"]:::external
    GPU["GPU / CUDA Environment"]:::external
    HF --> Parser
    HF --> Translation
    HF --> Trainer
    GPU --> Translation
    GPU --> Trainer
    Docs["Project Documentation"]:::external

    %% Click Events
    click Ingestion "https://github.com/vyasylum/lawlinks/tree/main/Case Files/"
    click Parser "https://github.com/vyasylum/lawlinks/blob/main/MarianMT.ipynb"
    click Translation "https://github.com/vyasylum/lawlinks/blob/main/MarianMT.ipynb"
    click TranslatedStore "https://github.com/vyasylum/lawlinks/tree/main/Translated Outputs/"
    click DatasetStore "https://github.com/vyasylum/lawlinks/tree/main/Datasets/"
    click Preprocessor "https://github.com/vyasylum/lawlinks/blob/main/queryprocess_final.ipynb"
    click Trainer "https://github.com/vyasylum/lawlinks/blob/main/queryprocess_final.ipynb"
    click QueryProc "https://github.com/vyasylum/lawlinks/blob/main/queryprocess_final.ipynb"
    click HF "https://github.com/vyasylum/lawlinks/blob/main/requirements.txt"
    click Docs "https://github.com/vyasylum/lawlinks/blob/main/ReadMe.md"

    %% Styles
    classDef translation fill:#D0E8FF,stroke:#333,stroke-width:1px
    classDef qa fill:#DFFFE0,stroke:#333,stroke-width:1px
    classDef storage fill:#FFE4B5,stroke:#333,stroke-width:1px
    classDef external fill:#F0F0F0,stroke:#333,stroke-width:1px
```
