# nmnguyen130/mathsolver architecture

> Architecture diagram of the GitHub repository nmnguyen130/mathsolver, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nmnguyen130/mathsolver
- Repository: https://github.com/nmnguyen130/mathsolver

## Overview

##### Project type and purpose
- Type: A machine-learning pipeline with three specialized subsystems (image2latex, mathsolver, mathwriting) plus shared preprocessing utilities and a small web demo.
- Purpose: Ingest handwritten mathematical expression data (InkML), preprocess it, train and evaluate models for (a) LaTeX transcription, (b) equation solving, (c) handwriting detection/recognition, and optionally serve results via a simple web UI.

##### File structure
- build/ (scripts for dataset preparation and visualization)
- run.py (entry point orchestration)
- src/
– image2latex/ (data modules, preprocessing, model definitions, trainers)
– mathsolver/ (data modules, tokenizer, solver/trainer scripts)
– mathwriting/ (data modules, encoders, detector/trainer)
– shared/preprocessing/ (common inkml loaders, renderers, symbol extractors)
- web/ (static HTML/CSS/JS demo)
- requirements.txt, checkpoints folders (serialized model artifacts)

##### README for insights
- Data: InkML files split into train/valid/test/symbols/synthetic with metadata.
- Workflow: Preprocess raw InkML → generate tokenized datasets → train model → produce checkpoints → run inference.
- References: Im2LaTeX, Google Research MathWriting dataset.

a. Main components
1) Data Sources
- Raw InkML dataset (train/valid/test/synthetic)
- JSONL metadata (synthetic-bboxes.jsonl, symbols.jsonl)
2) Preprocessing Layer
- shared/preprocessing: InkML loader, stroke renderer, symbol extractor
- image2latex/preprocessing, mathsolver/preprocessing, mathwriting/preprocessing
3) DataModule Layer
- Dataset constructors (dataset.py)
- Dataloaders (dataloader.py, mw_dataloader.py)
4) Model Layer
- image2latex.models (seq2seq POS and baseline)
- mathsolver.models (solver network)
- mathwriting.models (Swin/VIT encoders + detector)
5) Training Orchestration
- image2latex/scripts/{trainer,mw_trainer}.py
- mathsolver/scripts/trainer.py
- mathwriting/scripts/trainer.py
- run.py (high-level runner)
6) Checkpoint Storage
- src/*/checkpoints/*.pt, vocab.txt
7) Inference/Serving
- image2latex/scripts/… inference logic
- mathsolver/scripts/solver.py
- mathwriting/scripts/detector.py
- web/ (home.html, script.js, styles.css)
8) Build Utilities
- build/*.py (pre_render_dataset, resize_image, visualize_batch, api wrapper)

b. Relationships & Data‐flows
– Data Sources → Preprocessing
– Preprocessing → DataModule → Model Training
– Model Training ↔ Model (iterative updates)
– Model Training → Checkpoints → Inference/Serving
– Inference/Serving → Web UI (optional)
- External dependencies (Python, PyTorch, Lightning, libraries in requirements.txt)

c. Architectural patterns & principles
- Clear separation of concerns:
– Data ingestion vs. preprocessing vs. modeling vs. serving
- Modular design: three parallel subprojects sharing common utilities
- Pipeline pattern: sequential stages connected by well-defined interfaces (Dataset → Dataloader → Model)
- Configurable entry points: individual scripts + unified run.py

d. Technologies/frameworks
- Python 3.x, PyTorch (or TensorFlow)
- Lightning/PyTorch Lightning (implied by trainer scripts)
- Web: HTML/CSS/JS for lightweight demo
- InkML parsing libraries

##### Diagramming conventions
– Another for compute components (rectangles for scripts/libraries)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Sources
    RAW_DATA["Raw InkML Dataset"]:::data
    METADATA["JSONL Metadata"]:::data

    %% Build Utilities
    subgraph "Build Utilities"
        PRE_RENDER["pre_render_dataset.py"]:::compute
        RESIZE_IMG["resize_image.py"]:::compute
        VIS_BATCH["visualize_batch.py"]:::compute
        API_WRAPPER["api.py"]:::compute
    end

    %% Shared Preprocessing
    subgraph "Shared Preprocessing"
        INKML_LOADER["InkML Loader"]:::compute
        INK_RENDER["Ink Renderer"]:::compute
        STROKE_RENDER["Stroke Renderer"]:::compute
        BBOX_LOADER["BBox Loader"]:::compute
        SYMBOL_EXT["Symbol Extractor"]:::compute
        SHARED_MAIN["main.py"]:::compute
    end

    %% Image2LaTeX Subsystem
    subgraph "Image2LaTeX"
        subgraph "Preprocessing"
            I2L_DM["data_manager.py"]:::compute
            I2L_DP["data_processing.py"]:::compute
            I2L_FE["filter_error_image.py"]:::compute
            I2L_GL["generate_latex_file.py"]:::compute
            I2L_LP["latex_processing.py"]:::compute
            I2L_TOK["tokenizer.py"]:::compute
        end
        subgraph "DataModule"
            I2L_DS["dataset.py"]:::compute
            I2L_DL["dataloader.py"]:::compute
            I2L_MWDL["mw_dataloader.py"]:::compute
        end
        subgraph "Models"
            I2L_MODEL["model.py"]:::compute
            I2L_POS["pos_model.py"]:::compute
        end
        subgraph "Training"
            I2L_TR["trainer.py"]:::compute
            I2L_MWTR["mw_trainer.py"]:::compute
        end
        I2L_CP["Checkpoints"]:::storage
    end

    %% MathSolver Subsystem
    subgraph "MathSolver"
        subgraph "Preprocessing"
            MS_TOK["tokenizer.py"]:::compute
        end
        subgraph "DataModule"
            MS_CD["create_dataset.py"]:::compute
            MS_DS["dataset.py"]:::compute
            MS_DL["dataloader.py"]:::compute
        end
        subgraph "Models"
            MS_MODEL["model.py"]:::compute
        end
        subgraph "Training/Inference"
            MS_TR["trainer.py"]:::compute
            MS_SOLVER["solver.py"]:::compute
        end
        MS_CP["Checkpoints"]:::storage
    end

    %% MathWriting Subsystem
    subgraph "MathWriting"
        subgraph "Preprocessing"
            MW_TOK["tokenizer.py"]:::compute
        end
        subgraph "DataModule"
            MW_DS["dataset.py"]:::compute
            MW_DL["dataloader.py"]:::compute
            MW_TRANS["transforms.py"]:::compute
        end
        subgraph "Models"
            MW_MODEL["model.py"]:::compute
            SWIN["swin_encoder.py"]:::compute
            VIT["vit_encoder.py"]:::compute
        end
        subgraph "Training/Inference"
            MW_TR["trainer.py"]:::compute
            MW_DET["detector.py"]:::compute
        end
        MW_CP["Checkpoints"]:::storage
    end

    %% Inference & Serving
    subgraph "Inference/Serving"
        I2L_INF["Image2LaTeX Inference"]:::compute
        MS_INF["MathSolver Inference"]:::compute
        MW_INF["MathWriting Inference"]:::compute
    end

    %% Web Demo
    subgraph "Web Demo"
        HOME["home.html"]:::service
        SCRIPT["script.js"]:::service
        STYLE["styles.css"]:::service
    end

    %% Entry Point
    RUN["run.py"]:::compute

    %% Connections
    RAW_DATA --> PRE_RENDER
    RAW_DATA --> INKML_LOADER
    METADATA --> INKML_LOADER

    PRE_RENDER --> INKML_LOADER
    RESIZE_IMG --> INK_RENDER
    VIS_BATCH --> STROKE_RENDER
    API_WRAPPER --> SHARED_MAIN

    INKML_LOADER --> I2L_DM
    INK_RENDER --> I2L_DP
    STROKE_RENDER --> I2L_DP
    SYMBOL_EXT --> I2L_DP
    BBOX_LOADER --> I2L_DP

    I2L_DM --> I2L_DS
    I2L_DP --> I2L_DS
    I2L_TOK --> I2L_DS

    I2L_DS --> I2L_MODEL
    I2L_DL --> I2L_MODEL

    I2L_MODEL --> I2L_TR
    I2L_POS --> I2L_TR

    I2L_TR --> I2L_CP
    I2L_CP --> I2L_INF

    MS_TOK --> MS_CD
    MS_CD --> MS_DS
    MS_DS --> MS_MODEL
    MS_MODEL --> MS_TR
    MS_TR --> MS_CP
    MS_CP --> MS_INF

    MW_TOK --> MW_DS
    MW_TRANS --> MW_DS
    MW_DS --> MW_MODEL
    SWIN --> MW_MODEL
    VIT --> MW_MODEL
    MW_MODEL --> MW_TR
    MW_TR --> MW_CP
    MW_CP --> MW_INF

    I2L_INF --> HOME
    MS_INF --> HOME
    MW_INF --> HOME

    RUN -.-> I2L_TR
    RUN -.-> MS_TR
    RUN -.-> MW_TR

    %% Click Events
    click PRE_RENDER "https://github.com/nmnguyen130/mathsolver/blob/main/build/pre_render_dataset.py"
    click RESIZE_IMG "https://github.com/nmnguyen130/mathsolver/blob/main/build/resize_image.py"
    click VIS_BATCH "https://github.com/nmnguyen130/mathsolver/blob/main/build/visualize_batch.py"
    click API_WRAPPER "https://github.com/nmnguyen130/mathsolver/blob/main/build/api.py"
    click RUN "https://github.com/nmnguyen130/mathsolver/blob/main/run.py"

    click INKML_LOADER "https://github.com/nmnguyen130/mathsolver/blob/main/src/shared/preprocessing/inkml_loader.py"
    click INK_RENDER "https://github.com/nmnguyen130/mathsolver/blob/main/src/shared/preprocessing/ink_renderer.py"
    click STROKE_RENDER "https://github.com/nmnguyen130/mathsolver/blob/main/src/shared/preprocessing/stroke_renderer.py"
    click BBOX_LOADER "https://github.com/nmnguyen130/mathsolver/blob/main/src/shared/preprocessing/bbox_loader.py"
    click SYMBOL_EXT "https://github.com/nmnguyen130/mathsolver/blob/main/src/shared/preprocessing/symbol_extractor.py"
    click SHARED_MAIN "https://github.com/nmnguyen130/mathsolver/blob/main/src/shared/preprocessing/main.py"

    click I2L_DM "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/preprocessing/data_manager.py"
    click I2L_DP "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/preprocessing/data_processing.py"
    click I2L_FE "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/preprocessing/filter_error_image.py"
    click I2L_GL "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/preprocessing/generate_latex_file.py"
    click I2L_LP "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/preprocessing/latex_processing.py"
    click I2L_TOK "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/preprocessing/tokenizer.py"
    click I2L_DS "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/datamodule/dataset.py"
    click I2L_DL "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/datamodule/dataloader.py"
    click I2L_MWDL "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/datamodule/mw_dataloader.py"
    click I2L_MODEL "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/models/model.py"
    click I2L_POS "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/models/pos_model.py"
    click I2L_TR "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/scripts/trainer.py"
    click I2L_MWTR "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/scripts/mw_trainer.py"
    click I2L_CP "https://github.com/nmnguyen130/mathsolver/blob/main/src/image2latex/checkpoints/vocab.txt"

    click MS_TOK "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathsolver/preprocessing/tokenizer.py"
    click MS_CD "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathsolver/datamodule/create_dataset.py"
    click MS_DS "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathsolver/datamodule/dataset.py"
    click MS_DL "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathsolver/datamodule/dataloader.py"
    click MS_MODEL "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathsolver/models/model.py"
    click MS_TR "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathsolver/scripts/trainer.py"
    click MS_SOLVER "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathsolver/scripts/solver.py"
    click MS_CP "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathsolver/checkpoints/model_only.pt"

    click MW_TOK "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathwriting/preprocessing/tokenizer.py"
    click MW_DS "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathwriting/datamodule/dataset.py"
    click MW_DL "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathwriting/datamodule/dataloader.py"
    click MW_TRANS "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathwriting/datamodule/transforms.py"
    click MW_MODEL "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathwriting/models/model.py"
    click SWIN "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathwriting/models/swin_encoder.py"
    click VIT "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathwriting/models/vit_encoder.py"
    click MW_TR "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathwriting/scripts/trainer.py"
    click MW_DET "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathwriting/scripts/detector.py"
    click MW_CP "https://github.com/nmnguyen130/mathsolver/blob/main/src/mathwriting/checkpoints/model_only.pt"

    click HOME "https://github.com/nmnguyen130/mathsolver/blob/main/web/home.html"
    click SCRIPT "https://github.com/nmnguyen130/mathsolver/blob/main/web/script.js"
    click STYLE "https://github.com/nmnguyen130/mathsolver/blob/main/web/styles.css"

    %% Styles
    classDef data fill:#ffe4b5,stroke:#cc9900
    classDef compute fill:#cce5ff,stroke:#6699cc
    classDef storage fill:#d5f5e3,stroke:#28a745
    classDef service fill:#fbe4e1,stroke:#e5534b
```
