How nmnguyen130/mathsolver works
Project type and purpose
• Type: A machine-learning pipeline with three specialized subsystems (image2latex, mathsolver, mathwriting) plus shared preprocessing utilities and a small web demo.
• Purpose: Ingest handwritten mathematical expression data (InkML), preprocess it, train and evaluate models for (a) LaTeX transcription, (b) equation solving, (c) handwriting detection/recognition, and optionally serve results via a simple web UI.
File structure
• build/ (scripts for dataset preparation and visualization)
• run.py (entry point orchestration)
• src/
– image2latex/ (data modules, preprocessing, model definitions, trainers)
– mathsolver/ (data modules, tokenizer, solver/trainer scripts)
– mathwriting/ (data modules, encoders, detector/trainer)
– shared/preprocessing/ (common inkml loaders, renderers, symbol extractors)
• web/ (static HTML/CSS/JS demo)
• requirements.txt, checkpoints folders (serialized model artifacts)
README for insights
• Data: InkML files split into train/valid/test/symbols/synthetic with metadata.
• Workflow: Preprocess raw InkML → generate tokenized datasets → train model → produce checkpoints → run inference.
• References: Im2LaTeX, Google Research MathWriting dataset.
a. Main components
1) Data Sources
• Raw InkML dataset (train/valid/test/synthetic)
• JSONL metadata (synthetic-bboxes.jsonl, symbols.jsonl)
2) Preprocessing Layer
• shared/preprocessing: InkML loader, stroke renderer, symbol extractor
• image2latex/preprocessing, mathsolver/preprocessing, mathwriting/preprocessing
3) DataModule Layer
• Dataset constructors (dataset.py)
• Dataloaders (dataloader.py, mw_dataloader.py)
4) Model Layer
• image2latex.models (seq2seq POS and baseline)
• mathsolver.models (solver network)
• mathwriting.models (Swin/VIT encoders + detector)
5) Training Orchestration
• image2latex/scripts/{trainer,mw_trainer}.py
• mathsolver/scripts/trainer.py
• mathwriting/scripts/trainer.py
• run.py (high-level runner)
6) Checkpoint Storage
• src/*/checkpoints/*.pt, vocab.txt
7) Inference/Serving
• image2latex/scripts/… inference logic
• mathsolver/scripts/solver.py
• mathwriting/scripts/detector.py
• web/ (home.html, script.js, styles.css)
8) Build Utilities
• build/*.py (pre_render_dataset, resize_image, visualize_batch, api wrapper)
b. Relationships & Data‐flows
– Data Sources → Preprocessing
– Preprocessing → DataModule → Model Training
– Model Training ↔ Model (iterative updates)
– Model Training → Checkpoints → Inference/Serving
– Inference/Serving → Web UI (optional)
• External dependencies (Python, PyTorch, Lightning, libraries in requirements.txt)
c. Architectural patterns & principles
• Clear separation of concerns:
– Data ingestion vs. preprocessing vs. modeling vs. serving
• Modular design: three parallel subprojects sharing common utilities
• Pipeline pattern: sequential stages connected by well-defined interfaces (Dataset → Dataloader → Model)
• Configurable entry points: individual scripts + unified run.py
d. Technologies/frameworks
• Python 3.x, PyTorch (or TensorFlow)
• Lightning/PyTorch Lightning (implied by trainer scripts)
• Web: HTML/CSS/JS for lightweight demo
• InkML parsing libraries
Diagramming conventions
– Another for compute components (rectangles for scripts/libraries)