How whuir/matinf works
Project Type and Purpose
• This is a research‐oriented Python/PyTorch codebase (not a web or enterprise app) that implements the MATINF dataset and model for multitask Chinese NLP.
• It provides data loading, vocabulary construction, a seq2seq model supporting classification, QA, and summarization, training orchestration (run.py), and evaluation (ROUGE).
File Structure Analysis
• Top‐level scripts:
• run.py – main entry point, switches training phases via a stage flag
• dataset.py – reading/parsing the MATINF data
• vocab.py – building and managing token→index mappings
• seq2seq_model.py – definition of the PyTorch multitask seq2seq model
• utils.py – shared helper functions (e.g., batching, logging)
• rouge/ – contained evaluation modules:
• rouge.py & rouge_score.py – compute ROUGE metrics
README Insights
• Confirms use of PyTorch, manual stage toggling, and that the pipeline is: load dataset → build vocab → train model → evaluate with ROUGE.
1. Data Ingestion Layer
• dataset.py
• External dataset source (downloaded MATINF files)
2. Vocabulary Management
• vocab.py
3. Model Definition
• seq2seq_model.py (multi‐head outputs for classification, QA, summarization)
4. Training Orchestrator
• run.py (reads stage, calls data loader, vocab builder, model trainer)
5. Evaluation Module
• rouge/ (rouge.py, rouge_score.py)
6. Utility Services
• utils.py (batching, logging, checkpointing)
b. Relationships & Data Flow
c. Architectural Patterns & Principles
• Modular pipeline – clear separation of data, vocab, model, training, evaluation
• Multitask learning – single seq2seq core with multiple output heads
• Configuration via flags (stage variable) to switch phases
d. Key Technologies & Frameworks
• Python 3.x
• PyTorch (tensor ops, backprop, optimizers)
• ROUGE evaluation scripts
• Standard Python modules (argparse, os, logging)