# whuir/matinf architecture

> Architecture diagram of the GitHub repository whuir/matinf, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-05). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/whuir/matinf
- Repository: https://github.com/whuir/matinf

## Overview

##### Project Type and Purpose
- This is a research‐oriented Python/PyTorch codebase (not a web or enterprise app) that implements the MATINF dataset and model for multitask Chinese NLP.
- It provides data loading, vocabulary construction, a seq2seq model supporting classification, QA, and summarization, training orchestration (run.py), and evaluation (ROUGE).

##### File Structure Analysis
- Top‐level scripts:
- run.py – main entry point, switches training phases via a `stage` flag
- dataset.py – reading/parsing the MATINF data
- vocab.py – building and managing token→index mappings
- seq2seq_model.py – definition of the PyTorch multitask seq2seq model
- utils.py – shared helper functions (e.g., batching, logging)
- rouge/ – contained evaluation modules:
- rouge.py & rouge_score.py – compute ROUGE metrics

##### README Insights
- Confirms use of PyTorch, manual stage toggling, and that the pipeline is: load dataset → build vocab → train model → evaluate with ROUGE.

1. Data Ingestion Layer
- dataset.py
- External dataset source (downloaded MATINF files)
2. Vocabulary Management
- vocab.py
3. Model Definition
- seq2seq_model.py (multi‐head outputs for classification, QA, summarization)
4. Training Orchestrator
- run.py (reads `stage`, calls data loader, vocab builder, model trainer)
5. Evaluation Module
- rouge/ (rouge.py, rouge_score.py)
6. Utility Services
- utils.py (batching, logging, checkpointing)
b. Relationships & Data Flow
c. Architectural Patterns & Principles
- Modular pipeline – clear separation of data, vocab, model, training, evaluation
- Multitask learning – single seq2seq core with multiple output heads
- Configuration via flags (stage variable) to switch phases
d. Key Technologies & Frameworks
- Python 3.x
- PyTorch (tensor ops, backprop, optimizers)
- ROUGE evaluation scripts
- Standard Python modules (argparse, os, logging)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Resource
    ExternalDataSource(("External MATINF Dataset")):::external

    %% Data Ingestion Layer
    DatasetLoader["dataset.py"]:::data
    click DatasetLoader "https://github.com/whuir/matinf/blob/master/dataset.py"

    %% Vocabulary Management
    VocabManager["vocab.py"]:::vocab
    click VocabManager "https://github.com/whuir/matinf/blob/master/vocab.py"

    %% Model Definition
    Seq2SeqModel["seq2seq_model.py\n(Multitask Seq2Seq)"]:::model
    click Seq2SeqModel "https://github.com/whuir/matinf/blob/master/seq2seq_model.py"

    %% Training Orchestrator
    Orchestrator["run.py\n(Stage-based Entry)"]:::orchestrator
    click Orchestrator "https://github.com/whuir/matinf/blob/master/run.py"

    %% Utility Services
    Utils["utils.py\n(Batching, Logging,\nCheckpointing)"]:::util
    click Utils "https://github.com/whuir/matinf/blob/master/utils.py"

    %% Evaluation Module
    subgraph "Evaluation Module (ROUGE)" 
        RougeMain["rouge.py"]:::eval
        click RougeMain "https://github.com/whuir/matinf/blob/master/rouge/rouge.py"
        RougeScore["rouge_score.py"]:::eval
        click RougeScore "https://github.com/whuir/matinf/blob/master/rouge/rouge_score.py"
    end

    %% Data Flow
    ExternalDataSource -->|provide raw data| DatasetLoader
    DatasetLoader -->|produce examples| VocabManager
    VocabManager -->|produce indexed batches| Orchestrator
    Orchestrator -->|initialize model| Seq2SeqModel
    Orchestrator -->|run training loop| Seq2SeqModel
    Orchestrator -->|log & checkpoint| Utils
    Seq2SeqModel -->|trained model| Orchestrator
    Orchestrator -->|deliver trained model| RougeMain
    RougeMain -->|compute metrics| RougeScore

    %% Styles
    classDef external fill:#f9f,stroke:#333,stroke-width:2px;
    classDef data fill:#cce5ff,stroke:#004085,stroke-width:1px;
    classDef vocab fill:#d1ecf1,stroke:#0c5460,stroke-width:1px;
    classDef model fill:#d4edda,stroke:#155724,stroke-width:1px;
    classDef orchestrator fill:#e2e3e5,stroke:#6c757d,stroke-width:1px;
    classDef util fill:#f8f9fa,stroke:#6c757d,stroke-width:1px;
    classDef eval fill:#ffe5b4,stroke:#ff8c00,stroke-width:1px;
```
