# kcdichdangu/kc4.0_multilingualnmt architecture

> Architecture diagram of the GitHub repository kcdichdangu/kc4.0_multilingualnmt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kcdichdangu/kc4.0_multilingualnmt
- Repository: https://github.com/kcdichdangu/kc4.0_multilingualnmt

## Overview

##### Project type and purpose
- This is an open-source Neural Machine Translation (NMT) toolkit with both CLI and web-based interfaces.
- Core features: data loading, model definition (Transformer by default), training, inference (beam search, greedy, sampling), evaluation (BLEU), and a Flask-based web UI.

##### File structure at a high level
- bin/ – CLI entry points (train/infer via bin/main.py; web serve via bin/serve.py; flask support).
- modules/ – Core libraries:
- config & constants (global settings)
- loader (data ingestion pipelines)
- optim (optimizers & schedulers)
- inference (decoding strategies)
- prototypes (abstract interfaces)
- layers/ & models/ – Neural network building blocks and model definitions (default, Transformer).
- utils/ – Utilities (logging, metrics, data I/O, saving).
- data/ – Raw and processed datasets.
- web/ – Flask application: templates, static assets (CSS/JS/fonts/images), config.yml for UI settings.
- third-party/ – External scripts (multi-bleu.perl).
- config/ – YAML/JSON config files for experiments.

##### README for insights
- Training workflow: prepare data → edit YAML config → `python -m bin.main train …`
- Inference: `python -m bin.main infer …`
- Web service: likely `python -m bin.serve` or `bin/flask.py`
- Evaluation via third-party BLEU script.
- Technologies: Python ≥3.6, PyTorch (torchtext), Flask, YAML, Perl script.

##### Main components
a. Data Layer
- Raw data on disk (data/)
- Tokenization & preprocessing (modules/loader)
b. Modeling Layer
- layers/ (positional encoding, attention, feed-forward)
- models/ (Default, Transformer)
c. Training & Optimization
- bin/main.py (entry point)
- modules/optim (Adam, AdaBelief, schedulers)
- Logging & checkpointing (utils/save, utils/logging)
d. Inference
- modules/inference (BeamSearch, GreedySearch, Sampling)
- bin/main.py → inference mode
e. Evaluation
- third-party/multi-bleu.perl
f. Web UI (Flask)
- bin/serve.py or bin/flask.py
- web/config.yml → server settings
- web/template (Jinja2 HTML templates)
- web/static (JS/CSS/images/fonts)
- Routes: translate_sentence, translate_file, manage users/files, authentication flows
g. Configuration
- config/ (prototype configs)
- modules/config.py → loads YAML/JSON into the system

##### Relationships and interactions
- Data flow: Disk → Loader → Model → Optimizer → Disk (checkpoints/logs)
- CLI ↔ core libraries (modules + models + utils)
- Web UI ↔ Flask app ↔ Inference modules ↔ Model artifacts on disk
- Evaluation script invoked post-inference
- Config files loaded at application startup (both CLI and Flask)

##### Architectural patterns & principles
- Layered architecture: separation between data ingestion, model definition, training/inference, and presentation (web).
- Modular design: interchangeable components (loader, optimizer, inference strategies).
- MVC for the web app: Flask controllers (bin/serve.py), templates (web/template), static view assets.
- Plugin-like extensibility: adding new optimizers, decoding strategies by implementing prototypes.

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data & I/O Layer"
        RawData["Raw Data<br/>(data/)"]:::data
        DefaultLoader["Default Loader<br/>(modules/loader/default_loader.py)"]:::data
        MultiLoader["Multilingual Loader<br/>(modules/loader/multilingual_loader.py)"]:::data
    end

    subgraph "Modeling Layer"
        LayerImpl["Core Layers<br/>(layers/)"]:::core
        LayerProto["Layer Prototypes<br/>(layers/prototypes.py)"]:::core
        DefaultModel["Default Model<br/>(models/default.py)"]:::core
        TransformerModel["Transformer Model<br/>(models/transformer.py)"]:::core
    end

    subgraph "Training & Optimization"
        CLI["CLI EntryPoint<br/>(bin/main.py)"]:::cli
        OptimAdam["Adam Optimizer<br/>(modules/optim/adam.py)"]:::cli
        OptimAdaBelief["AdaBelief Optimizer<br/>(modules/optim/adabelief.py)"]:::cli
        Scheduler["LR Scheduler<br/>(modules/optim/scheduler.py)"]:::cli
        LogUtil["Logging Utility<br/>(utils/logging.py)"]:::cli
        SaveUtil["Checkpoint Save Utility<br/>(utils/save.py)"]:::cli
        ConfigModule["Config Loader<br/>(modules/config.py)"]:::cli
        ConstModule["Global Constants<br/>(modules/constants.py)"]:::cli
    end

    subgraph "Inference"
        BeamSearch["Beam Search<br/>(modules/inference/beam_search.py)"]:::core
        GreedySearch["Greedy Search<br/>(modules/inference/greedy_search.py)"]:::core
        Sampling["Sampling Strategy<br/>(modules/inference/sampling_temperature.py)"]:::core
        DecodeProto["Decode Interfaces<br/>(modules/inference/decode_strategy.py)"]:::core
    end

    subgraph "Web Service"
        ServeCLI["Flask Serve Entry<br/>(bin/serve.py)"]:::web
        FlaskCLI["Flask CLI Entry<br/>(bin/flask.py)"]:::web
        WebConfig["Web Config<br/>(web/config.yml)"]:::web
        WebRoutes["Flask Controllers<br/>(web/direct_translate.py)"]:::web
        Templates["Jinja2 Templates<br/>(web/template/*.html)"]:::web
        StaticCSS["CSS Assets<br/>(web/static/css/)"]:::web
        StaticJS["JS Assets<br/>(web/static/js/)"]:::web
        StaticFonts["Fonts<br/>(web/static/fonts/)"]:::web
        StaticImages["Images<br/>(web/static/images/)"]:::web
    end

    subgraph "Configuration Files"
        ExpYML["Experiment Config<br/>(config/bilingual_prototype.yml)"]:::external
        ExpJSON["Prototype Config<br/>(config/prototype.json)"]:::external
    end

    subgraph "External Tools"
        BLEU["BLEU Script<br/>(third-party/multi-bleu.perl)"]:::external
    end

    %% Data flow
    RawData -->|"load raw"| DefaultLoader
    RawData -->|"load raw"| MultiLoader
    DefaultLoader -->|"preprocessed data"| LayerImpl
    MultiLoader -->|"preprocessed data"| LayerImpl

    %% Modeling connections
    LayerImpl --> LayerProto
    LayerProto --> DefaultModel
    LayerProto --> TransformerModel

    %% Training workflow
    CLI -->|"loads"| ConfigModule
    CLI -->|"uses constants"| ConstModule
    CLI -->|"preprocess"| DefaultLoader
    CLI -->|"builds model"| DefaultModel
    CLI -->|"builds model"| TransformerModel
    CLI -->|"optimize"| OptimAdam
    CLI --> OptimAdaBelief
    CLI --> Scheduler
    OptimAdam --> SaveUtil
    OptimAdaBelief --> SaveUtil
    SaveUtil --> RawData
    CLI --> LogUtil

    %% Inference workflow
    CLI -->|"infer mode"| DecodeProto
    DecodeProto --> BeamSearch
    DecodeProto --> GreedySearch
    DecodeProto --> Sampling
    BeamSearch --> DefaultModel
    GreedySearch --> DefaultModel
    Sampling --> DefaultModel
    CLI -->|"invoke BLEU"| BLEU

    %% Web service flow
    ServeCLI --> WebConfig
    FlaskCLI --> WebConfig
    ServeCLI --> WebRoutes
    FlaskCLI --> WebRoutes
    WebRoutes --> DecodeProto
    WebRoutes --> DefaultModel
    WebRoutes --> TransformerModel
    WebRoutes --> BLEU
    WebRoutes --> Templates
    WebRoutes --> StaticCSS
    WebRoutes --> StaticJS
    WebRoutes --> StaticFonts
    WebRoutes --> StaticImages
    WebRoutes --> ConfigModule
    ConfigModule --> ExpYML
    ConfigModule --> ExpJSON

    %% Click Events
    click RawData "https://github.com/kcdichdangu/kc4.0_multilingualnmt/tree/master/data/"
    click DefaultLoader "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/modules/loader/default_loader.py"
    click MultiLoader "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/modules/loader/multilingual_loader.py"
    click LayerImpl "https://github.com/kcdichdangu/kc4.0_multilingualnmt/tree/master/layers/"
    click LayerProto "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/layers/prototypes.py"
    click DefaultModel "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/models/default.py"
    click TransformerModel "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/models/transformer.py"
    click CLI "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/bin/main.py"
    click OptimAdam "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/modules/optim/adam.py"
    click OptimAdaBelief "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/modules/optim/adabelief.py"
    click Scheduler "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/modules/optim/scheduler.py"
    click LogUtil "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/utils/logging.py"
    click SaveUtil "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/utils/save.py"
    click BeamSearch "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/modules/inference/beam_search.py"
    click GreedySearch "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/modules/inference/greedy_search.py"
    click Sampling "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/modules/inference/sampling_temperature.py"
    click DecodeProto "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/modules/inference/decode_strategy.py"
    click ServeCLI "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/bin/serve.py"
    click FlaskCLI "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/bin/flask.py"
    click WebConfig "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/web/config.yml"
    click WebRoutes "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/web/direct_translate.py"
    click Templates "https://github.com/kcdichdangu/kc4.0_multilingualnmt/tree/master/web/template/"
    click StaticCSS "https://github.com/kcdichdangu/kc4.0_multilingualnmt/tree/master/web/static/css/"
    click StaticJS "https://github.com/kcdichdangu/kc4.0_multilingualnmt/tree/master/web/static/js/"
    click StaticFonts "https://github.com/kcdichdangu/kc4.0_multilingualnmt/tree/master/web/static/fonts/"
    click StaticImages "https://github.com/kcdichdangu/kc4.0_multilingualnmt/tree/master/web/static/images/"
    click ExpYML "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/config/bilingual_prototype.yml"
    click ExpJSON "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/config/prototype.json"
    click BLEU "https://github.com/kcdichdangu/kc4.0_multilingualnmt/blob/master/third-party/multi-bleu.perl"

    classDef data fill:#e0ffe0,stroke:#2d572c;
    classDef core fill:#e0e0ff,stroke:#2a2a72;
    classDef cli fill:#ffe0b3,stroke:#7a5a0a;
    classDef web fill:#f0e0ff,stroke:#703f9a;
    classDef external fill:#dddddd,stroke:#555555;
```
