# daverlon/engtoid architecture

> Architecture diagram of the GitHub repository daverlon/engtoid, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/daverlon/engtoid
- Repository: https://github.com/daverlon/engtoid

## Overview

##### Main Components
- Data Ingestion & Dataset Preparation:
- Components for dataset download and initial ingestion (see src/download_dataset.py).
- Data cleaning and preprocessing (src/clean_dataset.py and src/utils/preprocessing.py).
- Dataset encoding and tokenization (src/encode_dataset.py, src/generate_vocab.py, and src/dataset/ENGtoID.py).
- Configuration & Experiment Management:
- A configuration module (src/config.py) which centrally manages key parameters.
- Notebooks (src/ICT303 - Assignment 2.ipynb) as interactive interfaces to manage experiments.
- Model Development:
- Model definition files in src/models (model.py, seq2seq_lstm.py).
- LSTM seq2seq architecture used in translation.
- Training & Evaluation:
- Training scripts (src/train_model.py, src/train_seq2seq_lstm.py).
- Trainer implementations in src/trainers (trainer.py).
- Test scripts (src/test_dataset.py, src/test_preprocessing.py, src/test_trainer.py, src/test_valid.py) indicating unit or integration tests.
- Utility & Helper Functions:
- Utility functions (src/utils/coder.py, src/utils/metrics.py) that support the processing and evaluation, ensuring data flows and performance metrics are managed.

##### Out Relationships & Data Flow
- Dataset download feeding into cleaning scripts.
- Pass the cleaned data into encoding/tokenization.
- Generate vocabulary as a standalone step.
- Follow with the “Model Development” block:
- How the preprocessed, tokenized data enters the model training module.
- Training scripts with the trainer component that orchestrates the model’s learning process.
- Link “Configuration” as a central module that influences both data processing and the training modules.
- “Utility” functions as cross-cutting modules that provide support (e.g., metrics for evaluation, preprocessing wrappers).

##### Architectural Patterns & Frameworks
- Separation of concerns: Data preprocessing, model definition, training, and utility functions are all modularized.
- The project leverages a typical machine learning pipeline pattern: ingest data → preprocess → define model → train → evaluate.
- Although not explicitly a microservices architecture, the clear separation into modules (data, model, trainer, utilities) represents a loosely coupled design ideal for iterative experiments.

##### Additional Details
- Even though deployment isn’t explicitly mentioned, the setup script (src/setup.sh) hints at an automated setup process for the environment.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Ingestion & Preparation
    subgraph "Data Ingestion & Preparation"
        downloader["Dataset Downloader"]
        cleaner["Data Cleaner"]
        preprocessing["Preprocessing Utils"]
        encoder["Dataset Encoder"]
        vocab["Vocab Generator"]
        engtoid["ENG to ID Converter"]
    end

    %% Configuration & Experiment Management
    subgraph "Configuration & Experiment Management"
        config["Configuration Module"]
        notebook["Experiment Notebook"]
    end

    %% Model Development
    subgraph "Model Development"
        model["Model Definition"]
        seq2seq["Seq2Seq LSTM"]
    end

    %% Training & Evaluation
    subgraph "Training & Evaluation"
        train_model["Train Model"]
        train_seq2seq["Train Seq2Seq LSTM"]
        trainer["Trainer Module"]
        test_dataset["Test Dataset"]
        test_preprocessing["Test Preprocessing"]
        test_trainer["Test Trainer"]
        test_valid["Test Valid"]
    end

    %% Utility Functions
    subgraph "Utility Functions"
        coder["Utility Coder"]
        metrics["Utility Metrics"]
    end

    %% Data Pipeline Flow
    downloader -->|"feeds"| cleaner
    cleaner -->|"processes"| preprocessing
    preprocessing -->|"prepares"| encoder
    encoder -->|"generates"| vocab
    encoder -->|"converts"| engtoid

    %% Configuration Influence
    config ---|"configures"| downloader
    config ---|"configures"| cleaner
    config ---|"configures"| encoder
    config ---|"configures"| model
    config ---|"configures"| trainer
    notebook -->|"interactive"| config

    %% Model Development Flow
    encoder -->|"supplies data"| model
    vocab -->|"supports"| model
    engtoid -->|"supports"| model
    model -->|"specializes into"| seq2seq

    %% Training Flow
    train_model -->|"initiates"| trainer
    train_seq2seq -->|"initiates"| trainer
    seq2seq -->|"used by"| trainer
    trainer -->|"validates"| test_dataset
    trainer -->|"validates"| test_preprocessing
    trainer -->|"validates"| test_trainer
    trainer -->|"validates"| test_valid

    %% Utility Support
    coder -->|"assists"| preprocessing
    metrics -->|"monitors"| trainer

    %% Class Definitions
    classDef data fill:#AED6F1,stroke:#2471A3,stroke-width:2px;
    classDef config fill:#F9E79F,stroke:#F1C40F,stroke-width:2px;
    classDef model fill:#F5CBA7,stroke:#D35400,stroke-width:2px;
    classDef training fill:#A9DFBF,stroke:#239B56,stroke-width:2px;
    classDef utility fill:#D2B4DE,stroke:#6C3483,stroke-width:2px;

    class downloader,cleaner,preprocessing,encoder,vocab,engtoid data;
    class config,notebook config;
    class model,seq2seq model;
    class train_model,train_seq2seq,trainer,test_dataset,test_preprocessing,test_trainer,test_valid training;
    class coder,metrics utility;

    %% Click Events
    click downloader "https://github.com/daverlon/engtoid/blob/main/src/download_dataset.py"
    click cleaner "https://github.com/daverlon/engtoid/blob/main/src/clean_dataset.py"
    click preprocessing "https://github.com/daverlon/engtoid/blob/main/src/utils/preprocessing.py"
    click encoder "https://github.com/daverlon/engtoid/blob/main/src/encode_dataset.py"
    click vocab "https://github.com/daverlon/engtoid/blob/main/src/generate_vocab.py"
    click engtoid "https://github.com/daverlon/engtoid/blob/main/src/dataset/ENGtoID.py"
    click config "https://github.com/daverlon/engtoid/blob/main/src/config.py"
    click notebook "https://github.com/daverlon/engtoid/blob/main/src/ICT303 - Assignment 2.ipynb"
    click model "https://github.com/daverlon/engtoid/blob/main/src/models/model.py"
    click seq2seq "https://github.com/daverlon/engtoid/blob/main/src/models/seq2seq_lstm.py"
    click train_model "https://github.com/daverlon/engtoid/blob/main/src/train_model.py"
    click train_seq2seq "https://github.com/daverlon/engtoid/blob/main/src/train_seq2seq_lstm.py"
    click trainer "https://github.com/daverlon/engtoid/blob/main/src/trainers/trainer.py"
    click test_dataset "https://github.com/daverlon/engtoid/blob/main/src/test_dataset.py"
    click test_preprocessing "https://github.com/daverlon/engtoid/blob/main/src/test_preprocessing.py"
    click test_trainer "https://github.com/daverlon/engtoid/blob/main/src/test_trainer.py"
    click test_valid "https://github.com/daverlon/engtoid/blob/main/src/test_valid.py"
    click coder "https://github.com/daverlon/engtoid/blob/main/src/utils/coder.py"
    click metrics "https://github.com/daverlon/engtoid/blob/main/src/utils/metrics.py"
```
