# pafruchtenreich/ml-project-transformers-mlops architecture

> Architecture diagram of the GitHub repository pafruchtenreich/ml-project-transformers-mlops, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pafruchtenreich/ml-project-transformers-mlops
- Repository: https://github.com/pafruchtenreich/ml-project-transformers-mlops

## Overview

##### Main Components
- Main Driver (Orchestrator): Represented by main.py. This is the entry point that coordinates the whole pipeline – from data loading to training, evaluation, and generating predictions.
- Data Ingestion & Loading: Includes the modules in src/load_data (load_data.py and load_dataset_kaggle.py) that download and load the dataset.
- Data Preprocessing & Feature Engineering: Found under src/features, these modules (functions_preprocessing.py and tokenization.py) handle text cleaning and tokenization.
- Data Preparation Utilities: Modules such as create_dataloader.py (for converting data into batches) and create_scheduler.py (for scheduling training adjustments) are essential supporting components.
- Model Building & Training: Located in src/models (transformer.py for the model definition and train_models.py for the training process) where the Transformer architecture is implemented and trained.
- Evaluation: Under src/evaluation where model_evaluation.py assesses the model’s performance with metrics like ROUGE scores.
- Prediction/Inference: Handled by src/prediction (generate_summaries_transformer.py), this part is responsible for generating news summaries using the trained model.
- Utility and Setup: Modules like set_up_config_device.py and setup_logger.py configure device settings (e.g., GPU configurations) and logging. They are critical for running the code under different environments.
- Output Storage: The output folder stores the generated model predictions (e.g., predictions.parquet) and potentially other logs or interim results.

##### Relationships and Data Flow
- The flow starts with main.py initiating the process, which includes data retrieval from Kaggle via the load_data components.
- Data flows from ingestion to preprocessing – tokenization and other transformations ensure the inputs are ready for the model.
- The prepared data is then fed into the training pipeline (using dataloader, scheduler, and the transformer model) where the learning happens.
- Post-training, the evaluation module calculates metrics to validate performance.
- For generating summaries, the prediction module reads the data and invokes the already trained Transformer model.
- Outputs are stored in the output folder, completing the loop for later inspection and potential further analysis.

##### Architectural Patterns and Design Principles
- Modular Design: The project is broken down into distinct modules (data loading, preprocessing, training, evaluation, prediction) to keep concerns separated.
- Pipeline Flow: There’s a clear sequential flow from data ingestion to output generation, which can be diagrammed as linear stages with feedback loops (e.g., evaluation may trigger retraining).
- Reusability & Configuration: Modules like create_scheduler and setup_logger show that the design emphasizes reusability and ease of configuration.
- Command-Line Flexibility: With main.py accepting command-line arguments (e.g., --reload_data, --retrain_model), the system supports flexible execution paths.

##### Specific Technologies and Libraries
- Python as the programming language.
- Likely deep learning libraries (e.g., PyTorch or TensorFlow) used in src/models/transformer.py, although not explicitly stated, can be assumed.
- The usage of GPU acceleration (NVIDIA A100 as mentioned in the README) indicates that performance considerations and hardware utilizations are part of the design.

- Annotations: Add notes about specific configurations (like GPU usage) and external dependencies (like Kaggle dataset integration).

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Layer 1: Main"
        A["main.py"]:::main
    end

    subgraph "Utilities"
        L["set_up_config_device.py"]:::utility
        M["setup_logger.py"]:::utility
    end

    subgraph "Layer 2: Data Ingestion & Loading"
        B["load_data.py"]:::data
        C["load_dataset_kaggle.py"]:::data
    end

    subgraph "Layer 3: Data Preprocessing & Feature Engineering"
        D["functions_preprocessing.py"]:::data
        E["tokenization.py"]:::data
    end

    subgraph "Layer 4: Data Preparation Utilities"
        F["create_dataloader.py"]:::data
        G["create_scheduler.py"]:::data
    end

    subgraph "Layer 5: Model Building & Training"
        H["transformer.py"]:::model
        I["train_models.py"]:::model
    end

    subgraph "Layer 6: Evaluation"
        J["model_evaluation.py"]:::evaluation
    end

    subgraph "Layer 7: Prediction/Inference"
        K["generate_summaries_transformer.py"]:::prediction
    end

    subgraph "Layer 8: Output Storage"
        N["Output: model_predictions"]:::output
    end

    %% Connections
    A -->|"triggers"| B
    A -->|"triggers"| C
    A -->|"configures"| L
    A -->|"configures"| M
    B -->|"feeds data"| D
    C -->|"feeds data"| E
    D -->|"prepares"| F
    E -->|"prepares"| G
    F -->|"batches"| H
    G -->|"schedules"| I
    H -->|"defined in"| I
    I -->|"trains model"| J
    J -->|"evaluates"| K
    K -->|"outputs"| N
    J -.->|"feedback"| I

    %% Click Events
    click A "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/main.py"
    click B "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/load_data/load_data.py"
    click C "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/load_data/load_dataset_kaggle.py"
    click D "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/features/functions_preprocessing.py"
    click E "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/features/tokenization.py"
    click F "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/create_dataloader.py"
    click G "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/create_scheduler.py"
    click H "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/models/transformer.py"
    click I "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/models/train_models.py"
    click J "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/evaluation/model_evaluation.py"
    click K "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/prediction/generate_summaries_transformer.py"
    click L "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/set_up_config_device.py"
    click M "https://github.com/pafruchtenreich/ml-project-transformers-mlops/blob/master/src/setup_logger.py"
    click N "https://github.com/pafruchtenreich/ml-project-transformers-mlops/tree/master/output/model_predictions"

    %% Styles
    classDef main fill:#d3d3d3,stroke:#333,stroke-width:2px;
    classDef data fill:#ADD8E6,stroke:#000,stroke-width:1px;
    classDef model fill:#90EE90,stroke:#000,stroke-width:1px;
    classDef evaluation fill:#FFF3CD,stroke:#000,stroke-width:1px;
    classDef prediction fill:#F8D7DA,stroke:#000,stroke-width:1px;
    classDef utility fill:#E2D4F0,stroke:#000,stroke-width:1px;
    classDef output fill:#F0F0F0,stroke:#000,stroke-width:1px;
```
