# nicolasrossard/deep_learning_pytorch architecture

> Architecture diagram of the GitHub repository nicolasrossard/deep_learning_pytorch, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nicolasrossard/deep_learning_pytorch
- Repository: https://github.com/nicolasrossard/deep_learning_pytorch

## Overview

The project is a deep learning pipeline using PyTorch to build and test Convolutional Neural Networks on a satellite imagery dataset, with iterative improvements over several versions (v3 through v6). Each version shows evolutionary enhancements (from simple prediction in v3 to a full pipeline including dataset cleaning, training, model execution, and post-processing to remove duplicate detected ships in v6).

##### Main Components
- Data Ingestion & Acquisition:
- An external component (the Kaggle dataset) that supplies data. A note can mention that the dataset is initially acquired (e.g., shipsnet.json in v3).
- Data Cleaning & Preprocessing:
- Modules that clean and prepare the dataset (e.g., clean_v4.py, clean_v5.py, and data augmentation steps in v6).
- Data Storage:
- A component for Minio storage, as seen in v4/minioClient_v4.py, v5/minioClient_v5.py, etc. This illustrates how the dataset is managed between cleaning and training processes.
- Model Training & Tuning:
- Model Inference / Execution:
- For v6, show a module for executing the trained model on new satellite images (execute_model/main_v6_run.py) and retrieving detection results.
- Post-Processing:
- A module (execute_model/delete_duplicates_run_v6.py) that refines the results by eliminating duplicates (e.g., duplicate ship detections).
- Results & Performance Records:
- Components for saving and analyzing the model performance (folders for results in v5 and v6, and notebooks like models_trains_performances.ipynb).

##### Relationships and Data Flow
- Cleaned data being stored in Minio, from which the training modules retrieve the dataset.
- Link the training process to subsequent performance evaluation and storage of performance metrics.
- For v6, present a parallel pipeline where once training is complete, the trained model is used for inference on satellite images, and results are then fed into a post-processing module to remove duplicates.
- That components like the “infos” and “kaggle” scripts are shown as auxiliary modules that provide configuration, logging, or dataset-specific handling.

##### Architectural Patterns
- Pipeline (or “staged”) nature of the system design: data ingested → data cleaning/preprocessing → storage → training → inference → post-processing.
- Micro-level separation between training and execution in v6, reflecting a modular design that allows re-use or independent development.
- For each component, indicate any important design principles (e.g., modularity and separation of concerns) that help in isolating training logic from data manipulation and post-processing.

##### Specify Relevant Technologies and Frameworks
- Mention Python as the primary programming language, and note the use of Pandas for performance data management.
- Use of Minio for object storage in certain versions.
- That the project uses Jupyter Notebook for performance visualization (models_trains_performances.ipynb).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Modules Subgraph
    subgraph "Data Modules"
        A1["Data Acquisition (Kaggle v4)"]:::ingestion
        A2["Data Acquisition (Kaggle v5)"]:::ingestion
        A3["Data Acquisition (Kaggle v6)"]:::ingestion

        B1["Data Cleaning (v4)"]:::cleaning
        B2["Data Cleaning (v5)"]:::cleaning
        B3["Data Cleaning (v6)"]:::cleaning

        C1["Object Storage (Minio v4)"]:::storage
        C2["Object Storage (Minio v5)"]:::storage
        C3["Object Storage (Minio v6)"]:::storage

        D1["Auxiliary (Infos v4)"]:::auxiliary
        D2["Auxiliary (Infos v5)"]:::auxiliary
        D3["Auxiliary (Infos v6)"]:::auxiliary
    end

    %% Data Modules Connections
    A1 -->|"feeds"| B1
    A2 -->|"feeds"| B2
    A3 -->|"feeds"| B3

    B1 -->|"stores cleaned data in"| C1
    B2 -->|"stores cleaned data in"| C2
    B3 -->|"stores cleaned data in"| C3

    D1 ---|"configures"| B1
    D2 ---|"configures"| B2
    D3 ---|"configures"| B3

    %% Model Training & Tuning Subgraph
    subgraph "Model Training & Tuning"
        E1["CNN Training (v3)"]:::training
        E2["CNN Training (v4)"]:::training
        E3["CNN Training (v5)"]:::training
        E4["CNN Training (v6)"]:::training
        F1["Performance Tracking (Results v5)"]:::training
        F2["Performance Tracking (Results v6)"]:::training
    end

    %% Model Training Connections
    A1 -->|"baseline dataset"| E1
    C1 -->|"provides data to"| E2
    C2 -->|"provides data to"| E3
    C3 -->|"provides data to"| E4

    D2 ---|"configuration"| E3
    D3 ---|"configuration"| E4

    E3 -->|"saves metrics to"| F1
    E4 -->|"saves metrics to"| F2

    %% Inference & Post-Processing Subgraph (v6)
    subgraph "Inference & Post-Processing (v6)"
        G1["Model Inference (v6)"]:::inference
        G2["Post-Processing (Remove Duplicates)"]:::postprocessing
    end

    %% Inference & Post-Processing Connections
    E4 -->|"triggers"| G1
    G1 -->|"refines results using"| G2

    %% Styles for nodes
    classDef ingestion fill:#cfe2ff,stroke:#084298,stroke-width:2px;
    classDef cleaning fill:#d1e7dd,stroke:#0f5132,stroke-width:2px;
    classDef storage fill:#e2e3e5,stroke:#6c757d,stroke-width:2px;
    classDef training fill:#fff3cd,stroke:#ffc107,stroke-width:2px;
    classDef inference fill:#f8d7da,stroke:#842029,stroke-width:2px;
    classDef postprocessing fill:#ffe5d9,stroke:#e8590c,stroke-width:2px;
    classDef auxiliary fill:#d1c4e9,stroke:#512da8,stroke-width:2px;

    %% Click Events for Data Acquisition
    click A1 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v4/kaggle_v4.py"
    click A2 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v5/kaggle_v5.py"
    click A3 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v6/train_models/kaggle_v6.py"

    %% Click Events for Data Cleaning
    click B1 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v4/clean_v4.py"
    click B2 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v5/clean_v5.py"
    click B3 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v6/train_models/clean_v6.py"

    %% Click Events for Data Storage
    click C1 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v4/minioClient_v4.py"
    click C2 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v5/minioClient_v5.py"
    click C3 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v6/train_models/minioClient_v6.py"

    %% Click Events for Auxiliary Components
    click D1 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v4/infos_v4.py"
    click D2 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v5/infos_v5.py"
    click D3 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v6/train_models/infos_v6.py"

    %% Click Events for Model Training
    click E1 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v3/cnn_v3.py"
    click E2 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v4/cnn_v4.py"
    click E3 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v5/cnn_v5.py"
    click E4 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v6/train_models/cnn_v6.py"
    
    %% Click Events for Performance Tracking
    click F1 "https://github.com/nicolasrossard/deep_learning_pytorch/tree/master/v5/results_v5"
    click F2 "https://github.com/nicolasrossard/deep_learning_pytorch/tree/master/v6/train_models/results_v6"

    %% Click Events for Inference & Post-Processing
    click G1 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v6/execute_model/main_v6_run.py"
    click G2 "https://github.com/nicolasrossard/deep_learning_pytorch/blob/master/v6/execute_model/delete_duplicates_run_v6.py"
```
