How nicolasrossard/deep_learning_pytorch works
The project is a deep learning pipeline using PyTorch to build and test Convolutional Neural Networks on a satellite imagery dataset, with iterative improvements over several versions (v3 through v6). Each version shows evolutionary enhancements (from simple prediction in v3 to a full pipeline including dataset cleaning, training, model execution, and post-processing to remove duplicate detected ships in v6).
Main Components
• Data Ingestion & Acquisition:
• An external component (the Kaggle dataset) that supplies data. A note can mention that the dataset is initially acquired (e.g., shipsnet.json in v3).
• Data Cleaning & Preprocessing:
• Modules that clean and prepare the dataset (e.g., clean_v4.py, clean_v5.py, and data augmentation steps in v6).
• Data Storage:
• A component for Minio storage, as seen in v4/minioClient_v4.py, v5/minioClient_v5.py, etc. This illustrates how the dataset is managed between cleaning and training processes.
• Model Training & Tuning:
• Model Inference / Execution:
• For v6, show a module for executing the trained model on new satellite images (execute_model/main_v6_run.py) and retrieving detection results.
• Post-Processing:
• A module (execute_model/delete_duplicates_run_v6.py) that refines the results by eliminating duplicates (e.g., duplicate ship detections).
• Results & Performance Records:
• Components for saving and analyzing the model performance (folders for results in v5 and v6, and notebooks like models_trains_performances.ipynb).
Relationships and Data Flow
• Cleaned data being stored in Minio, from which the training modules retrieve the dataset.
• Link the training process to subsequent performance evaluation and storage of performance metrics.
• For v6, present a parallel pipeline where once training is complete, the trained model is used for inference on satellite images, and results are then fed into a post-processing module to remove duplicates.
• That components like the “infos” and “kaggle” scripts are shown as auxiliary modules that provide configuration, logging, or dataset-specific handling.
Architectural Patterns
• Pipeline (or “staged”) nature of the system design: data ingested → data cleaning/preprocessing → storage → training → inference → post-processing.
• Micro-level separation between training and execution in v6, reflecting a modular design that allows re-use or independent development.
• For each component, indicate any important design principles (e.g., modularity and separation of concerns) that help in isolating training logic from data manipulation and post-processing.
Specify Relevant Technologies and Frameworks
• Mention Python as the primary programming language, and note the use of Pandas for performance data management.
• Use of Minio for object storage in certain versions.
• That the project uses Jupyter Notebook for performance visualization (models_trains_performances.ipynb).