How triquang26/ml-course-01 works
Main Components Identification
• Data Storage & Ingestion:
• Data is stored under the “data” directory (raw, processed, external).
• Scripts in src/data (make_dataset.py, preprocess.py and model-specific preprocessing scripts) convert raw data into a usable format.
• Feature Engineering:
• The “src/features” directory handles feature extraction and feature selection with specific scripts/modules for each model type.
• Model Implementations:
• Under “src/models”, each subdirectory (bayesian, decision_tree, genetic_algorithm, graphical_model, neural_network) holds the model-specific code.
• Each model has its own core implementation, training scripts (main.py), prediction scripts (predict.py), testing modules (test.py), and persistence routines.
• Experimentation & Reporting:
• Notebooks in the “notebooks” directory serve for exploratory data analysis, model development, and tutorials.
• Reports and figures are placed for visual analysis (reports, figures directories).
• Orchestration & Integration:
• At a higher level, “src/models/train_model.py” and “src/models/predict_model.py” serve as centralized entry points to train and run the different models.
• Testing & Validation:
• The “tests” directory holds unit tests to ensure data processing scripts and model routines are functioning correctly.
Relationships and Data Flow
• Data flow begins at the raw data level, then moves through preprocessing (src/data/preprocess/*) into processed datasets.
• Processed data connects to the feature engineering module (src/features/build_features.py and features_selection submodules) to extract the relevant features.
• The processed and engineered outputs are then fed into the model-specific training pipelines within “src/models/[model_type]”.
• Once trained, models are saved in “trained” subdirectories; predictions and evaluations are computed with the associated scripts.
• Notebooks reflect exploratory experiments, while reports compile the visualizations from the “reports” and “figures” directories.
• Testing modules validate the above processes end to end.
Architectural Patterns and Design Principles
• Modularity – each model is implemented in its own module, allowing for independent development and testing.
• Pipeline Architecture – clear sequential stages (data ingestion, preprocessing, feature engineering, modeling, evaluation) which mirror common ML workflows.
• Repository Layout reflects separation of concerns: notebooks for exploration, src for production code, separate directories for each workflow stage (models, data, features, visualization, tests).
Relevant Technologies and Frameworks
• Python 3.11+, with key libraries including scikit-learn, numpy, pandas, PyTorch (for neural networks), pgmpy (for graphical models).
• Jupyter Notebooks for experimentation.
• joblib and PyTorch’s native persistence for saving trained models.
Detailed Steps
• Break the pipeline into modules:
• Data Acquisition/Storage → Data Preprocessing (src/data) → Feature Engineering (src/features) → Model Training (src/models/{model_type}).
• Bayesian Models block: Show subcomponents like “Core Models”, “Persistence”, and “Evaluation”.
• Decision Trees block: Include “Main Training”, “Prediction”, “Test”.
• Genetic Algorithm Ensemble block: Highlight “Core GA Implementation”, “Ensemble Integration”, and “Trained Weights”.
• Neural Networks block: Indicate “Vision Transformer” with training, testing, and persistence.
• Also create an auxiliary branch for Notebooks/Reports that interact with each modeling component for documentation and visualization.
• Optionally, add a layer for "Testing and Validation" that touches all parts of the pipeline.