How pgf3712/machine-learning---predicting-academic-success-and-dropout-risk-in-students- works
Main Components
• Data Storage and Ingestion:
– The “01_data” folder contains different stages of the dataset: raw data, processed data, training data, test data, and even a folder for images.
– Indicate separate sections for raw data ingestion, data processing/cleaning, and split datasets (training & testing).
• Data Exploration and Preprocessing:
– The “02_notebooks” directory holds multiple Jupyter notebooks for Exploratory Data Analysis (EDA) and data cleaning.
– Include a component for data validation, cleaning, and exploratory analysis, which feeds into the training process.
• Model Training and Evaluation:
– “03_src” contains scripts for data processing, training, and evaluation.
– The presence of multiple subdirectories in “03_src/training_evaluation_other_models” (covering supervised, unsupervised models, and neural networks) suggests that the system supports and compares various algorithms.
• Model Artifacts and Configuration:
– “04_models” stores the final and experimental model artifacts (e.g., .pkl files and a YAML configuration file).
– Show a dedicated component that manages model storage, versioning, and configuration.
• Web Application (Deployment):
– “05_app_streamlit” holds the Streamlit application, serving as the frontend where predictions and academic guidance are provided.
– Represent this as an interactive interface that consumes the model outputs to display predictions and recommendations.
• Documentation and Presentation:
– “06_docs” contains presentations and additional documentation that can be used for reporting or stakeholder communications.
– Optionally include a documentation component in the architecture.
Data and Workflow Relationships
– Start with Raw Data (01_data/01_raw) that flows into the Data Preprocessing stage (handled via notebooks and 03_src scripts).
– Preprocessed data is then split into training (01_data/03_train) and testing (01_data/04_test) sets.
– The Training module (03_src/02_training.py and its analogous files in alternative model folders) consumes the processed and training datasets.
– Model evaluation scripts (03_src/03_evaluation.py and related evaluation scripts in subfolders) compare different models, eventually selecting the winner (SVM in this case, as per the README).
– Final models and their configurations are saved in “04_models”, from where the Streamlit application reads the model for predictions.
– The web app (05_app_streamlit/app.py) acts as the consumer of the trained model outputs, integrating prediction results with career orientation information.
– Optionally, visualization outputs and additional images (from “01_data/05_images”) inform both internal validation and user interface elements.
Architectural Patterns and Design Principles
– The project follows a pipeline approach, where data flows from ingestion and preprocessing to model experimentation, training, evaluation, and deployment.
– There is a clear separation between notebooks for prototyping/EDA, and production-ready scripts in the “03_src” folder.
– The modular structure for training multiple models indicates an extensible design that allows adding or refining models without major restructuring.
– Separation of concerns is adhered to by isolating the web interface (Streamlit app) from data processing and model orchestration layers.
Technologies and Frameworks
– Python as the programming language.
– Libraries like scikit-learn, pandas, numpy, matplotlib, and seaborn for data manipulation, visualization, and ML.
– Streamlit for the web application interface.
– File formats such as pickle (.pkl) for model persistence and YAML for configuration.