How freskoko/cycletrafficweather works
Major System Components
• Data Ingestion and Pre-Processing:
– This component is responsible for reading the raw CSV files (found in raw_data directories under both “app” and “src”).
– The “src/utils” modules (such as dataframe_handling.py, file_parsing.py, graphing.py, and models.py) implement the pre-processing, feature engineering, and even parts of the model building (handling outliers, converting wind direction to x/y components, etc.).
• Model Training and Prediction Service:
– The main training and prediction logic resides in “src/project.py”. Here the pre-processed data goes through exploratory data analysis, outlier treatments, and is used to train the RandomForestRegressor (along with experiments on different ML models).
– The processing pipeline includes using KNNImputer for missing values and outputting prediction data in “predictions.csv” and an “out” folder.
• Web Application / Frontend:
– The “app” folder contains application scripts (app.py and appmodels.py) and website assets (templates/home.html). This is the interface that allows users to enter data, view live predictions, and trigger the model (using a pre-built model from a pickle file as noted in the README).
– It has its own “raw_data” folder setup instructions, allowing the same raw data to be used for prediction when a user interacts through a web browser.
• Storage and Outputs:
– File storage components include the raw data directories (under both “src/raw_data” and “app/raw_data”), “figs” for graphs, and “out” directories for processed outputs & prediction results.
Data Flow and Interactions Between Components
• Raw Data is ingested from CSV files (weather and traffic data) and passed into the Pre-Processing Pipeline.
• The pre-processing functions (in src/utils) clean the data – for example, replacing malformed values, combining files, engineering features (e.g., hour, day, weekend/holiday flags, summer/winter, wind direction transforms) – and then save cleaned outputs.
• The Model Training component (triggered by src/project.py) reads in the processed data, trains multiple models (with RandomForestRegressor coming out on top), tunes parameters (like “n_estimators”, or KNNImputer’s n_neighbors) and generates predictions. This part also exports trained model artifacts (using pickle) to be loaded later by the web application.
• The Web Application loads the saved model and imputer (via pickle) so that when a user submits new input via the website (hosted by app/app.py and rendered through app/templates/home.html), the input data is pre-processed (again using similar routines) and fed to the model for on-demand predictions.
• Final outputs (e.g., predictions.csv, forecast graphs, feature importance tables) are stored in the “out” and “figs” directories.
Architectural Patterns and Important Technologies
• The system follows a separation of concerns:
• Data processing and ML training (in the “src” directory) are separated from the presentation layer (in the “app” folder).
• The code is organized in modules (using subdirectories for utils and raw_data) and distinguishes between training and serving aspects.
• The design leverages popular Python libraries (pandas, scikit-learn for modeling, and likely Flask or a similar framework for the web server — even though the specific framework isn’t named) to implement machine learning and web serving.
• The model persistence (saving with pickle) is central to bridging between a computationally heavy training phase and a lightweight web prediction service.