How gentlew1nd/pricespeculator works
Main Components
• Data Pipeline Modules:
• Data Ingestion: Look at components like src/mlProject/components/data_ingestion.py and pipeline/stage_01_data_ingestion.py along with artifacts/data_ingestion.
• Data Validation: Represent the data_validation components in both src/mlProject/components/data_validation.py and stage_02_data_validation.py, plus artifacts/data_validation.
• Data Transformation: Depicted via src/mlProject/components/data_transformation.py and stage_03_data_transformation.py and the corresponding artifacts/data_transformation.
• Model Training: From src/mlProject/components/model_trainer.py and stage_04_model_trainer.py (also artifact/model_trainer), including tools like scaler and model.joblib.
• Model Evaluation: Through src/mlProject/components/model_evaluation.py and stage_05_model_evaluation.py, connecting to artifacts/model_evaluation.
• Configuration Management:
• Includes config files (config/config.yaml, params.yaml, schema.yaml) and the configuration management modules in src/mlProject/config.
• Web Application Layer:
• Backend: app.py, main.py that likely manage routing or orchestration.
• Frontend: The templates folder (index.html, results.html) and the static folder (assets, js, css) to render the UI.
• Research and Notebooks: Represent the research directory housing Jupyter notebooks for exploratory data analysis, trials, etc.
• Artifacts: A dedicated folder structure storing intermediate files, serialized objects (joblibs), and processed data which interface with the ML pipeline.
Relationships and Data Flow
• Pipeline Flow:
• Data Collection/Ingestion → Data Validation → Data Transformation → Model Training → Model Evaluation.
• Artifact creation at each stage (such as CSVs, joblib files, metrics.json).
• Integration with the Web App:
• Once the ML model is trained and evaluated, the prediction pipeline (src/mlProject/pipeline/prediction.py) interfaces with the web application (app.py) to generate and display results.
• Feedback/Updates:
• A feedback loop where outputs of training/evaluation might influence updates to configuration or further research iterations.
Architectural Patterns and Design Principles
• Modularity and Separation of Concerns: Each pipeline stage is a self-contained component. The src folder encapsulates the domain logic (components, configuration, pipeline, utilities).
• Layered Architecture: Divide the system into a backend service (ML processing) and a presentation layer (web UI). Clearly delineate how data moves from raw input in “artifacts” and “config” to processed predictions rendered for the end-user.
• Extensibility: Indicate how the design allows new components to be plugged in (for instance, incorporating new validation checks or transformation methods) without overhauling the system.
Specify Relevant Technologies/Frameworks
• Python: The core programming language.
• Joblib: Used for model and vectorizer serialization (seen in artifacts).
• YAML: For configuration and parameters.
• Jupyter Notebooks: Used for research and experimentation (research folder).
• Possibly Flask (or a similar lightweight framework): Deduce from the presence of app.py, templates, and static folders.
Elements
• Each major component (Data Ingestion, Data Validation, Data Transformation, Model Training, Model Evaluation, Prediction Pipeline, Configuration Manager, Web Frontend, Artifacts Storage).