How aaron2rodrigues/pp works
• Data Ingestion: Represented both as a dedicated component (data_ingestion.py) and its corresponding pipeline module (data_ingestion_pipeline.py). This component reads raw data and possibly interfaces with external data sources.
• Data Validation: Shown through the data_validation.py component and the dedicated data_validation_pipeline.py module. This part uses the schema (schema.yaml) to check incoming data quality.
• Data Transformation: Captured by data_transformation.py and its pipeline partner (data_transformation_pipeline.py) where the raw data is prepared for model training.
• Model Training: Found as model_trainer.py (with its pipeline implementation in model_trianer_pipeline.py). This module is responsible for creating a predictive model using the processed data.
• Model Evaluation: Illustrated via model_evaluation.py with its pipelining counterpart (model_evaluation_pipeline.py) for testing and evaluating model performances.
• Prediction Pipeline: There is also a prediction_pipeline.py which likely connects the trained model to a production inference service.
• Configuration Manager: This layer uses files like config.yaml in the config/ and src/Project1/config directories along with an entity definition (entity_config.py) to drive the project’s dynamic behavior.
• Common Utilities: Tools stored in the utils directory are supporting functions needed across components.
• Web Interface / API Layer: With templates (index.html and results.html) and app.py, there might be an interactive front-end or API endpoint to show results or interact with the model.
Component relationships and interactions
• Begin the flow by reading configuration and parameters (using config/config.yaml, params.yaml, and schema.yaml). Use the configuration manager (in src/Project1/config/configuration.py) to load and validate these settings.
• After transformation, the system splits into training and then evaluation. Note that the evaluation results could then be exposed via the prediction pipeline and ultimately through the web interface (index and results templates).
• The project also contains research notebooks that represent exploratory phases and may serve as a documentation or prototype source; however, they might not be “part” of the live architecture.
Architectural patterns and design principles
• Modularity: Notice how the project separates concerns clearly (components, pipelines, configuration, and utilities). This modular design allows for easy updates to individual stages.
• Config-driven behavior: Showing that the system’s behavior is driven by external YAML configuration files encourages reusability and flexibility.
• Pipeline Abstraction: The different pipelines (for ingestion, transformation, etc.) invoke their respective components in a sequential manner. This reflects a typical ETL (Extract, Transform, Load) framework commonly used in data science projects.
Relevant technologies and frameworks
• Python is the primary language, as seen by all the .py files.
• YAML for configuration management (config.yaml, schema.yaml, params.yaml).
• Docker for containerization (Dockerfile).
• Jupyter Notebooks for research and exploration (in the research directory).
• Basic web templating (HTML files in templates) possibly with a lightweight framework hinted at by app.py.
• GitHub Actions (under .github/workflows) indicate CI/CD integration.