How ishanpandey15/datascienceproject works
Project type and purpose
• This is an end-to-end data science application with two main “faces”:
• A modular ML pipeline (data ingestion → validation → transformation → training → evaluation → prediction)
• A lightweight web UI (Flask) that serves model predictions
• The README confirms that it’s a full-stack data science project with notebooks, config files, and an app entrypoint.
File structure
• Top-level:
• config/config.yaml, params.yaml, schema.yaml → central configuration and schema definitions
• research/*.ipynb → exploratory work, pipeline prototyping
• src/datascience/components → individual processing stages
• src/datascience/pipeline → orchestrated pipelines per stage plus prediction
• src/datascience/config, entity, constants, utils → support modules (config loader, DTOs, constants, helpers)
• main.py → command-line orchestrator for the full ML pipeline
• app.py & templates/ → Flask-based web front end for live predictions
• Dockerfile, setup.py, requirements.txt → packaging and containerization
• .github/workflows → CI/CD definitions (likely auto-run tests, lint, build, deploy)
README for additional insights
• Clear 5-stage workflow: ingestion, validation, transformation, trainer, evaluation
• MLflow & DagsHub mentioned for experiment tracking
• Steps emphasize updating config, schema, params, entities, components, pipeline, and main.py
1. Data Sources & Config
• Raw data (CSV, database, external API)
• config/config.yaml, params.yaml, schema.yaml
2. Configuration Manager
• src/datascience/config/configuration.py → reads YAMLs and provides settings
3. Pipeline Orchestrator
• main.py → kicks off the full end-to-end workflow
4. Component Modules
• Data Ingestion
• Data Validation
• Data Transformation
• Model Trainer
• Model Evaluator
• Prediction Pipeline (used by app.py)
5. Model Artifact Store & Tracking
• MLflow server (or local folder)
• DagsHub remote repo
6. Web UI
• Flask app (app.py)
• Jinja2 templates (templates/index.html, templates/results.html)
7. External Services
• MLflow Tracking Server
• DagsHub
b. Relationships and interactions
• Trainer writes model artifacts + metrics to MLflow / DagsHub
• Evaluator reads artifacts, logs evaluation to tracking server
• Prediction Pipeline reads the final model artifact
• Web UI sends input to Prediction Pipeline → returns results to user
• All components load parameters via Configuration Manager
c. Architectural patterns / design principles
• Modular, component-based pipeline (single responsibility per component)
• Config-driven (YAML-first approach)
• Layered directories: config, entity (DTOs), components, pipeline, utils
• Experiment tracking integration (MLflow / DagsHub)
• Separation of concerns: ML pipeline vs. serving layer
d. Key technologies / frameworks / libraries
• Python 3.x
• Flask (app.py + Jinja2 templates)
• PyYAML for config
• Pandas, scikit-learn (in components)
• MLflow, DagsHub for experiment tracking
• Docker for containerization
• GitHub Actions for CI/CD
Final recommendations
• Explicitly indicate where each component reads from and writes to (e.g., ingested raw files, transformed data folder, model.pkl in artifact store)