How larrysman/machine_learning_pipeline_with_dvc works
Project type and purpose
• This is an end-to-end, modular machine-learning pipeline project built in Python. It uses DVC to define, execute, and version each stage of the data workflow (a DAG of ETL → feature engineering → model training/evaluation → deployment). Git tracks code and DVC tracks data, models, and metrics.
File structure
• Top-level:
• .dvc/, .dvcignore, dvc.yaml & dvc.lock: DVC configuration and pipeline definition files.
• requirements.txt: Python dependencies.
• src/: six single-responsibility Python scripts, each implementing one pipeline stage (data_collection, data_preparation, feature_engineering, data_preprocessing, model_building, model_evaluation).
• No traditional “frontend/backend” split—this is a data-processing pipeline. The core components are the code modules, the DVC orchestration layer, the raw/intermediate data store, the model artifacts store, and the metrics/artifacts output.
README for additional insights
• External services: local DVC remote (dvcstore), but could be S3/GDrive/Azure in other contexts.
Detail without over-splitting
• Of the six pipeline stages as individual components.
• DVC’s two roles: orchestration (reading dvc.yaml → running stages) and artifact versioning (push/pull to remote).
• Feedback loop if a stage output changes and the cache invalidates downstream stages.