How equipo59/absenteeism_at_work works
Project type and purpose
• This is an MLOps‐driven machine learning service (“Absenteeism at Work”) that spans from data versioning and model experimentation to automated deployment of a prediction API.
• It follows the Cookiecutter Data Science pattern, uses DVC for data pipelines, MLflow for experiment tracking, and AWS ECS/Fargate (via CloudFormation) for serving.
Main architectural components
a. Data layer
• Raw data (data/raw) versioned with DVC and stored in a remote DVC backend (e.g., S3).
• Intermediate data (data/interim) and processed data (data/processed) artifacts produced by the pipeline.
b. Compute / pipeline layer
• Preprocessing module (preprocess_data.py → data/processed)
• Feature engineering (features.py)
• Model training pipeline (modeling/train.py, Jupyter notebooks)
• Experiment tracking server (MLflow mlruns/) logging params, metrics, artifacts
c. Model registry / artifact store
• Persisted model files in models/ and mlruns/models/...
d. Serving layer
• Flask/FastAPI app (app.py) plus static web UI (static/index.html, script.js, style.css)
• Containerized via Dockerfile and docker-compose.yml
e. CI/CD & infrastructure
• GitHub Actions workflow (.github/workflows/deploy.yml)
• Infrastructure as code (CloudFormation template in infrastructure/cloudformation/infrastructure.yaml)
• AWS resources: ECR repository, ECS Fargate cluster & service, ALB, auto‐scaling, CloudWatch logs
f. External systems
• GitHub (source + CI/CD trigger)
• DVC remote storage (e.g., S3)
• AWS services (ECS, ECR, ALB, CloudWatch)
Relationships & data flow
1. Code commit (master → web) triggers GitHub Actions
2. DVC pipeline (make data-pull / dvc repro) reads raw data → preprocessing → processed data
3. Training module consumes processed data → produces model artifacts + logs to MLflow
4. GitHub Actions uses CloudFormation to provision/update AWS infra
5. Docker build packs app.py, absenteeism_at_work/, selected model artifact → pushes image to ECR
6. ECS service (behind ALB) pulls new image → runs API container
7. Clients (browser or other services) call API endpoint via ALB → Flask app loads model from local container FS
8. CloudWatch aggregates logs and health‐check metrics
Architectural patterns & design principles
• Pipeline as code: DVC + dvc.yaml define reproducible ETL & training steps
• Modular code: separate modules for config, dataset, features, modeling, API
• Experiment tracking: MLflow for metrics, params, artifact versioning
• Infrastructure as code: CloudFormation for full-stack AWS resources
• Containerization & microservice: API container deployed on ECS Fargate behind ALB
• CI/CD: GitHub Actions driving infra provision, image build/push, service update
Essential elements
• Layer separation (Data → Pipeline → Registry → Serving → External)
• Clear labels for each artifact, service, and interaction
• Notes on health checks and logging (CloudWatch)