How ccao-data/model-res-avm works
Project Type and Purpose
• This repository implements an end-to-end automated valuation model (AVM) for residential property in Cook County. It is a data-science/ML pipeline rather than a traditional web application.
• Key indicators:
• pipeline/00-ingest.R through pipeline/08-api.R scripts following stages “Ingest → Train → Assess → Evaluate → Interpret → Finalize → Upload → Export.”
• DVC (dvc.yaml, dvc.lock) for data versioning and pipeline orchestration.
• R (renv, DESCRIPTION) as the primary language, with some Python utilities.
• CI/CD workflows in .github/workflows for build, test, and AWS Batch runs.
• Input data in input/, outputs in output/, documentation in docs/ and reports/.
File-Structure Analysis
• Top-level: configuration (params.yaml, .pre-commit, .github), environment (renv.*), Dockerfile, DVC.
• pipeline/: one script per pipeline stage.
• input/ & output/: versioned data (DVC, S3).
• R/: shared helper functions and setup logic.
• python/: specialized components (comps.py) and unit tests.
• reports/ & analyses/: Quarto/QuMD analyses and diagnostic reports.
• .github/workflows: CI, model run, clean-up.
README Insights
• External services: AWS S3 (storage of inputs/outputs), AWS Batch (compute), AWS Athena (ingest), GitHub Actions (CI/CD).
• Dependency management via renv and containerization via Docker.
• DVC coordinates data and stage dependencies.
• Final outputs feed internal Tableau dashboards and downstream systems (iasWorld).
Main Components and Their Interactions
a. Data Sources and Storage
• Athena (SQL) → S3 input buckets (DVC-tracked).
• Local input/ for public users.
b. Pipeline Orchestrator
• DVC: DAG defined in dvc.yaml, parameters in params.yaml.
• Scripts (pipeline/00-08) as nodes.
c. Compute Environment
• R scripts executed locally or in Docker container (Dockerfile).
• Python helpers for comparables.
• AWS Batch jobs triggered via GitHub Actions.
d. Modeling Stages
• Ingest → data preparation
• Train → LightGBM (via Tidymodels + Lightsnip)
• Assess → batch prediction
• Evaluate → metrics (RMSE, COD, PRD, MKI)
• Interpret → SHAP values, feature importance, comparable generation
• Finalize → Quarto report rendering
• Upload → push artifacts to S3/Athena (via AWS SDK/SNS)
• Export → spreadsheet/text for Desk Review & iasWorld
e. Dependency Management
• renv.lock for core; renv/profiles/dev and renv/profiles/reporting for extras.
f. CI/CD and Orchestration
• GitHub Actions: build, test (.github/workflows/test.yaml), model run, cleanup.
• CloudWatch for log monitoring.
Architectural Patterns & Design Principles
• Modular Pipeline (DAG): each stage isolated, re-runnable, cached.
• Separation of Concerns: data ingestion, modeling, reporting, and deployment are distinct.
• Infrastructure as Code: pipeline and CI config versioned in Git.
• Reproducibility: DVC for data, renv for packages, Docker for environment.
• Scalability: scale compute via AWS Batch and parallel LightGBM threads.
• Idempotency: running dvc repro or pipeline scripts yields the same outputs given same inputs & params.