# ccao-data/model-res-avm architecture

> Architecture diagram of the GitHub repository ccao-data/model-res-avm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-07). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/ccao-data/model-res-avm
- Repository: https://github.com/ccao-data/model-res-avm

## Overview

##### Project Type and Purpose
- This repository implements an end-to-end automated valuation model (AVM) for residential property in Cook County. It is a **data-science/ML pipeline** rather than a traditional web application.
- Key indicators:
- `pipeline/00-ingest.R` through `pipeline/08-api.R` scripts following stages “Ingest → Train → Assess → Evaluate → Interpret → Finalize → Upload → Export.”
- DVC (`dvc.yaml`, `dvc.lock`) for data versioning and pipeline orchestration.
- R (`renv`, `DESCRIPTION`) as the primary language, with some Python utilities.
- CI/CD workflows in `.github/workflows` for build, test, and AWS Batch runs.
- Input data in `input/`, outputs in `output/`, documentation in `docs/` and `reports/`.

##### File-Structure Analysis
- Top-level: configuration (`params.yaml`, `.pre-commit`, `.github`), environment (`renv.*`), Dockerfile, DVC.
- `pipeline/`: one script per pipeline stage.
- `input/` & `output/`: versioned data (DVC, S3).
- `R/`: shared helper functions and setup logic.
- `python/`: specialized components (`comps.py`) and unit tests.
- `reports/` & `analyses/`: Quarto/QuMD analyses and diagnostic reports.
- `.github/workflows`: CI, model run, clean-up.

##### README Insights
- External services: AWS S3 (storage of inputs/outputs), AWS Batch (compute), AWS Athena (ingest), GitHub Actions (CI/CD).
- Dependency management via `renv` and containerization via Docker.
- DVC coordinates data and stage dependencies.
- Final outputs feed internal Tableau dashboards and downstream systems (iasWorld).

##### Main Components and Their Interactions
a. **Data Sources and Storage**
- Athena (SQL) → S3 input buckets (DVC-tracked).
- Local `input/` for public users.
b. **Pipeline Orchestrator**
- DVC: DAG defined in `dvc.yaml`, parameters in `params.yaml`.
- Scripts (`pipeline/00-08`) as nodes.
c. **Compute Environment**
- R scripts executed locally or in Docker container (`Dockerfile`).
- Python helpers for comparables.
- AWS Batch jobs triggered via GitHub Actions.
d. **Modeling Stages**
- Ingest → data preparation
- Train → LightGBM (via Tidymodels + Lightsnip)
- Assess → batch prediction
- Evaluate → metrics (RMSE, COD, PRD, MKI)
- Interpret → SHAP values, feature importance, comparable generation
- Finalize → Quarto report rendering
- Upload → push artifacts to S3/Athena (via AWS SDK/SNS)
- Export → spreadsheet/text for Desk Review & iasWorld
e. **Dependency Management**
- `renv.lock` for core; `renv/profiles/dev` and `renv/profiles/reporting` for extras.
f. **CI/CD and Orchestration**
- GitHub Actions: build, test (`.github/workflows/test.yaml`), model run, cleanup.
- CloudWatch for log monitoring.

##### Architectural Patterns & Design Principles
- **Modular Pipeline (DAG)**: each stage isolated, re-runnable, cached.
- **Separation of Concerns**: data ingestion, modeling, reporting, and deployment are distinct.
- **Infrastructure as Code**: pipeline and CI config versioned in Git.
- **Reproducibility**: DVC for data, `renv` for packages, Docker for environment.
- **Scalability**: scale compute via AWS Batch and parallel LightGBM threads.
- **Idempotency**: running `dvc repro` or pipeline scripts yields the same outputs given same inputs & params.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer"
        direction TB
        Athena["Athena (SQL)"]:::data
        S3In["S3 Input Bucket"]:::data
        LocalIn["Local Input Folder"]:::data
    end

    %% Orchestrator
    subgraph "Orchestration" 
        direction TB
        DVC["DVC Pipeline\ndvc.yaml, dvc.lock, params.yaml"]:::orchestration
    end

    %% Compute & Environment
    subgraph "Compute & Environment"
        direction TB
        Docker["Docker Container\nDockerfile"]:::compute
        RRuntime["R Runtime\nDESCRIPTION"]:::compute
        PythonRuntime["Python Runtime"]:::compute
        RENv["renv Environment\nrenv.lock, renv/, renv/settings.json"]:::dependency
    end

    %% Pipeline Stages
    subgraph "Pipeline Stages (DAG)"
        direction TB
        Ingest["Ingest\npipeline/00-ingest.R"]:::pipeline
        Train["Train\npipeline/01-train.R"]:::pipeline
        Assess["Assess\npipeline/02-assess.R"]:::pipeline
        Evaluate["Evaluate\npipeline/03-evaluate.R"]:::pipeline
        Interpret["Interpret\npipeline/04-interpret.R"]:::pipeline
        Finalize["Finalize\npipeline/05-finalize.R"]:::pipeline
        Upload["Upload\npipeline/06-upload.R"]:::pipeline
        Export["Export\npipeline/07-export.R"]:::pipeline
        API["API/Workbooks\npipeline/08-api.R"]:::pipeline
    end

    %% CI/CD
    subgraph "CI/CD & Monitoring"
        direction TB
        GitHubActions["GitHub Actions\nWorkflows"]:::cicd
        AWSBatch["AWS Batch Jobs"]:::cicd
        CloudWatch["CloudWatch Logs"]:::cicd
    end

    %% Output Layer
    subgraph "Output Layer"
        direction TB
        S3Out["S3 Output Bucket"]:::data
        AthenaTables["Athena Tables"]:::data
        LocalOut["Local Output Folder"]:::data
        Tableau["Tableau Dashboards"]:::external
        IAS["iasWorld Exports"]:::external
    end

    %% Connections: Data to Orchestrator
    Athena -->|queries| S3In
    S3In -->|versioned by DVC| DVC
    LocalIn -->|versioned by DVC| DVC

    %% Orchestration to Compute & Env
    DVC -->|reads params| RENv
    DVC -->|triggers| Docker
    Docker -->|provides| RRuntime
    Docker -->|provides| PythonRuntime

    %% Orchestrator triggers Pipeline
    DVC -->|executes| Ingest
    Ingest -->|produces data| Train
    Train -->|model artifacts| Assess
    Assess -->|predictions| Evaluate
    Evaluate -->|metrics| Interpret
    Interpret -->|reports| Finalize
    Finalize -->|artifacts| Upload
    Upload -->|push to S3/Athena| Export
    Export -->|feeds| API

    %% CI/CD triggers Pipeline
    GitHubActions -->|build & push| Docker
    GitHubActions -->|trigger| AWSBatch
    AWSBatch -->|runs| DVC
    AWSBatch --> CloudWatch

    %% Pipeline outputs
    Upload -->|to| S3Out
    Upload -->|to| AthenaTables
    Export -->|to| LocalOut
    S3Out --> Tableau
    S3Out --> IAS

    %% Click Events
    click LocalIn "https://github.com/ccao-data/model-res-avm/tree/master/input/"
    click DVC "https://github.com/ccao-data/model-res-avm/blob/master/dvc.yaml"
    click RENv "https://github.com/ccao-data/model-res-avm/blob/master/renv.lock"
    click Docker "https://github.com/ccao-data/model-res-avm/tree/master/Dockerfile"
    click RRuntime "https://github.com/ccao-data/model-res-avm/tree/master/DESCRIPTION"
    click Ingest "https://github.com/ccao-data/model-res-avm/blob/master/pipeline/00-ingest.R"
    click Train "https://github.com/ccao-data/model-res-avm/blob/master/pipeline/01-train.R"
    click Assess "https://github.com/ccao-data/model-res-avm/blob/master/pipeline/02-assess.R"
    click Evaluate "https://github.com/ccao-data/model-res-avm/blob/master/pipeline/03-evaluate.R"
    click Interpret "https://github.com/ccao-data/model-res-avm/blob/master/pipeline/04-interpret.R"
    click Finalize "https://github.com/ccao-data/model-res-avm/blob/master/pipeline/05-finalize.R"
    click Upload "https://github.com/ccao-data/model-res-avm/blob/master/pipeline/06-upload.R"
    click Export "https://github.com/ccao-data/model-res-avm/blob/master/pipeline/07-export.R"
    click API "https://github.com/ccao-data/model-res-avm/blob/master/pipeline/08-api.R"

    %% Styles
    classDef data fill:#ADD8E6,stroke:#036,stroke-width:1px
    classDef compute fill:#98FB98,stroke:#0A3,stroke-width:1px
    classDef dependency fill:#90EE90,stroke:#0A3,stroke-width:1px
    classDef orchestration fill:#FFA500,stroke:#A53,stroke-width:1px
    classDef pipeline fill:#E0FFE0,stroke:#0A3,stroke-width:1px
    classDef cicd fill:#DDA0DD,stroke:#A3A,stroke-width:1px
    classDef external fill:#FFE4B5,stroke:#AA6,stroke-width:1px
```
