# dssgxuk/dncp architecture

> Architecture diagram of the GitHub repository dssgxuk/dncp, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-02-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/dssgxuk/dncp
- Repository: https://github.com/dssgxuk/dncp

## Overview

This is an **R-based machine learning / data science pipeline** for **estimating “adequate” prices in public procurement (DNCP Paraguay)**. It is not a long-running service; it’s primarily a **batch workflow** driven by R scripts with:
- **Config-driven stages** (YAML in `conf/`)
- **Reusable feature engineering + training utilities** (`src/functions/`)
- **Stage entrypoints** (`src/scripts/`: `fe.R`, `training.R`, `predict.R`, `training_mlflow.R`)
- **Offline analysis reports** (`reports/*.Rmd`)
- **Reproducible dependency snapshotting** via **Packrat** (`packrat/`)

##### Orchestration / Entrypoints (stage runners)
- `src/scripts/fe.R` — feature engineering stage
- `src/scripts/training.R` — model training stage
- `src/scripts/predict.R` — inference/prediction stage
- `src/scripts/training_mlflow.R` — training with experiment tracking (MLflow)

##### Business logic / libraries (reusable modules)
- `src/functions/fe_*` — feature engineering modules split by domain:
- `fe_group_buyer.R`, `fe_group_context.R` (domain/group feature sets)
- `fe_outlier_detection.R` (data quality / anomaly filtering)
- `fe_helpers.R` (common FE utilities)
- `src/functions/training_helpers.R` — training utilities (splits, metrics, model setup)
- `src/functions/predict_helpers.R` — loading artifacts + running inference consistently

##### Configuration layer (controls behavior)
- `conf/training_conf.yaml`, `conf/predict_conf.yaml`, `conf/fe_conf.yaml`, `conf/eda_conf.yaml`
- `conf/merge_sparse.yaml` (likely feature merge rules)
- `conf/valid_categories.yaml` (categorical constraints/validation)
- `conf/meta/training_set_data_dict_v02.xlsx` (data dictionary / schema reference)

##### Data & artifacts (stateful inputs/outputs)
- `data/raw/` — raw datasets landing zone
- `data/output/` — derived datasets / features / predictions / models (project artifacts)

##### Reports / documentation outputs
- `reports/training_set_eda.Rmd`, `reports/Evaluation.Rmd` (+ generated PDF)
These are “consumer” artifacts that depend on intermediate outputs (cleaned data, metrics, model artifacts).

##### Runtime environment & dependency management
- `packrat/` (snapshot lock + sources) indicates **reproducible R environment**

##### External system worth diagramming (implied)
- **MLflow tracking server** (implied by `training_mlflow.R`)
Even without code, it’s important: show “MLflow Tracking” as an external dependency receiving metrics/artifacts.

##### Feature Engineering Job (`src/scripts/fe.R`)
- uses `conf/fe_conf.yaml`, `conf/valid_categories.yaml`, `conf/merge_sparse.yaml`
- uses FE modules in `src/functions/fe_*.R`
- outputs “Feature Dataset / Model Inputs” → `data/output/`
##### Training Job (`src/scripts/training.R` or `training_mlflow.R`)
- uses `conf/training_conf.yaml`
- uses `training_helpers.R`
- outputs: **Model artifacts**, **metrics**, **feature metadata** → `data/output/`
- (if MLflow) logs **params/metrics/artifacts** → **MLflow Tracking Server**
##### Evaluation / Reporting (`reports/*.Rmd`)
- reads features + metrics + maybe predictions
- outputs PDFs/HTML (store in `reports/` or `data/output/` depending on your convention)
##### Prediction Job (`src/scripts/predict.R`)
- uses `conf/predict_conf.yaml`
- uses `predict_helpers.R`
- reads trained model artifacts + new data
- outputs “Predictions / Price estimates” → `data/output/`

**Edges to include**

- where the “system of record” data is,
- how FE/training/prediction connect,
- what gets persisted and reused.

Goal: show maintainability boundaries: scripts vs shared functions vs config.

**Nodes to include**
- A “Scripts” cluster: `fe.R`, `training.R`, `predict.R`, `training_mlflow.R`
- A “Functions” cluster: FE modules, training_helpers, predict_helpers
- A “Config” cluster: YAMLs + data dictionary
- A “Data” cluster: raw + output
- Optional: “Reports” cluster (Rmds)

**Edges**
- Script → Function modules it calls
- Script → Config files it reads
- Script → Data read/write
- training_mlflow.R → MLflow external

This view helps reviewers quickly answer: “where do I change feature logic?” vs “where do I change hyperparameters?” vs “where do artifacts land?”

Use repo-accurate naming in labels to avoid ambiguity:

- **Feature Engineering**: `src/scripts/fe.R`
- “FE modules”: `src/functions/fe_group_buyer.R`, `fe_group_context.R`, `fe_outlier_detection.R`, `fe_helpers.R`
- **Training**: `src/scripts/training.R` / `src/scripts/training_mlflow.R`
- “Training utils”: `src/functions/training_helpers.R`
- **Prediction**: `src/scripts/predict.R`
- “Predict utils”: `src/functions/predict_helpers.R`
- **Config**: `conf/*.yaml`, `conf/meta/*.xlsx`
- **Raw Data**: `data/raw/`
- **Artifacts / Outputs**: `data/output/`
- **Reports**: `reports/*.Rmd` → `reports/*.pdf`
- **Runtime**: “R + Packrat” (`packrat/packrat.lock`)

- “Feature table(s)”
- “Trained model object(s)”
- “Metrics (AUC/RMSE/etc.)”
- “Predictions / estimated prices”

##### Architectural patterns to explicitly highlight
These are the key patterns this repo suggests; they should be visually encoded:

##### Config-driven batch pipeline
- YAML configs controlling FE/training/predict behavior
##### Separation of concerns
- `src/scripts/` (orchestration) vs `src/functions/` (reusable logic)
##### Reproducibility
- Packrat locks dependency versions (treat as part of the architecture)
##### Experiment tracking (optional path)
- MLflow integration is a separate training “mode”

##### What not to include (to keep it accurate)
- Don’t invent a REST API, UI, or database server—nothing in the tree indicates one.
- Don’t model Packrat packages as separate components; represent Packrat as the **environment manager**.
- Don’t over-index on Rmd reports as core runtime; they are **analysis consumers** of pipeline outputs.

##### `data/output/` (predictions)
Bottom lane:
- `reports/*.Rmd` reading from `data/output/` producing PDFs

This yields the “as many components as possible” separation while staying faithful to the repository.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% ================
    %% External / Runtime
    %% ================
    subgraph "Execution Environment"
        RRuntime["R Runtime"]:::runtime
        Packrat["Packrat (reproducible dependencies)"]:::runtime
        PackratLock["packrat.lock (dependency snapshot)"]:::doc
        PackratInit["packrat/init.R (bootstrap)"]:::doc
        RRuntime -->|"uses"| Packrat
        Packrat -->|"pins_versions"| PackratLock
        Packrat -->|"initializes"| PackratInit
    end

    MLflow["MLflow Tracking Server (external)"]:::external

    %% ================
    %% Configuration & Schema
    %% ================
    subgraph "Configuration & Schema (config-driven pipeline)"
        FeConf["conf/fe_conf.yaml"]:::config
        TrainConf["conf/training_conf.yaml"]:::config
        PredConf["conf/predict_conf.yaml"]:::config
        EdaConf["conf/eda_conf.yaml"]:::config
        MergeSparse["conf/merge_sparse.yaml"]:::config
        ValidCats["conf/valid_categories.yaml"]:::config
        DataDict["training_set_data_dict_v02.xlsx (schema/data dictionary)"]:::doc
    end

    %% ================
    %% Data & Artifact Storage
    %% ================
    subgraph "Stateful Storage (filesystem folders)"
        RawData[("data/raw/ (raw datasets)")]:::storage
        OutputData[("data/output/ (features/models/metrics/predictions)")]:::storage
    end

    %% ================
    %% Shared Libraries (Reusable Modules)
    %% ================
    subgraph "Reusable R Modules (src/functions/)"
        subgraph "Feature Engineering Modules"
            FeBuyer["fe_group_buyer.R"]:::lib
            FeContext["fe_group_context.R"]:::lib
            FeOutliers["fe_outlier_detection.R"]:::lib
            FeHelpers["fe_helpers.R"]:::lib
        end

        TrainHelpers["training_helpers.R"]:::lib
        PredHelpers["predict_helpers.R"]:::lib
    end

    %% ================
    %% Pipeline Jobs / Entrypoints (src/scripts/)
    %% ================
    subgraph "Batch Pipeline Stages (src/scripts/)"
        FEJob["Feature Engineering Job (fe.R)"]:::job
        TrainJob["Training Job (training.R)"]:::job
        TrainMLflowJob["Training Job + Tracking (training_mlflow.R)"]:::job
        PredJob["Prediction Job (predict.R)"]:::job
    end

    %% ================
    %% Reports (offline analysis)
    %% ================
    subgraph "Reports & Documentation (reports/)"
        EdaRmd["training_set_eda.Rmd"]:::report
        EvalRmd["Evaluation.Rmd"]:::report
        EdaPdf["training_set_eda_2021-08-25.pdf"]:::artifact
    end

    %% =========================
    %% Vertical end-to-end flow
    %% =========================
    RawData -->|"read_raw_data"| FEJob
    FEJob -->|"write_feature_dataset/cleaned_training_set/sparse_merge_output"| OutputData

    OutputData -->|"read_features/model_inputs"| TrainJob
    TrainJob -->|"write_model_artifacts+preprocessing_metadata+metrics"| OutputData

    OutputData -->|"read_features/model_inputs"| TrainMLflowJob
    TrainMLflowJob -->|"write_model_artifacts+preprocessing_metadata+metrics"| OutputData
    TrainMLflowJob -.->|"log_params/metrics/artifacts"| MLflow

    OutputData -->|"read_model_artifacts+new_data_features"| PredJob
    PredJob -->|"write_predictions/estimated_prices"| OutputData

    %% =========================
    %% Reports consumption path
    %% =========================
    OutputData -->|"read_features/metrics/(optional)predictions"| EdaRmd
    OutputData -->|"read_features/metrics/(optional)predictions"| EvalRmd
    EdaRmd -->|"render_pdf/html"| EdaPdf

    %% =========================
    %% Config dependencies (dashed)
    %% =========================
    FeConf -.->|"configure"| FEJob
    MergeSparse -.->|"merge_rules"| FEJob
    ValidCats -.->|"validate_categories"| FEJob
    DataDict -.->|"schema_reference"| FEJob

    TrainConf -.->|"configure"| TrainJob
    DataDict -.->|"schema_reference"| TrainJob

    TrainConf -.->|"configure"| TrainMLflowJob
    DataDict -.->|"schema_reference"| TrainMLflowJob

    PredConf -.->|"configure"| PredJob
    DataDict -.->|"schema_reference"| PredJob

    EdaConf -.->|"configure_reporting"| EdaRmd
    EdaConf -.->|"configure_reporting"| EvalRmd

    %% =========================
    %% Code/module dependencies
    %% =========================
    RRuntime -->|"executes"| FEJob
    RRuntime -->|"executes"| TrainJob
    RRuntime -->|"executes"| TrainMLflowJob
    RRuntime -->|"executes"| PredJob
    RRuntime -->|"executes"| EdaRmd
    RRuntime -->|"executes"| EvalRmd

    FEJob -->|"uses"| FeBuyer
    FEJob -->|"uses"| FeContext
    FEJob -->|"uses"| FeOutliers
    FEJob -->|"uses"| FeHelpers

    TrainJob -->|"uses"| TrainHelpers
    TrainMLflowJob -->|"uses"| TrainHelpers

    PredJob -->|"uses"| PredHelpers

    %% =========================
    %% Feedback loop (conceptual)
    %% =========================
    EvalRmd -.->|"adjust_hyperparameters/feature_set/filters"| TrainConf

    %% =========================
    %% Click Events (from mapping)
    %% =========================
    click FEJob "https://github.com/dssgxuk/dncp/blob/master/src/scripts/fe.R"
    click TrainJob "https://github.com/dssgxuk/dncp/blob/master/src/scripts/training.R"
    click TrainMLflowJob "https://github.com/dssgxuk/dncp/blob/master/src/scripts/training_mlflow.R"
    click PredJob "https://github.com/dssgxuk/dncp/blob/master/src/scripts/predict.R"

    click FeBuyer "https://github.com/dssgxuk/dncp/blob/master/src/functions/fe_group_buyer.R"
    click FeContext "https://github.com/dssgxuk/dncp/blob/master/src/functions/fe_group_context.R"
    click FeHelpers "https://github.com/dssgxuk/dncp/blob/master/src/functions/fe_helpers.R"
    click FeOutliers "https://github.com/dssgxuk/dncp/blob/master/src/functions/fe_outlier_detection.R"

    click TrainHelpers "https://github.com/dssgxuk/dncp/blob/master/src/functions/training_helpers.R"
    click PredHelpers "https://github.com/dssgxuk/dncp/blob/master/src/functions/predict_helpers.R"

    click FeConf "https://github.com/dssgxuk/dncp/blob/master/conf/fe_conf.yaml"
    click TrainConf "https://github.com/dssgxuk/dncp/blob/master/conf/training_conf.yaml"
    click PredConf "https://github.com/dssgxuk/dncp/blob/master/conf/predict_conf.yaml"
    click EdaConf "https://github.com/dssgxuk/dncp/blob/master/conf/eda_conf.yaml"
    click MergeSparse "https://github.com/dssgxuk/dncp/blob/master/conf/merge_sparse.yaml"
    click ValidCats "https://github.com/dssgxuk/dncp/blob/master/conf/valid_categories.yaml"
    click DataDict "https://github.com/dssgxuk/dncp/blob/master/conf/meta/training_set_data_dict_v02.xlsx"

    click RawData "https://github.com/dssgxuk/dncp/tree/master/data/raw/"
    click OutputData "https://github.com/dssgxuk/dncp/tree/master/data/output/"

    click EvalRmd "https://github.com/dssgxuk/dncp/blob/master/reports/Evaluation.Rmd"
    click EdaRmd "https://github.com/dssgxuk/dncp/blob/master/reports/training_set_eda.Rmd"
    click EdaPdf "https://github.com/dssgxuk/dncp/blob/master/reports/training_set_eda_2021-08-25.pdf"

    click Packrat "https://github.com/dssgxuk/dncp/tree/master/packrat/"
    click PackratLock "https://github.com/dssgxuk/dncp/blob/master/packrat/packrat.lock"
    click PackratInit "https://github.com/dssgxuk/dncp/blob/master/packrat/init.R"

    %% =========================
    %% Styles (color is critical)
    %% =========================
    classDef job fill:#1f6feb,stroke:#0b3d91,stroke-width:1.5px,color:#ffffff
    classDef lib fill:#7c3aed,stroke:#4c1d95,stroke-width:1.2px,color:#ffffff
    classDef storage fill:#0ea5e9,stroke:#075985,stroke-width:1.5px,color:#ffffff
    classDef config fill:#f59e0b,stroke:#92400e,stroke-width:1.2px,color:#111827
    classDef doc fill:#fde68a,stroke:#b45309,stroke-width:1.2px,color:#111827
    classDef report fill:#22c55e,stroke:#166534,stroke-width:1.2px,color:#ffffff
    classDef artifact fill:#86efac,stroke:#166534,stroke-width:1.2px,color:#064e3b
    classDef runtime fill:#64748b,stroke:#334155,stroke-width:1.2px,color:#ffffff
    classDef external fill:#ef4444,stroke:#7f1d1d,stroke-width:1.5px,color:#ffffff
```
