How dssgxuk/dncp works
This is an R-based machine learning / data science pipeline for estimating “adequate” prices in public procurement (DNCP Paraguay). It is not a long-running service; it’s primarily a batch workflow driven by R scripts with:
• Config-driven stages (YAML in conf/)
• Reusable feature engineering + training utilities (src/functions/)
• Stage entrypoints (src/scripts/: fe.R, training.R, predict.R, training_mlflow.R)
• Offline analysis reports (reports/*.Rmd)
• Reproducible dependency snapshotting via Packrat (packrat/)
Orchestration / Entrypoints (stage runners)
• src/scripts/fe.R — feature engineering stage
• src/scripts/training.R — model training stage
• src/scripts/predict.R — inference/prediction stage
• src/scripts/training_mlflow.R — training with experiment tracking (MLflow)
Business logic / libraries (reusable modules)
• src/functions/fe_* — feature engineering modules split by domain:
• fe_group_buyer.R, fe_group_context.R (domain/group feature sets)
• fe_outlier_detection.R (data quality / anomaly filtering)
• fe_helpers.R (common FE utilities)
• src/functions/training_helpers.R — training utilities (splits, metrics, model setup)
• src/functions/predict_helpers.R — loading artifacts + running inference consistently
Configuration layer (controls behavior)
• conf/training_conf.yaml, conf/predict_conf.yaml, conf/fe_conf.yaml, conf/eda_conf.yaml
• conf/merge_sparse.yaml (likely feature merge rules)
• conf/valid_categories.yaml (categorical constraints/validation)
• conf/meta/training_set_data_dict_v02.xlsx (data dictionary / schema reference)
Data & artifacts (stateful inputs/outputs)
• data/raw/ — raw datasets landing zone
• data/output/ — derived datasets / features / predictions / models (project artifacts)
Reports / documentation outputs
• reports/training_set_eda.Rmd, reports/Evaluation.Rmd (+ generated PDF)
These are “consumer” artifacts that depend on intermediate outputs (cleaned data, metrics, model artifacts).
Runtime environment & dependency management
• packrat/ (snapshot lock + sources) indicates reproducible R environment
External system worth diagramming (implied)
• MLflow tracking server (implied by training_mlflow.R)
Even without code, it’s important: show “MLflow Tracking” as an external dependency receiving metrics/artifacts.
Feature Engineering Job (src/scripts/fe.R)
• uses conf/fe_conf.yaml, conf/valid_categories.yaml, conf/merge_sparse.yaml
• uses FE modules in src/functions/fe_*.R
• outputs “Feature Dataset / Model Inputs” → data/output/
Training Job (src/scripts/training.R or training_mlflow.R)
• uses conf/training_conf.yaml
• uses training_helpers.R
• outputs: Model artifacts, metrics, feature metadata → data/output/
• (if MLflow) logs params/metrics/artifacts → MLflow Tracking Server
Evaluation / Reporting (reports/*.Rmd)
• reads features + metrics + maybe predictions
• outputs PDFs/HTML (store in reports/ or data/output/ depending on your convention)
Prediction Job (src/scripts/predict.R)
• uses conf/predict_conf.yaml
• uses predict_helpers.R
• reads trained model artifacts + new data
• outputs “Predictions / Price estimates” → data/output/
Edges to include
• where the “system of record” data is,
• how FE/training/prediction connect,
• what gets persisted and reused.
Goal: show maintainability boundaries: scripts vs shared functions vs config.
Nodes to include
• A “Scripts” cluster: fe.R, training.R, predict.R, training_mlflow.R
• A “Functions” cluster: FE modules, training_helpers, predict_helpers
• A “Config” cluster: YAMLs + data dictionary
• A “Data” cluster: raw + output
• Optional: “Reports” cluster (Rmds)
Edges
• Script → Function modules it calls
• Script → Config files it reads
• Script → Data read/write
• training_mlflow.R → MLflow external
This view helps reviewers quickly answer: “where do I change feature logic?” vs “where do I change hyperparameters?” vs “where do artifacts land?”
Use repo-accurate naming in labels to avoid ambiguity:
• Feature Engineering: src/scripts/fe.R
• “FE modules”: src/functions/fe_group_buyer.R, fe_group_context.R, fe_outlier_detection.R, fe_helpers.R
• Training: src/scripts/training.R / src/scripts/training_mlflow.R
• “Training utils”: src/functions/training_helpers.R
• Prediction: src/scripts/predict.R
• “Predict utils”: src/functions/predict_helpers.R
• Config: conf/*.yaml, conf/meta/*.xlsx
• Raw Data: data/raw/
• Artifacts / Outputs: data/output/
• Reports: reports/*.Rmd → reports/*.pdf
• Runtime: “R + Packrat” (packrat/packrat.lock)
• “Feature table(s)”
• “Trained model object(s)”
• “Metrics (AUC/RMSE/etc.)”
• “Predictions / estimated prices”
Architectural patterns to explicitly highlight
These are the key patterns this repo suggests; they should be visually encoded:
Config-driven batch pipeline
• YAML configs controlling FE/training/predict behavior
Separation of concerns
• src/scripts/ (orchestration) vs src/functions/ (reusable logic)
Reproducibility
• Packrat locks dependency versions (treat as part of the architecture)
Experiment tracking (optional path)
• MLflow integration is a separate training “mode”
What not to include (to keep it accurate)
• Don’t invent a REST API, UI, or database server—nothing in the tree indicates one.
• Don’t model Packrat packages as separate components; represent Packrat as the environment manager.
• Don’t over-index on Rmd reports as core runtime; they are analysis consumers of pipeline outputs.
data/output/ (predictions)
Bottom lane:
• reports/*.Rmd reading from data/output/ producing PDFs
This yields the “as many components as possible” separation while staying faithful to the repository.