# emilfabel04/dvx architecture

> Architecture diagram of the GitHub repository emilfabel04/dvx, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/emilfabel04/dvx
- Repository: https://github.com/emilfabel04/dvx

## Overview

dvx is a pre-alpha design for measuring a dataset’s impact on a target language model. The intended workflow loads and samples data, scans for novelty, duplicates, and model exposure, then fine-tunes and evaluates the model against benchmarks and a generic-data control to produce an impact report. The README also describes scaling analysis, contamination and cluster contributions, and a future local corpus. The latter evaluation and reporting stages are documented capabilities, not verified implementations in the sampled source; no runtime orchestrator was shown.

Main entry points: [`src/dvx/data.py`](https://github.com/emilfabel04/dvx/blob/main/src/dvx/data.py) (Dataset Loading), [`src/dvx`](https://github.com/emilfabel04/dvx/tree/main/src/dvx) (Benchmark Evaluation), [`src/dvx/scan/losses.py`](https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/losses.py) (Model Scoring), [`src/dvx/scan/calibration.py`](https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/calibration.py) (Exposure Calibration)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_inputs["Inputs and Models"]
  node_dataset["Dataset Loading<br/>[data.py]"]
  node_model["Model Loading<br/>[models.py]"]
  node_config["Run Profiles<br/>[config.py]"]
  node_prompting["Prompt Encoding<br/>[prompting.py]"]
end

subgraph group_scan["Dataset Scans"]
  node_losses["Model Scoring<br/>[losses.py]"]
  node_duplicates["Duplicate Detection<br/>[duplicates.py]"]
  node_novelty["Novelty Buckets<br/>[novelty.py]"]
  node_calibration["Exposure Calibration<br/>[calibration.py]"]
  node_exposure["Exposure Estimate<br/>[exposure.py]"]
end

subgraph group_impact["Impact Evaluation"]
  node_fine_tuning["LoRA Fine-tuning"]
  node_benchmarks["Benchmark Evaluation"]
  node_scaling["Data Scaling Curve"]
  node_contamination["Contamination Analysis"]
  node_clusters["Cluster Contributions"]
end

subgraph group_records["Reports and Records"]
  node_report["Impact Report"]
  node_corpus[("Impact Corpus")]
end

node_user(("Dataset Evaluator"))
node_wikipedia["Wikipedia<br/>external system"]

node_user -->|"provides data"| node_dataset
node_user -->|"selects model"| node_model
node_dataset -->|"supplies examples"| node_duplicates
node_dataset -->|"supplies examples"| node_novelty
node_dataset -->|"provides examples"| node_prompting
node_dataset -->|"provides examples"| node_losses
node_prompting -->|"encodes examples"| node_losses
node_model -.->|"scores with model"| node_losses
node_calibration -.->|"fetches from"| node_user
node_calibration -->|"fetches texts"| node_wikipedia
node_losses -->|"provides log-probs"| node_exposure
node_calibration -.->|"provides calibration texts"| node_exposure
node_user -->|"requests impact run"| node_fine_tuning
node_fine_tuning -->|"scores before and after"| node_benchmarks
node_benchmarks -->|"measures across data sizes"| node_scaling
node_benchmarks -->|"compares raw and clean gain"| node_contamination
node_benchmarks -->|"reports cluster contributions"| node_clusters
node_benchmarks -->|"contributes results"| node_report
node_scaling -->|"contributes curve"| node_report
node_contamination -->|"contributes analysis"| node_report
node_clusters -->|"contributes analysis"| node_report
node_report -->|"writes run record"| node_corpus

click node_dataset "https://github.com/emilfabel04/dvx/blob/main/src/dvx/data.py"
click node_model "https://github.com/emilfabel04/dvx/blob/main/src/dvx/models.py"
click node_config "https://github.com/emilfabel04/dvx/blob/main/src/dvx/config.py"
click node_prompting "https://github.com/emilfabel04/dvx/blob/main/src/dvx/prompting.py"
click node_losses "https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/losses.py"
click node_duplicates "https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/duplicates.py"
click node_novelty "https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/novelty.py"
click node_calibration "https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/calibration.py"
click node_exposure "https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/exposure.py"
click node_fine_tuning "https://github.com/emilfabel04/dvx/tree/main/src/dvx"
click node_benchmarks "https://github.com/emilfabel04/dvx/tree/main/src/dvx"
click node_scaling "https://github.com/emilfabel04/dvx/tree/main/src/dvx"
click node_contamination "https://github.com/emilfabel04/dvx/tree/main/src/dvx"
click node_clusters "https://github.com/emilfabel04/dvx/tree/main/src/dvx"
click node_report "https://github.com/emilfabel04/dvx/tree/main/src/dvx"
click node_corpus "https://github.com/emilfabel04/dvx/tree/main/src/dvx"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_dataset,node_model,node_config,node_prompting toneBlue
class node_losses,node_duplicates,node_novelty,node_calibration,node_exposure toneAmber
class node_fine_tuning,node_benchmarks,node_scaling,node_contamination,node_clusters toneMint
class node_report,node_corpus toneRose
class node_user,node_wikipedia toneIndigo
```

## Components

### Inputs and Models

- [Dataset Loading](https://github.com/emilfabel04/dvx/blob/main/src/dvx/data.py): `src/dvx/data.py`
- [Model Loading](https://github.com/emilfabel04/dvx/blob/main/src/dvx/models.py): `src/dvx/models.py`
- [Run Profiles](https://github.com/emilfabel04/dvx/blob/main/src/dvx/config.py): `src/dvx/config.py`
- [Prompt Encoding](https://github.com/emilfabel04/dvx/blob/main/src/dvx/prompting.py): `src/dvx/prompting.py`

### Dataset Scans

- [Model Scoring](https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/losses.py): `src/dvx/scan/losses.py`
- [Duplicate Detection](https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/duplicates.py): `src/dvx/scan/duplicates.py`
- [Novelty Buckets](https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/novelty.py): `src/dvx/scan/novelty.py`
- [Exposure Calibration](https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/calibration.py): `src/dvx/scan/calibration.py`
- [Exposure Estimate](https://github.com/emilfabel04/dvx/blob/main/src/dvx/scan/exposure.py): `src/dvx/scan/exposure.py`

### Impact Evaluation

- [LoRA Fine-tuning](https://github.com/emilfabel04/dvx/tree/main/src/dvx): `src/dvx`
- [Benchmark Evaluation](https://github.com/emilfabel04/dvx/tree/main/src/dvx): `src/dvx`
- [Data Scaling Curve](https://github.com/emilfabel04/dvx/tree/main/src/dvx): `src/dvx`
- [Contamination Analysis](https://github.com/emilfabel04/dvx/tree/main/src/dvx): `src/dvx`
- [Cluster Contributions](https://github.com/emilfabel04/dvx/tree/main/src/dvx): `src/dvx`

### Reports and Records

- [Impact Report](https://github.com/emilfabel04/dvx/tree/main/src/dvx): `src/dvx`
- [Impact Corpus](https://github.com/emilfabel04/dvx/tree/main/src/dvx): `src/dvx`

### Other components

- **Dataset Evaluator**
- **Wikipedia**: external system

## Connections

- Dataset Evaluator → Dataset Loading: provides data (evidence: `README.md`)
- Dataset Evaluator → Model Loading: selects model (evidence: `README.md`)
- Dataset Loading → Duplicate Detection: supplies examples (evidence: `src/dvx/scan/duplicates.py`)
- Dataset Loading → Novelty Buckets: supplies examples (evidence: `src/dvx/scan/novelty.py`)
- Dataset Loading → Prompt Encoding: provides examples (evidence: `src/dvx/prompting.py`)
- Dataset Loading → Model Scoring: provides examples (evidence: `src/dvx/scan/losses.py`)
- Prompt Encoding → Model Scoring: encodes examples (evidence: `src/dvx/scan/losses.py`)
- Model Loading → Model Scoring: scores with model
- Exposure Calibration → Dataset Evaluator: fetches from
- Exposure Calibration → Wikipedia: fetches texts (evidence: `src/dvx/scan/calibration.py`)
- Model Scoring → Exposure Estimate: provides log-probs (evidence: `src/dvx/scan/exposure.py`)
- Exposure Calibration → Exposure Estimate: provides calibration texts
- Dataset Evaluator → LoRA Fine-tuning: requests impact run (evidence: `README.md`)
- LoRA Fine-tuning → Benchmark Evaluation: scores before and after (evidence: `README.md`)
- Benchmark Evaluation → Data Scaling Curve: measures across data sizes (evidence: `README.md`)
- Benchmark Evaluation → Contamination Analysis: compares raw and clean gain (evidence: `README.md`)
- Benchmark Evaluation → Cluster Contributions: reports cluster contributions (evidence: `README.md`)
- Benchmark Evaluation → Impact Report: contributes results (evidence: `README.md`)
- Data Scaling Curve → Impact Report: contributes curve (evidence: `README.md`)
- Contamination Analysis → Impact Report: contributes analysis (evidence: `README.md`)
- Cluster Contributions → Impact Report: contributes analysis (evidence: `README.md`)
- Impact Report → Impact Corpus: writes run record (evidence: `README.md`)
