# manishcooks/range-deduplication architecture

> Architecture diagram of the GitHub repository manishcooks/range-deduplication, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/manishcooks/range-deduplication
- Repository: https://github.com/manishcooks/range-deduplication

## Overview

This is a vector-database range-deduplication benchmarking framework. The principal workflow loads an HDF5 dataset, runs a two-phase Bloom-filter and MinHash-LSH deduplication workload against a configured database adapter, and records benchmark results; orchestration can also collect system metrics and generate plots. The graph preserves the documented adapter, configuration, monitoring, and plotting capabilities. Some orchestration-to-workload and adapter-selection wiring is not visible in the excerpts, so those links are dashed rather than asserted as inspected calls.

Main entry points: [`deduplication_workload/main.py`](https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/main.py) (Workload runner), [`orchestrator/main.py`](https://github.com/manishcooks/range-deduplication/blob/main/orchestrator/main.py) (Orchestrator), [`database_adapter/milvus_adapter.py`](https://github.com/manishcooks/range-deduplication/blob/main/database_adapter/milvus_adapter.py) (Milvus adapter), [`database_adapter/qdrant_adapter.py`](https://github.com/manishcooks/range-deduplication/blob/main/database_adapter/qdrant_adapter.py) (Qdrant adapter)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_control["Configuration &amp; control"]
  node_runner["CLI entry<br/>[run.py]"]
  node_orchestrator["Orchestrator<br/>[main.py]"]
  node_config_parser["Config parser<br/>[parser.py]"]
  node_config_schema["Config schema<br/>[schema.py]"]
end

subgraph group_workload["Deduplication workload"]
  node_workload["Workload runner<br/>[main.py]"]
  node_hdf5_loader["HDF5 loader<br/>[hdf5_loader.py]"]
  node_vector_utils["Vector utilities<br/>[utils.py]"]
  node_hashing["MinHash hashing<br/>[hashing.py]"]
  node_ground_truth["Dedup ground truth<br/>[dedup_gt.py]"]
  node_dedup_ingest["Dedup ingestion<br/>[dedup_ingest.py]"]
  node_dedup_query["Dedup queries<br/>[dedup_query.py]"]
end

subgraph group_database["Database adapters"]
  node_adapter_base["Adapter contract<br/>[base.py]"]
  node_adapter_errors["Adapter errors<br/>[exceptions.py]"]
  node_milvus["Milvus adapter<br/>[milvus_adapter.py]"]
  node_qdrant["Qdrant adapter<br/>[qdrant_adapter.py]"]
end

subgraph group_observability["Metrics &amp; reporting"]
  node_metrics["Workload metrics<br/>[metrics.py]"]
  node_monitor["System monitor<br/>[system_monitor.py]"]
  node_plotting["Result plotting<br/>[plotting.py]"]
end

node_user(("Engineer"))
node_vector_db[("Vector database")]
node_results["Benchmark outputs"]

node_user -->|"starts run"| node_runner
node_runner -->|"calls"| node_orchestrator
node_orchestrator -->|"loads config"| node_config_parser
node_orchestrator -->|"validates config"| node_config_schema
node_orchestrator -.->|"dispatches workload"| node_workload
node_workload -->|"loads dataset"| node_hdf5_loader
node_hdf5_loader -->|"uses utilities"| node_vector_utils
node_workload -->|"computes signatures"| node_hashing
node_workload -->|"collects metrics"| node_metrics
node_workload -.->|"uses ingestion"| node_dedup_ingest
node_workload -.->|"uses query logic"| node_dedup_query
node_workload -.->|"uses ground truth"| node_ground_truth
node_workload -.->|"uses adapter"| node_adapter_base
node_milvus -->|"implements contract"| node_adapter_base
node_qdrant -->|"implements contract"| node_adapter_base
node_milvus -->|"uses errors"| node_adapter_errors
node_qdrant -->|"uses errors"| node_adapter_errors
node_milvus -->|"connects to"| node_vector_db
node_qdrant -->|"connects to"| node_vector_db
node_orchestrator -->|"starts monitoring"| node_monitor
node_orchestrator -->|"generates plots"| node_plotting
node_orchestrator -->|"writes results"| node_results
node_metrics -.->|"provides measurements"| node_results

click node_runner "https://github.com/manishcooks/range-deduplication/blob/main/run.py"
click node_orchestrator "https://github.com/manishcooks/range-deduplication/blob/main/orchestrator/main.py"
click node_config_parser "https://github.com/manishcooks/range-deduplication/blob/main/parser/parser.py"
click node_config_schema "https://github.com/manishcooks/range-deduplication/blob/main/parser/schema.py"
click node_workload "https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/main.py"
click node_hdf5_loader "https://github.com/manishcooks/range-deduplication/blob/main/data_loader/hdf5_loader.py"
click node_vector_utils "https://github.com/manishcooks/range-deduplication/blob/main/data_loader/utils.py"
click node_hashing "https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/hashing.py"
click node_ground_truth "https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/dedup_gt.py"
click node_dedup_ingest "https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/dedup_ingest.py"
click node_dedup_query "https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/dedup_query.py"
click node_adapter_base "https://github.com/manishcooks/range-deduplication/blob/main/database_adapter/base.py"
click node_adapter_errors "https://github.com/manishcooks/range-deduplication/blob/main/database_adapter/exceptions.py"
click node_milvus "https://github.com/manishcooks/range-deduplication/blob/main/database_adapter/milvus_adapter.py"
click node_qdrant "https://github.com/manishcooks/range-deduplication/blob/main/database_adapter/qdrant_adapter.py"
click node_metrics "https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/metrics.py"
click node_monitor "https://github.com/manishcooks/range-deduplication/blob/main/orchestrator/operations/system_monitor.py"
click node_plotting "https://github.com/manishcooks/range-deduplication/blob/main/orchestrator/operations/plotting.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_runner,node_orchestrator,node_config_parser,node_config_schema toneBlue
class node_workload,node_hdf5_loader,node_vector_utils,node_hashing,node_ground_truth,node_dedup_ingest,node_dedup_query,node_vector_db toneAmber
class node_adapter_base,node_adapter_errors,node_milvus,node_qdrant toneMint
class node_metrics,node_monitor,node_plotting toneRose
class node_user,node_results toneIndigo
```

## Components

### Configuration & control

- [CLI entry](https://github.com/manishcooks/range-deduplication/blob/main/run.py): `run.py`
- [Orchestrator](https://github.com/manishcooks/range-deduplication/blob/main/orchestrator/main.py): `orchestrator/main.py`
- [Config parser](https://github.com/manishcooks/range-deduplication/blob/main/parser/parser.py): `parser/parser.py`
- [Config schema](https://github.com/manishcooks/range-deduplication/blob/main/parser/schema.py): `parser/schema.py`

### Deduplication workload

- [Workload runner](https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/main.py): `deduplication_workload/main.py`
- [HDF5 loader](https://github.com/manishcooks/range-deduplication/blob/main/data_loader/hdf5_loader.py): `data_loader/hdf5_loader.py`
- [Vector utilities](https://github.com/manishcooks/range-deduplication/blob/main/data_loader/utils.py): `data_loader/utils.py`
- [MinHash hashing](https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/hashing.py): `deduplication_workload/hashing.py`
- [Dedup ground truth](https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/dedup_gt.py): `deduplication_workload/dedup_gt.py`
- [Dedup ingestion](https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/dedup_ingest.py): `deduplication_workload/dedup_ingest.py`
- [Dedup queries](https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/dedup_query.py): `deduplication_workload/dedup_query.py`

### Database adapters

- [Adapter contract](https://github.com/manishcooks/range-deduplication/blob/main/database_adapter/base.py): `database_adapter/base.py`
- [Adapter errors](https://github.com/manishcooks/range-deduplication/blob/main/database_adapter/exceptions.py): `database_adapter/exceptions.py`
- [Milvus adapter](https://github.com/manishcooks/range-deduplication/blob/main/database_adapter/milvus_adapter.py): `database_adapter/milvus_adapter.py`
- [Qdrant adapter](https://github.com/manishcooks/range-deduplication/blob/main/database_adapter/qdrant_adapter.py): `database_adapter/qdrant_adapter.py`

### Metrics & reporting

- [Workload metrics](https://github.com/manishcooks/range-deduplication/blob/main/deduplication_workload/metrics.py): `deduplication_workload/metrics.py`
- [System monitor](https://github.com/manishcooks/range-deduplication/blob/main/orchestrator/operations/system_monitor.py): `orchestrator/operations/system_monitor.py`
- [Result plotting](https://github.com/manishcooks/range-deduplication/blob/main/orchestrator/operations/plotting.py): `orchestrator/operations/plotting.py`

### Other components

- **Engineer**
- **Vector database**
- **Benchmark outputs**

## Connections

- Engineer → CLI entry: starts run (evidence: `README.md`)
- CLI entry → Orchestrator: calls (evidence: `run.py`)
- Orchestrator → Config parser: loads config (evidence: `orchestrator/main.py`)
- Orchestrator → Config schema: validates config (evidence: `orchestrator/main.py`)
- Orchestrator → Workload runner: dispatches workload
- Workload runner → HDF5 loader: loads dataset (evidence: `deduplication_workload/main.py`)
- HDF5 loader → Vector utilities: uses utilities (evidence: `data_loader/__init__.py`)
- Workload runner → MinHash hashing: computes signatures (evidence: `deduplication_workload/main.py`)
- Workload runner → Workload metrics: collects metrics (evidence: `deduplication_workload/main.py`)
- Workload runner → Dedup ingestion: uses ingestion
- Workload runner → Dedup queries: uses query logic
- Workload runner → Dedup ground truth: uses ground truth
- Workload runner → Adapter contract: uses adapter
- Milvus adapter → Adapter contract: implements contract (evidence: `database_adapter/milvus_adapter.py`)
- Qdrant adapter → Adapter contract: implements contract (evidence: `database_adapter/qdrant_adapter.py`)
- Milvus adapter → Adapter errors: uses errors (evidence: `database_adapter/milvus_adapter.py`)
- Qdrant adapter → Adapter errors: uses errors (evidence: `database_adapter/qdrant_adapter.py`)
- Milvus adapter → Vector database: connects to (evidence: `README.md`)
- Qdrant adapter → Vector database: connects to (evidence: `README.md`)
- Orchestrator → System monitor: starts monitoring (evidence: `orchestrator/main.py`)
- Orchestrator → Result plotting: generates plots (evidence: `orchestrator/main.py`)
- Orchestrator → Benchmark outputs: writes results (evidence: `orchestrator/main.py`)
- Workload metrics → Benchmark outputs: provides measurements
