How manishcooks/range-deduplication works
This is a vector-database range-deduplication benchmarking framework. The principal workflow loads an HDF5 dataset, runs a two-phase Bloom-filter and MinHash-LSH deduplication workload against a configured database adapter, and records benchmark results; orchestration can also collect system metrics and generate plots. The graph preserves the documented adapter, configuration, monitoring, and plotting capabilities. Some orchestration-to-workload and adapter-selection wiring is not visible in the excerpts, so those links are dashed rather than asserted as inspected calls.
Main entry points: deduplication_workload/main.py Workload runner · orchestrator/main.py Orchestrator · milvus_adapter.py Milvus adapter · qdrant_adapter.py Qdrant adapter
Components 21
Configuration & control
- CLI entry
run.py - Orchestrator
orchestrator/main.py - Config parser
parser/parser.py - Config schema
parser/schema.py
Deduplication workload
- Workload runner
deduplication_workload/main.py - HDF5 loader
data_loader/hdf5_loader.py - Vector utilities
data_loader/utils.py - MinHash hashing
deduplication_workload/hashing.py - Dedup ground truth
deduplication_workload/dedup_gt.py - Dedup ingestion
deduplication_workload/dedup_ingest.py - Dedup queries
deduplication_workload/dedup_query.py
Database adapters
- Adapter contract
database_adapter/base.py - Adapter errors
database_adapter/exceptions.py - Milvus adapter
database_adapter/milvus_adapter.py - Qdrant adapter
database_adapter/qdrant_adapter.py
Metrics & reporting
- Workload metrics
deduplication_workload/metrics.py - System monitor
orchestrator/operations/system_monitor.py - Result plotting
orchestrator/operations/plotting.py
Other components
- Engineer
- Vector database
- Benchmark outputs
Connections 23
- Engineer → CLI entry: starts run
README.md - CLI entry → Orchestrator: calls
run.py - Orchestrator → Config parser: loads config
orchestrator/main.py - Orchestrator → Config schema: validates config
orchestrator/main.py - Orchestrator → Workload runner: dispatches workload
- Workload runner → HDF5 loader: loads dataset
deduplication_workload/main.py - HDF5 loader → Vector utilities: uses utilities
data_loader/__init__.py - Workload runner → MinHash hashing: computes signatures
deduplication_workload/main.py - Workload runner → Workload metrics: collects metrics
deduplication_workload/main.py - Workload runner → Dedup ingestion: uses ingestion
- Workload runner → Dedup queries: uses query logic
- Workload runner → Dedup ground truth: uses ground truth
- Workload runner → Adapter contract: uses adapter
- Milvus adapter → Adapter contract: implements contract
database_adapter/milvus_adapter.py - Qdrant adapter → Adapter contract: implements contract
database_adapter/qdrant_adapter.py - Milvus adapter → Adapter errors: uses errors
database_adapter/milvus_adapter.py - Qdrant adapter → Adapter errors: uses errors
database_adapter/qdrant_adapter.py - Milvus adapter → Vector database: connects to
README.md - Qdrant adapter → Vector database: connects to
README.md - Orchestrator → System monitor: starts monitoring
orchestrator/main.py - Orchestrator → Result plotting: generates plots
orchestrator/main.py - Orchestrator → Benchmark outputs: writes results
orchestrator/main.py - Workload metrics → Benchmark outputs: provides measurements