How amandev17/deduplication works
Document Deduplication Pipeline is a Django web app for finding duplicate pages across uploaded PDFs, Word documents, and images. Users choose a file or folder comparison mode, upload inputs, and follow a background job through progress to a report of duplicate clusters and unique pages with thumbnails. The core pipeline ingests and normalizes pages, hashes them, narrows candidate pairs, compares similarity, clusters confirmed matches, and writes a JSON report. The README documents these capabilities; the sampled source shows the main pipeline wiring, while some UI and integration details remain unsampled.
Main entry points: run_pipeline.py Pipeline runner · views.py Upload and jobs · reporter.py Report writer · urls.py App routes
Components 13
Web application
- Upload and jobs
dedup_app/views.py - App routes
dedup_app/urls.py - Web UI
dedup_app/templates/dedup_app
Page detection
- Page ingestion
dedup_app/pipeline/ingestion.py - Page hashing
dedup_app/pipeline/hashing.py - Candidate pairs
dedup_app/pipeline/nodes.py - Pair comparison
dedup_app/pipeline/similarity.py - Pipeline runner
dedup_app/pipeline/run_pipeline.py
Clustering and reports
- Match graph
dedup_app/pipeline/graph.py - Report writer
dedup_app/pipeline/reporter.py - JSON report and thumbnails
dedup_app/pipeline/reporter.py - Media storage
dedup_project/settings.py
Other components
- User
Connections 13
- User → Upload and jobs: uploads files
dedup_app/views.py - App routes → Upload and jobs: routes requests
dedup_app/urls.py - Upload and jobs → Web UI: renders pages
dedup_app/views.py - Upload and jobs → Pipeline runner: starts pipeline
dedup_app/views.py - Pipeline runner → Page ingestion: reads pages
dedup_app/pipeline/run_pipeline.py - Pipeline runner → Page hashing: hashes pages
dedup_app/pipeline/run_pipeline.py - Pipeline runner → Candidate pairs: builds pairs
dedup_app/pipeline/run_pipeline.py - Pipeline runner → Pair comparison: compares pairs
dedup_app/pipeline/run_pipeline.py - Pipeline runner → Match graph: clusters matches
dedup_app/pipeline/run_pipeline.py - Pipeline runner → Report writer: writes results
dedup_app/pipeline/run_pipeline.py - Report writer → JSON report and thumbnails: writes JSON and images
dedup_app/pipeline/reporter.py - Report writer → Media storage: stores outputs
dedup_app/pipeline/reporter.py - Upload and jobs → JSON report and thumbnails: loads results
dedup_app/views.py