# amandev17/deduplication architecture

> Architecture diagram of the GitHub repository amandev17/deduplication, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-11-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/amandev17/deduplication
- Repository: https://github.com/amandev17/deduplication

## Overview

Document Deduplication Pipeline is a Django web app for finding duplicate pages across uploaded PDFs, Word documents, and images. Users choose a file or folder comparison mode, upload inputs, and follow a background job through progress to a report of duplicate clusters and unique pages with thumbnails. The core pipeline ingests and normalizes pages, hashes them, narrows candidate pairs, compares similarity, clusters confirmed matches, and writes a JSON report. The README documents these capabilities; the sampled source shows the main pipeline wiring, while some UI and integration details remain unsampled.

Main entry points: [`dedup_app/pipeline/run_pipeline.py`](https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/run_pipeline.py) (Pipeline runner), [`dedup_app/views.py`](https://github.com/amandev17/deduplication/blob/main/dedup_app/views.py) (Upload and jobs), [`dedup_app/pipeline/reporter.py`](https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/reporter.py) (Report writer), [`dedup_app/urls.py`](https://github.com/amandev17/deduplication/blob/main/dedup_app/urls.py) (App routes)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_web["Web application"]
  node_views["Upload and jobs<br/>[views.py]"]
  node_routes["App routes<br/>[urls.py]"]
  node_templates["Web UI"]
end

subgraph group_detect["Page detection"]
  node_ingestion["Page ingestion<br/>[ingestion.py]"]
  node_hashing["Page hashing<br/>[hashing.py]"]
  node_candidates["Candidate pairs<br/>[nodes.py]"]
  node_similarity["Pair comparison<br/>[similarity.py]"]
  node_orchestrator["Pipeline runner<br/>[run_pipeline.py]"]
end

subgraph group_results["Clustering and reports"]
  node_graph["Match graph<br/>[graph.py]"]
  node_reporter["Report writer<br/>[reporter.py]"]
  node_report["JSON report and thumbnails<br/>[reporter.py]"]
  node_media[("Media storage<br/>[settings.py]")]
end

node_user(("User"))

node_user -->|"uploads files"| node_views
node_routes -->|"routes requests"| node_views
node_views -->|"renders pages"| node_templates
node_views -->|"starts pipeline"| node_orchestrator
node_orchestrator -->|"reads pages"| node_ingestion
node_orchestrator -->|"hashes pages"| node_hashing
node_orchestrator -->|"builds pairs"| node_candidates
node_orchestrator -->|"compares pairs"| node_similarity
node_orchestrator -->|"clusters matches"| node_graph
node_orchestrator -->|"writes results"| node_reporter
node_reporter -->|"writes JSON and images"| node_report
node_reporter -->|"stores outputs"| node_media
node_views -->|"loads results"| node_report

click node_views "https://github.com/amandev17/deduplication/blob/main/dedup_app/views.py"
click node_routes "https://github.com/amandev17/deduplication/blob/main/dedup_app/urls.py"
click node_templates "https://github.com/amandev17/deduplication/tree/main/dedup_app/templates/dedup_app"
click node_ingestion "https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/ingestion.py"
click node_hashing "https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/hashing.py"
click node_candidates "https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/nodes.py"
click node_similarity "https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/similarity.py"
click node_orchestrator "https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/run_pipeline.py"
click node_graph "https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/graph.py"
click node_reporter "https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/reporter.py"
click node_report "https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/reporter.py"
click node_media "https://github.com/amandev17/deduplication/blob/main/dedup_project/settings.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_views,node_routes,node_templates,node_user toneBlue
class node_ingestion,node_hashing,node_candidates,node_similarity,node_orchestrator toneAmber
class node_graph,node_reporter,node_report,node_media toneMint
```

## Components

### Web application

- [Upload and jobs](https://github.com/amandev17/deduplication/blob/main/dedup_app/views.py): `dedup_app/views.py`
- [App routes](https://github.com/amandev17/deduplication/blob/main/dedup_app/urls.py): `dedup_app/urls.py`
- [Web UI](https://github.com/amandev17/deduplication/tree/main/dedup_app/templates/dedup_app): `dedup_app/templates/dedup_app`

### Page detection

- [Page ingestion](https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/ingestion.py): `dedup_app/pipeline/ingestion.py`
- [Page hashing](https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/hashing.py): `dedup_app/pipeline/hashing.py`
- [Candidate pairs](https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/nodes.py): `dedup_app/pipeline/nodes.py`
- [Pair comparison](https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/similarity.py): `dedup_app/pipeline/similarity.py`
- [Pipeline runner](https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/run_pipeline.py): `dedup_app/pipeline/run_pipeline.py`

### Clustering and reports

- [Match graph](https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/graph.py): `dedup_app/pipeline/graph.py`
- [Report writer](https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/reporter.py): `dedup_app/pipeline/reporter.py`
- [JSON report and thumbnails](https://github.com/amandev17/deduplication/blob/main/dedup_app/pipeline/reporter.py): `dedup_app/pipeline/reporter.py`
- [Media storage](https://github.com/amandev17/deduplication/blob/main/dedup_project/settings.py): `dedup_project/settings.py`

### Other components

- **User**

## Connections

- User → Upload and jobs: uploads files (evidence: `dedup_app/views.py`)
- App routes → Upload and jobs: routes requests (evidence: `dedup_app/urls.py`)
- Upload and jobs → Web UI: renders pages (evidence: `dedup_app/views.py`)
- Upload and jobs → Pipeline runner: starts pipeline (evidence: `dedup_app/views.py`)
- Pipeline runner → Page ingestion: reads pages (evidence: `dedup_app/pipeline/run_pipeline.py`)
- Pipeline runner → Page hashing: hashes pages (evidence: `dedup_app/pipeline/run_pipeline.py`)
- Pipeline runner → Candidate pairs: builds pairs (evidence: `dedup_app/pipeline/run_pipeline.py`)
- Pipeline runner → Pair comparison: compares pairs (evidence: `dedup_app/pipeline/run_pipeline.py`)
- Pipeline runner → Match graph: clusters matches (evidence: `dedup_app/pipeline/run_pipeline.py`)
- Pipeline runner → Report writer: writes results (evidence: `dedup_app/pipeline/run_pipeline.py`)
- Report writer → JSON report and thumbnails: writes JSON and images (evidence: `dedup_app/pipeline/reporter.py`)
- Report writer → Media storage: stores outputs (evidence: `dedup_app/pipeline/reporter.py`)
- Upload and jobs → JSON report and thumbnails: loads results (evidence: `dedup_app/views.py`)