How bulatenkom/dedup works
Project type and purpose
• This is a lightweight, open‑source, command‑line Python utility (“dedup”) whose sole purpose is to scan a target directory, detect near‑duplicate files (by filename Levenshtein distance and file‑size threshold), and optionally delete the duplicates while preserving one copy of each group.
File structure
• Top‑level script: dedup.py – contains the entire application logic.
• Configuration / dependency file: requirements.txt – pulls in the Jellyfish library for string distance.
• No separate frontend or backend directories: it’s a single‑process CLI tool.
Extract insights from the README
• Core modules (implied by usage flags): argument parser, file enumerator, similarity calculator (Levenshtein via Jellyfish), duplicate grouper, deletion engine, interactive confirmation.
• Optional future features: recursive search, backups, glob filters, progress bars, STDOUT redirection, etc.
Main components
a. CLI Argument Parser
• Reads filepath, --filename-distance, --filesize-threshold, --suffix, --skip-confirmation, --debug.
b. File Scanner
• Walks the target directory (non‑recursive by default; recursive if implemented). Filters files by suffix.
c. Duplicate Detector
• Calculates pairwise or bucketed Levenshtein distances via the Jellyfish library. Compares file sizes. Groups files into “original + duplicates.”
d. Confirmation & Logger
• Presents grouped files to the user on STDOUT. Waits for “y/n” input (unless --skip-confirmation).
e. Deletion Engine
• Receives the list of duplicates to remove, deletes them from the file system.
f. Python Runtime / Virtual Environment
• Underpins all components and loads external dependency Jellyfish.
Relationships and data flow
• User invokes python dedup.py → CLI Argument Parser → File Scanner → Duplicate Detector → Confirmation & Logger → (if yes) → Deletion Engine → STDOUT “Done.”
• All components run in a single process; Jellyfish is called by the Duplicate Detector.
Architectural patterns and design principles
• Single‑process pipeline (sequential modules).
• Modular separation by function: parsing, scanning, detection, confirmation, deletion.
• Extensibility hooks (e.g., you could swap in a different distance algorithm or add recursion).
Technologies, frameworks, libraries
• Python 3.10+ (virtualenv)
• Jellyfish (MIT) for Levenshtein
• Standard library modules: argparse, os/pathlib, sys, subprocess (if spawning), logging.
Key takeaways
• Five core modules plus the runtime and external library.
• Be detailed in labeling: thresholds, flags, and decision points (confirmation).