How bulatenkom/dedup works
<explanation>
1. Identify the project type and purpose
- This is a lightweight, open‑source, command‑line Python utility (“dedup”) whose sole purpose is to scan a target directory, detect near‑duplicate files (by filename Levenshtein distance and file‑size threshold), and optionally delete the duplicates while preserving one copy of each group.
2. Analyze the file structure
- Top‑level script: dedup.py – contains the entire application logic.
- Configuration / dependency file: requirements.txt – pulls in the Jellyfish library for string distance.
- No separate frontend or backend directories: it’s a single‑process CLI tool.
3. Extract insights from the README
- Core modules (implied by usage flags): argument parser, file enumerator, similarity calculator (Levenshtein via Jellyfish), duplicate grouper, deletion engine, interactive confirmation.
- Optional future features: recursive search, backups, glob filters, progress bars, STDOUT redirection, etc.
4. Main components for the system diagram
a. CLI Argument Parser
- Reads filepath, --filename-distance, --filesize-threshold, --suffix, --skip-confirmation, --debug.
b. File Scanner
- Walks the target directory (non‑recursive by default; recursive if implemented). Filters files by suffix.
c. Duplicate Detector
- Calculates pairwise or bucketed Levenshtein distances via the Jellyfish library. Compares file sizes. Groups files into “original + duplicates.”
d. Confirmation & Logger
- Presents grouped files to the user on STDOUT. Waits for “y/n” input (unless --skip-confirmation).
e. Deletion Engine
- Receives the list of duplicates to remove, deletes them from the file system.
f. Python Runtime / Virtual Environment
- Underpins all components and loads external dependency Jellyfish.
5. Relationships and data flow
- User invokes python dedup.py → CLI Argument Parser → File Scanner → Duplicate Detector → Confirmation & Logger → (if yes) → Deletion Engine → STDOUT “Done.”
- All components run in a single process; Jellyfish is called by the Duplicate Detector.
6. Architectural patterns and design principles
- Single‑process pipeline (sequential modules).
- Modular separation by function: parsing, scanning, detection, confirmation, deletion.
- Extensibility hooks (e.g., you could swap in a different distance algorithm or add recursion).
7. Technologies, frameworks, libraries
- Python 3.10+ (virtualenv)
- Jellyfish (MIT) for Levenshtein
- Standard library modules: argparse, os/pathlib, sys, subprocess (if spawning), logging.
8. Tailoring the diagram for an open‑source CLI tool
- Emphasize core pipeline modules as boxes in a left‑to‑right flow.
- Highlight the extension points (e.g., “Distance Algorithm” box that can be replaced).
- Show integration with external library (Jellyfish) as a separate box or service.
- Optionally annotate where future features (recursive search, backups) would plug in.
9. Diagramming guidelines
- Use clear rectangular boxes for each module: Parser, Scanner, Detector, Confirm, Delete.
- Draw directional arrows to show data flow: CLI input → Parser → Scanner → Detector → Confirmation → Deletion → Output.
- Color‑code:
• Blue for input/output (CLI, STDOUT)
• Green for core processing modules (Scanner, Detector)
• Orange for external dependencies (Jellyfish)
• Red for destructive actions (Deletion Engine)
- Use dashed arrows for optional or future flows (e.g., recursive search, backup step).
- Label each arrow with the data passed (e.g., “file list”, “grouped duplicates”, “user response”).
10. Key takeaways
- Keep the diagram linear but annotated with extension points.
- Capture all five core modules plus the runtime and external library.
- Stick to simple shapes and a consistent left‑to‑right flow so anyone can grasp the dedup pipeline immediately.
- Be detailed in labeling: thresholds, flags, and decision points (confirmation).
</explanation>