How gigachad80/del-packurls works
Project type and purpose
• del-packURLs is a command-line “bug-hunting” automation tool (not a web UI or microservice).
• It is written in Go for core URL fetching/filtering and in Python for PDF keyword analysis and AI-based suggestions.
• It retrieves archived URLs via the Wayback Machine CDX API, filters for user-specified extensions, checks historical 200 OK status, downloads the files (with concurrency), and then analyzes sensitive PDFs using Python scripts (sort-keywords.py) and AI suggestion (ai-suggestor.py).
File structure
Top level:
• .env, LICENSE, README.md
• del-packURLs.go & go.mod (Go core engine)
• ai-suggestor.py & sort-keywords.py (Python analysis pipelines)
• requirements.txt (Python dependencies for AI models & PyMuPDF)
• ai-suggestor.py (invokes AI APIs – Gemini, Claude, GPT)
Patterns:
• Single executable Go module that orchestrates HTTP requests, URL filtering, download concurrency, then invokes Python scripts as subprocesses.
• Two-stage pipeline: Go core → Python analysis.
Key insights from the README
• Core stages: CLI input → CDX API query → filter by extension → “200 History Mode” → concurrent download & classification → PDF text search → AI suggestion.
• Concurrency flag in Go accelerates downloads and PDF analysis.
• Python requirements install multiple AI model clients; each may be invoked via HTTP.
• Outputs: terminal logs, sorted-keywords.txt, on-screen AI recommendations.
Main components
a. User’s Terminal (CLI interface)
b. Go Core Engine
– CLI Parser
– CDX API Client
– URL Filter Module
– 200-Status Checker
– Download Manager (with concurrency)
– Subprocess Invoker (Python scripts)
c. Python Analysis Engine
– sort-keywords.py (PDF downloader & text keyword scanner using PyMuPDF)
– ai-suggestor.py (AI model client for sensitive-PDF suggestions)
d. External Services
– Wayback Machine CDX API (HTTP)
– AI Model Endpoints (Gemini, Claude, GPT)
e. Local File System
– Temporary download directory
– Output file (sorted-keywords.txt)
Relationships & data flows
1. User invokes del-packURLs.go via CLI
2. Go Core Engine sends HTTP requests to CDX API
3. API returns archived URL list
4. Go filters by extension and queries historical status
5. Download Manager fetches each file concurrently
6. For each PDF:
– Go spawns sort-keywords.py to scan for sensitive terms → writes to sorted-keywords.txt
– Go invokes ai-suggestor.py to call AI APIs → prints suggestions to terminal
7. Other file types (apk, dll, zip, etc.) may simply be logged or skipped
Architectural patterns & design principles
• Pipeline / modular design: each stage is a separate module or subprocess
• Concurrency in Go for high throughput of network I/O
• Polyglot approach: Go for speed & networking, Python for rich AI/ML libraries
• Clear separation of concerns: fetching/filtering vs. analysis
Technologies, frameworks, libraries
• Go standard library: net/http, flag, sync (goroutines)
• Python 3 + PyMuPDF (pdf text extraction)
• AI model SDKs (as listed in requirements.txt)
• External HTTP APIs: Wayback CDX end-point, AI model endpoints
Diagramming recommendations
User → Go CLI → CDX API → Go Filter → Download Manager → Python modules → File system & Terminal
• Blue for Go modules
• Green for Python analysis
• Gray/white for file system
• Cloud-shaded for external services
Essential details to capture
• CLI invocation & parsing
• Two distinct language runtimes (Go, Python)
• Concurrency mechanism in Download Manager
• External service calls: CDX API vs. AI APIs
• Output artifacts (terminal logs, sorted-keywords.txt)
• Error/ status flows (200 vs. 404 handling)