# gigachad80/del-packurls architecture

> Architecture diagram of the GitHub repository gigachad80/del-packurls, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/gigachad80/del-packurls
- Repository: https://github.com/gigachad80/del-packurls

## Overview

##### Project type and purpose
- del-packURLs is a command-line “bug-hunting” automation tool (not a web UI or microservice).
- It is written in Go for core URL fetching/filtering and in Python for PDF keyword analysis and AI-based suggestions.
- It retrieves archived URLs via the Wayback Machine CDX API, filters for user-specified extensions, checks historical 200 OK status, downloads the files (with concurrency), and then analyzes sensitive PDFs using Python scripts (sort-keywords.py) and AI suggestion (ai-suggestor.py).

##### File structure
Top level:
- .env, LICENSE, README.md
- del-packURLs.go & go.mod (Go core engine)
- ai-suggestor.py & sort-keywords.py (Python analysis pipelines)
- requirements.txt (Python dependencies for AI models & PyMuPDF)
- ai-suggestor.py (invokes AI APIs – Gemini, Claude, GPT)
Patterns:
- Single executable Go module that orchestrates HTTP requests, URL filtering, download concurrency, then invokes Python scripts as subprocesses.
- Two-stage pipeline: Go core → Python analysis.

##### Key insights from the README
- Core stages: CLI input → CDX API query → filter by extension → “200 History Mode” → concurrent download & classification → PDF text search → AI suggestion.
- Concurrency flag in Go accelerates downloads and PDF analysis.
- Python requirements install multiple AI model clients; each may be invoked via HTTP.
- Outputs: terminal logs, sorted-keywords.txt, on-screen AI recommendations.

##### Main components
a. User’s Terminal (CLI interface)
b. Go Core Engine
– CLI Parser
– CDX API Client
– URL Filter Module
– 200-Status Checker
– Download Manager (with concurrency)
– Subprocess Invoker (Python scripts)
c. Python Analysis Engine
– sort-keywords.py (PDF downloader & text keyword scanner using PyMuPDF)
– ai-suggestor.py (AI model client for sensitive-PDF suggestions)
d. External Services
– Wayback Machine CDX API (HTTP)
– AI Model Endpoints (Gemini, Claude, GPT)
e. Local File System
– Temporary download directory
– Output file (sorted-keywords.txt)

##### Relationships & data flows
1. User invokes del-packURLs.go via CLI
2. Go Core Engine sends HTTP requests to CDX API
3. API returns archived URL list
4. Go filters by extension and queries historical status
5. Download Manager fetches each file concurrently
6. For each PDF:
– Go spawns sort-keywords.py to scan for sensitive terms → writes to sorted-keywords.txt
– Go invokes ai-suggestor.py to call AI APIs → prints suggestions to terminal
7. Other file types (apk, dll, zip, etc.) may simply be logged or skipped

##### Architectural patterns & design principles
- Pipeline / modular design: each stage is a separate module or subprocess
- Concurrency in Go for high throughput of network I/O
- Polyglot approach: Go for speed & networking, Python for rich AI/ML libraries
- Clear separation of concerns: fetching/filtering vs. analysis

##### Technologies, frameworks, libraries
- Go standard library: net/http, flag, sync (goroutines)
- Python 3 + PyMuPDF (pdf text extraction)
- AI model SDKs (as listed in requirements.txt)
- External HTTP APIs: Wayback CDX end-point, AI model endpoints

##### Diagramming recommendations
User → Go CLI → CDX API → Go Filter → Download Manager → Python modules → File system & Terminal
- Blue for Go modules
- Green for Python analysis
- Gray/white for file system
- Cloud-shaded for external services

##### Essential details to capture
- CLI invocation & parsing
- Two distinct language runtimes (Go, Python)
- Concurrency mechanism in Download Manager
- External service calls: CDX API vs. AI APIs
- Output artifacts (terminal logs, sorted-keywords.txt)
- Error/ status flows (200 vs. 404 handling)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External User
    User["User Terminal (CLI)"]:::external

    %% Go Core Engine
    subgraph "Go Core Engine"
        CLI["CLI Parser"]:::go
        CDX["CDX API Client"]:::go
        Filter["URL Filter"]:::go
        Status["200-Status Checker"]:::go
        Download["Download Manager\n(concurrency)"]:::go
        Subproc["Subprocess Invoker"]:::go
    end

    %% Python Analysis Engine
    subgraph "Python Analysis Engine"
        Scanner["sort-keywords.py\n(PDF Keyword Scanner)"]:::py
        AISuggest["ai-suggestor.py\n(AI Suggestor)"]:::py
    end

    %% External Services
    CDXAPI["Wayback Machine CDX API"]:::external
    AIAPIs["AI Model Endpoints\n(Gemini/Claude/GPT)"]:::external

    %% File System
    TempDir["Temporary Download Directory"]:::fs
    OutputFile["sorted-keywords.txt"]:::fs
    TerminalOut["Terminal Logs & AI Suggestions"]:::fs

    %% Project Files
    subgraph "Project Files"
        Env[".env"]:::file
        GoMod["go.mod"]:::file
        Req["requirements.txt"]:::file
        Readme["README.md"]:::file
        LicenseF["LICENSE"]:::file
    end

    %% Data Flows
    User -->|"invoke CLI"| CLI
    CLI -->|"INIT CDX Query"| CDX
    CDX -->|"GET /cdx"| CDXAPI
    CDXAPI -->|"list of URLs"| Filter
    Filter -->|"filtered URLs"| Status
    Status -->|"200 OK URLs"| Download
    Download -->|"download files"| TempDir
    Download -->|"spawn analysis"| Subproc
    Subproc -->|"spawn process"| Scanner
    Scanner -->|"extracted keywords"| OutputFile
    Subproc -->|"spawn process"| AISuggest
    AISuggest -->|"HTTP requests"| AIAPIs
    AIAPIs -->|"AI suggestions"| TerminalOut
    AISuggest -->|"print suggestions"| TerminalOut

    %% Click Events
    click CLI "https://github.com/gigachad80/del-packurls/blob/main/del-packURLs.go"
    click CDX "https://github.com/gigachad80/del-packurls/blob/main/del-packURLs.go"
    click Filter "https://github.com/gigachad80/del-packurls/blob/main/del-packURLs.go"
    click Status "https://github.com/gigachad80/del-packurls/blob/main/del-packURLs.go"
    click Download "https://github.com/gigachad80/del-packurls/blob/main/del-packURLs.go"
    click Subproc "https://github.com/gigachad80/del-packurls/blob/main/del-packURLs.go"
    click GoMod "https://github.com/gigachad80/del-packurls/blob/main/go.mod"
    click Scanner "https://github.com/gigachad80/del-packurls/blob/main/sort-keywords.py"
    click AISuggest "https://github.com/gigachad80/del-packurls/blob/main/ai-suggestor.py"
    click Req "https://github.com/gigachad80/del-packurls/blob/main/requirements.txt"
    click Env "https://github.com/gigachad80/del-packurls/blob/main/.env"
    click Readme "https://github.com/gigachad80/del-packurls/blob/main/README.md"
    click LicenseF "https://github.com/gigachad80/del-packurls/tree/main/LICENSE"

    %% Styles
    classDef go fill:#aec7e8,stroke:#333;
    classDef py fill:#98df8a,stroke:#333;
    classDef external fill:#f7b6d2,stroke:#333;
    classDef fs fill:#c7c7c7,stroke:#333;
    classDef file fill:#fff2ae,stroke:#333;
```
