# firecat53/urlscan architecture

> Architecture diagram of the GitHub repository firecat53/urlscan, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/firecat53/urlscan
- Repository: https://github.com/firecat53/urlscan

## Overview

##### Project Type and Purpose
- Urlscan is a standalone, open-source CLI utility (Python-based) that extracts URLs (and email addresses) from arbitrary text or email messages, presents them in a TUI (Urwid) interface, and lets the user open them in a browser (or pass them to any external command). It primarily integrates with mutt/neomutt via shell pipes.

##### File Structure Analysis
- Top-level scripts/config:
- urlscan (POSIX script wrapper or entry point)
- urlscan.1 (man page)
- pyproject.toml, requirements.txt, flake.nix, flake.lock (build/packaging/dev-shell definitions)
- .github/workflows/main.yml (CI pipeline, PyPI/TestPyPI deploy)
- Core Python package: urlscan/
- __main__.py – the console entry point (`python -m urlscan`)
- urlscan.py – main logic: argument parsing, orchestration
- urlchoose.py – the Urwid-based TUI for choosing URLs and context display
- assets/tlds-alpha-by-domain.txt – trusted TLD list used by URL validator
- __init__.py – package metadata

##### README Insights
- Describes architecture in prose:
- URL extraction and regex engine
- Context rendering (plain text + crude HTML → text converter)
- Interactive TUI via Urwid
- Browser/command invocation abstraction (webbrowser, xdg-open, or user-provided commands)
- Packaging/dev via Hatch, flake.nix, GitHub Actions

##### Main Components
a. Input Layer
- STDIN or file argument (email message or arbitrary text)
- Pipe from mutt (`<pipe-message> urlscan`)
b. URL Extraction Engine
- Core module in urlscan.py
- Regex patterns (built-in or user-supplied via `-E`)
- Deduplication, reversing, context cropping
- HTML-to-text converter for context (inside urlscan.py)
c. Configuration Manager
- Falls back to default theme embedded in assets or code
d. UI Layer (TUI)
- urlchoose.py uses Urwid widgets
- Presents list of URLs + context
- Handles keybindings (a, o, s, c, etc.)
e. Action Handlers / Output Layer
- Browser invoker (python webbrowser or xdg-open)
- Custom commands (`--run` / `--run-safe`)
- Clipboard handlers (xsel / xclip)
- Stdout pipe (`-p` flag outputs list)
f. External Dependencies & Integrations
- Python 3.7+ runtime
- Urwid library (TUI)
- xdg-open, xclip/xsel (optional)
- Mutt / Neomutt macro integration

##### Relationships & Interactions
1. User invokes `urlscan` (or shell macro via mutt).
2. Input Layer feeds raw message/text to URL Extraction Engine.
3. Extraction Engine parses text, applies regex, extracts URLs + context.
4. Configuration Manager loads user settings → passed to UI Layer.
5. UI Layer displays results and captures keystrokes.
6. On selection, UI signals Action Handler → external browser/command/clipboard.
7. Exit or repeat until quit.

##### Architectural Patterns & Design Principles
- Modular separation: extraction logic vs. UI vs. action handlers
- Pluggable command runner (strategy pattern for browser/xdg-open/custom)
- Configurable via JSON (external configuration)
- Single responsibility modules: urlscan.py (engine), urlchoose.py (UI)
- Assets folder for static data (TLD list)

##### Detailed Elements to Capture
- Entry point script & __main__.py
- assets/tlds-alpha-by-domain.txt (validation resource)
- Config file location and format
- Keybinding cycle (p), context toggle (c), shorten toggle (s)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Main Pipeline
    subgraph "CLI Invocation"
        direction TB
        User["User / Mutt"]:::external
        EntryScript["Entry Point Script: urlscan"]:::internal
        ManPage["Man Page: urlscan.1"]:::config
    end

    subgraph "Console Entry"
        direction TB
        MainModule["Console Entry Point: __main__.py"]:::internal
        click EntryScript "https://github.com/firecat53/urlscan/tree/main/urlscan"
        click ManPage "https://github.com/firecat53/urlscan/blob/main/urlscan.1"
        click MainModule "https://github.com/firecat53/urlscan/blob/main/urlscan/__main__.py"
    end

    subgraph "Extraction Engine"
        direction TB
        Engine["URL Extraction Engine: urlscan.py"]:::internal
        Asset["TLD List Asset"]:::config
        click Engine "https://github.com/firecat53/urlscan/blob/main/urlscan/urlscan.py"
        click Asset "https://github.com/firecat53/urlscan/blob/main/urlscan/assets/tlds-alpha-by-domain.txt"
    end

    subgraph "Configuration Manager"
        direction TB
        UserConfig["User Config: ~/.config/urlscan/config.json"]:::config
        DefaultConfig["Default Theme/Assets"]:::internal
    end

    subgraph "UI Layer"
        direction TB
        UI["Interactive TUI: urlchoose.py"]:::internal
        click UI "https://github.com/firecat53/urlscan/blob/main/urlscan/urlchoose.py"
    end

    subgraph "Action Handlers"
        direction TB
        Browser["Browser Invoker (webbrowser/xdg-open)"]:::external
        CustomCmd["Custom Commands (--run/--run-safe)"]:::external
        Clipboard["Clipboard Handler (xsel/xclip)"]:::external
        Stdout["Stdout Pipe (-p)"]:::external
    end

    subgraph "External Dependencies"
        direction TB
        Python["Python 3.7+"]:::external
        Urwid["Urwid Library"]:::external
        Mutt["Mutt/Neomutt"]:::external
        click Python "https://github.com/firecat53/urlscan/blob/main/requirements.txt"
        click Urwid "https://github.com/firecat53/urlscan/blob/main/requirements.txt"
        click Mutt "https://github.com/firecat53/urlscan/blob/main/.gitignore"
    end

    subgraph "Build & CI/CD"
        direction TB
        Pyproject["Project Metadata: pyproject.toml"]:::config
        Requirements["Dependencies: requirements.txt"]:::config
        FlakeNix["Nix Shell & Packaging: flake.nix, flake.lock"]:::config
        CI["CI/CD Pipeline: .github/workflows/main.yml"]:::config
        Docs["Documentation: README.md"]:::config
        Gitignore["Git Ignore Rules"]:::config
        click Pyproject "https://github.com/firecat53/urlscan/blob/main/pyproject.toml"
        click Requirements "https://github.com/firecat53/urlscan/blob/main/requirements.txt"
        click FlakeNix "https://github.com/firecat53/urlscan/blob/main/flake.nix"
        click FlakeNix "https://github.com/firecat53/urlscan/blob/main/flake.lock"
        click CI "https://github.com/firecat53/urlscan/blob/main/.github/workflows/main.yml"
        click Docs "https://github.com/firecat53/urlscan/blob/main/README.md"
        click Gitignore "https://github.com/firecat53/urlscan/blob/main/.gitignore"
    end

    %% Flows
    User -->|invokes shell| EntryScript
    EntryScript --> MainModule
    MainModule --> Engine
    Engine --> Asset
    Engine --> UI
    MainModule --> UserConfig
    MainModule --> DefaultConfig
    UserConfig --> UI
    DefaultConfig --> UI
    UI --> Browser
    UI --> CustomCmd
    UI --> Clipboard
    UI --> Stdout

    %% Dependencies
    MainModule --> Python
    Engine --> Python
    UI --> Urwid
    EntryScript --> Mutt

    %% Styling
    classDef internal fill:#cce5ff,stroke:#0366d6;
    classDef external fill:#d4edda,stroke:#28a745;
    classDef config fill:#ffe5b4,stroke:#d9822b;
```
