How firecat53/urlscan works
Project Type and Purpose
• Urlscan is a standalone, open-source CLI utility (Python-based) that extracts URLs (and email addresses) from arbitrary text or email messages, presents them in a TUI (Urwid) interface, and lets the user open them in a browser (or pass them to any external command). It primarily integrates with mutt/neomutt via shell pipes.
File Structure Analysis
• Top-level scripts/config:
• urlscan (POSIX script wrapper or entry point)
• urlscan.1 (man page)
• pyproject.toml, requirements.txt, flake.nix, flake.lock (build/packaging/dev-shell definitions)
• .github/workflows/main.yml (CI pipeline, PyPI/TestPyPI deploy)
• Core Python package: urlscan/
• __main__.py – the console entry point (python -m urlscan)
• urlscan.py – main logic: argument parsing, orchestration
• urlchoose.py – the Urwid-based TUI for choosing URLs and context display
• assets/tlds-alpha-by-domain.txt – trusted TLD list used by URL validator
• __init__.py – package metadata
README Insights
• Describes architecture in prose:
• URL extraction and regex engine
• Context rendering (plain text + crude HTML → text converter)
• Interactive TUI via Urwid
• Browser/command invocation abstraction (webbrowser, xdg-open, or user-provided commands)
• Packaging/dev via Hatch, flake.nix, GitHub Actions
Main Components
a. Input Layer
• STDIN or file argument (email message or arbitrary text)
• Pipe from mutt (<pipe-message> urlscan)
b. URL Extraction Engine
• Core module in urlscan.py
• Regex patterns (built-in or user-supplied via -E)
• Deduplication, reversing, context cropping
• HTML-to-text converter for context (inside urlscan.py)
c. Configuration Manager
• Falls back to default theme embedded in assets or code
d. UI Layer (TUI)
• urlchoose.py uses Urwid widgets
• Presents list of URLs + context
• Handles keybindings (a, o, s, c, etc.)
e. Action Handlers / Output Layer
• Browser invoker (python webbrowser or xdg-open)
• Custom commands (--run / --run-safe)
• Clipboard handlers (xsel / xclip)
• Stdout pipe (-p flag outputs list)
f. External Dependencies & Integrations
• Python 3.7+ runtime
• Urwid library (TUI)
• xdg-open, xclip/xsel (optional)
• Mutt / Neomutt macro integration
Relationships & Interactions
1. User invokes urlscan (or shell macro via mutt).
2. Input Layer feeds raw message/text to URL Extraction Engine.
3. Extraction Engine parses text, applies regex, extracts URLs + context.
4. Configuration Manager loads user settings → passed to UI Layer.
5. UI Layer displays results and captures keystrokes.
6. On selection, UI signals Action Handler → external browser/command/clipboard.
7. Exit or repeat until quit.
Architectural Patterns & Design Principles
• Modular separation: extraction logic vs. UI vs. action handlers
• Pluggable command runner (strategy pattern for browser/xdg-open/custom)
• Configurable via JSON (external configuration)
• Single responsibility modules: urlscan.py (engine), urlchoose.py (UI)
• Assets folder for static data (TLD list)
Detailed Elements to Capture
• Entry point script & __main__.py
• assets/tlds-alpha-by-domain.txt (validation resource)
• Config file location and format
• Keybinding cycle (p), context toggle (c), shorten toggle (s)