How pingc0y/urlfinder works
Project Type and Purpose
• URLFinder is a standalone, CLI-based Go application for web page crawling, JS/URL extraction, and 404-based fuzzing.
• It runs as a single binary but comprises multiple logical modules.
File Structure
• Top-level:
• cmd/cmd.go → CLI parsing & entrypoint
• main.go → CLI → command dispatch
• config/config.go → YAML or flag configuration loader
• crawler/… → Core crawling and extraction logic
• mode/mode.go → Defines crawl modes (normal, deep, safe)
• result/result.go → Report aggregation & export (CSV/JSON/HTML)
• util/utils.go → Shared utilities (HTTP client wrapper, logging)
• Build & Release:
• .github/workflows, goreleaser.yaml → CI/CD and multi-platform builds
• Static assets:
• img/ & LICENSE
README Insights
• Workflow image (process.png) suggests a linear pipeline: input → fetch → parse/fuzz → filter → output.
• Multiple flags influence behavior: concurrency (–t), depth/–mode, fuzz (–z), status filtering (–s), proxy (–x), output format (–o).
Main Components & Relationships
a. Components:
1. User/CLI
2. Configuration Loader
3. Controller (core orchestrator in main.go + mode)
4. Crawler Manager (crawler/run.go)
5. Fetcher (HTTP client in util)
6. Parser/Extractor (crawler/find.go & jsFuzz.go & urlFuzz.go)
7. State Manager (crawler/state.go)
8. Filter (crawler/filter.go)
9. Output Aggregator (result/result.go)
10. Exporters (HTML/CSV/JSON writer)
11. External Services (DNS, Target Web Servers, Proxy if set)
b. Interactions & Data Flow:
• CLI → Config Loader → Controller
• Controller → Worker Pool (spawned by Crawler Manager)
• Each Worker: Fetcher → Parser → State Manager → enqueues new URLs (per mode rules)
• After crawl completion → Filter → Aggregator → Exporters → Filesystem
c. Architectural Patterns & Principles:
• Modular separation: CLI vs. config vs. crawler vs. result vs. util
• Worker-pool concurrency for crawling threads
• Pipeline pattern: fetch → parse → fuzz → filter → output
• Configuration-driven behavior injection (mode, filters, headers)
d. Key Technologies & Libraries:
• Go standard library: net/http, regexp, sync.WaitGroup
• YAML for config
• goroutines & channels for concurrency
• html/template (for HTML report)