# pingc0y/urlfinder architecture

> Architecture diagram of the GitHub repository pingc0y/urlfinder, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pingc0y/urlfinder
- Repository: https://github.com/pingc0y/urlfinder

## Overview

##### Project Type and Purpose
- URLFinder is a standalone, CLI-based Go application for web page crawling, JS/URL extraction, and 404-based fuzzing.
- It runs as a single binary but comprises multiple logical modules.

##### File Structure
- Top-level:
- cmd/cmd.go            → CLI parsing & entrypoint
- main.go               → CLI → command dispatch
- config/config.go      → YAML or flag configuration loader
- crawler/…             → Core crawling and extraction logic
- mode/mode.go          → Defines crawl modes (normal, deep, safe)
- result/result.go      → Report aggregation & export (CSV/JSON/HTML)
- util/utils.go         → Shared utilities (HTTP client wrapper, logging)
- Build & Release:
- .github/workflows, goreleaser.yaml → CI/CD and multi-platform builds
- Static assets:
- img/ & LICENSE

##### README Insights
- Workflow image (process.png) suggests a linear pipeline: input → fetch → parse/fuzz → filter → output.
- Multiple flags influence behavior: concurrency (–t), depth/–mode, fuzz (–z), status filtering (–s), proxy (–x), output format (–o).

##### Main Components & Relationships
a. Components:
1. User/CLI
2. Configuration Loader
3. Controller (core orchestrator in main.go + mode)
4. Crawler Manager (crawler/run.go)
5. Fetcher (HTTP client in util)
6. Parser/Extractor (crawler/find.go & jsFuzz.go & urlFuzz.go)
7. State Manager (crawler/state.go)
8. Filter (crawler/filter.go)
9. Output Aggregator (result/result.go)
10. Exporters (HTML/CSV/JSON writer)
11. External Services (DNS, Target Web Servers, Proxy if set)
b. Interactions & Data Flow:
- CLI → Config Loader → Controller
- Controller → Worker Pool (spawned by Crawler Manager)
- Each Worker: Fetcher → Parser → State Manager → enqueues new URLs (per mode rules)
- After crawl completion → Filter → Aggregator → Exporters → Filesystem
c. Architectural Patterns & Principles:
- Modular separation: CLI vs. config vs. crawler vs. result vs. util
- Worker-pool concurrency for crawling threads
- Pipeline pattern: fetch → parse → fuzz → filter → output
- Configuration-driven behavior injection (mode, filters, headers)
d. Key Technologies & Libraries:
- Go standard library: net/http, regexp, sync.WaitGroup
- YAML for config
- goroutines & channels for concurrency
- html/template (for HTML report)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Nodes
    CMD["User/CLI"]:::cli
    Config["Configuration Loader"]:::config
    Controller["Controller"]:::core
    subgraph "Crawler Manager"
        direction TB
        WorkerPool["Worker Pool"]:::core
        subgraph "Worker"
            direction TB
            Fetcher["Fetcher"]:::util
            Parser["Parser/Extractor"]:::core
            StateMgr["State Manager"]:::core
        end
    end
    Filter["Filter"]:::core
    Aggregator["Output Aggregator"]:::core
    Exporters["Exporters"]:::core
    Filesystem["Filesystem (output/)"]:::storage
    External["External Web Servers/Proxy"]:::external

    %% Flow
    CMD -->|invokes| Config
    Config -->|provides configs| Controller
    Controller -->|spawns| WorkerPool
    WorkerPool -->|assigns| Fetcher
    Fetcher -->|fetches URLs| Parser
    Parser -->|records state| StateMgr
    StateMgr -->|enqueues URLs| WorkerPool
    WorkerPool -->|on completion| Filter
    Filter -->|filters results| Aggregator
    Aggregator -->|aggregates data| Exporters
    Exporters -->|writes files| Filesystem
    Fetcher -->|HTTP requests| External

    %% Click events
    click CMD "https://github.com/pingc0y/urlfinder/blob/master/cmd/cmd.go"
    click Config "https://github.com/pingc0y/urlfinder/blob/master/config/config.go"
    click Controller "https://github.com/pingc0y/urlfinder/blob/master/mode/mode.go"
    click WorkerPool "https://github.com/pingc0y/urlfinder/blob/master/crawler/run.go"
    click Fetcher "https://github.com/pingc0y/urlfinder/blob/master/util/utils.go"
    click Parser "https://github.com/pingc0y/urlfinder/blob/master/crawler/find.go"
    click StateMgr "https://github.com/pingc0y/urlfinder/blob/master/crawler/state.go"
    click Filter "https://github.com/pingc0y/urlfinder/blob/master/crawler/filter.go"
    click Aggregator "https://github.com/pingc0y/urlfinder/blob/master/result/result.go"
    click Exporters "https://github.com/pingc0y/urlfinder/blob/master/result/report.html"

    %% Styles
    classDef cli fill:#D5E8D4,stroke:#82B366
    classDef config fill:#FFF2CC,stroke:#D6B656
    classDef core fill:#DAE8FC,stroke:#6C8EBF
    classDef util fill:#E1D5E7,stroke:#9673A6
    classDef external fill:#F8CECC,stroke:#B85450
    classDef storage fill:#FFE6CC,stroke:#D99000
```
