How jnanasaj/web-copilot works
Project Type and Purpose
• WebCopilot is an open-source command-line automation tool for subdomain enumeration and vulnerability scanning. It orchestrates multiple third-party tools in a pipeline to discover subdomains, filter live hosts, extract assets (titles, screenshots, endpoints), and then run vulnerability scanners on filtered parameters. Results are printed to the console and saved to a specified directory.
File Structure Analysis
• Top-level files:
• LICENSE, README.md – project metadata
• install.sh – installer script for dependencies and setup
• webcopilot – the main executable script (likely a shell or Python script orchestrating the workflow)
• No separate “src” or “lib” folder: the core logic lives in the “webcopilot” script.
• No frontend/backend split—this is a single‐purpose CLI tool.
• External dependencies are invoked at runtime by the script.
Key Components
a. User Interface
• CLI Entry Point (“webcopilot” script) – accepts target domain and output directory.
b. Orchestration Engine
• Core workflow logic that calls each stage in sequence, handles intermediate files and logging.
c. Enumeration Stage
• Passive enumeration tools: assetfinder, sublister, subfinder, amass, findomain, hackertarget, riddler, crt.sh
• Aggregator/merger – collects and deduplicates subdomain lists.
d. Active Enumeration
• gobuster with SecLists DNS wordlists
• amass in brute-force mode
• dnsx for live host filtering.
e. Asset Extraction
• httpx and aquatone for HTTP probing, title extraction, screenshots
• subjack for takeover detection.
f. Endpoint Crawling & Parameter Filtering
• waybackurls, gauplus for URL collection
• gf patterns to isolate parameters for XSS, SQLi, SSRF, open redirect, RCE.
g. Vulnerability Scanning
• Tools per vulnerability: dalfox (XSS), kxss, openredirex, nuclei, sqlmap, etc.
• Parallel execution manager (if used) or sequential invocation.
h. Output Management
• Result formatter (console output)
• Directory structure for saving raw tool outputs, consolidated reports, logs.
Relationships and Data Flow
• Orchestration Engine → Enumeration Stage (passes domain, receives file1.txt)
• Enumeration → Active Enumeration (file1.txt → file2.txt)
• Active Enumeration → Asset Extraction (live_hosts.txt → assets.json/screenshots/)
• Asset Extraction → Endpoint Crawling (assets.json → urls.txt)
• Endpoint Crawling → Parameter Filtering (urls.txt → param_lists/)
• Parameter Filtering → Vulnerability Scanning (param_lists/ → vuln_reports/)
• Vulnerability Scanning → Output Management (vuln_reports/ → final_report, console)
Architectural Patterns and Principles
• Pipeline/chain-of-responsibility: each stage takes input, processes, outputs to the next.
• Loose coupling: each tool invocation is a separate process; new tools can be added at any stage.
• Single responsibility: each stage handles one concern (enumeration vs. scanning vs. reporting).
Technologies and Extensibility
• Shell scripting (bash/zsh) or Python for orchestration
• External CLI tools: assetfinder, subfinder, amass, gobuster, dnsx, httpx, aquatone, waybackurls, gauplus, gf, dalfox, nuclei, etc.
• Output folders on filesystem; optional JSON, CSV or plain text.