How rayan70/reconnaissance-web-passive-recon-tool- works
Project type and purpose
• This is a CLI-based open-source Python tool for passive web reconnaissance.
• Purpose: ingest a list of subdomains, scan each via HTTP(S), parse HTML/JS/robots/sitemap/headers, and emit a structured JSON of discovered endpoints.
File structure
• Top-level directories:
• data/ (subdomains.txt) – input data
• utils/ – modular parsers and HTTP helpers
• output/ – results.json
• Top-level files:
• recon.py – the orchestrator script
• requirements.txt – dependency manifest
• No separate “frontend” or “backend” – it’s a single-process pipeline with clear separation of concerns via utils.
README insights
• Each parser (HTML, JS, robots, sitemap, HTTP headers) is implemented in its own module.
• recon.py coordinates the steps in sequence per subdomain.
• Dependencies: requests, BeautifulSoup4, lxml.
a. Main components
• Input component: data/subdomains.txt
• Orchestrator: recon.py
• Parser modules (in utils/):
• HTTP Utils (http_utils.py)
• HTML Parser (html_parser.py)
• JS Analyzer (js_analyzer.py)
• Robots Parser (robots_parser.py)
• Sitemap Parser (sitemap_parser.py)
• Data aggregator (in-memory collector inside recon.py)
• Output component: output/results.json
• External service: target web servers (HTTP(S) endpoints)
b. Relationships and interactions
1. recon.py reads subdomains.txt
2. For each domain, it invokes HTTP Utils to fetch headers & raw content
3. Passes raw HTML to HTML Parser → extracts links, forms, inline scripts
4. Identifies & downloads .js files → hands to JS Analyzer → extracts API endpoints
5. Fetches /robots.txt → Robots Parser → extracts allow/disallow paths
6. Fetches /sitemap.xml → Sitemap Parser → extracts declared URLs
7. Aggregates all findings in a result object
8. Writes JSON to results.json
9. Interactions with external web servers are one‐way (requests out, responses back)
c. Architectural patterns / design principles
• Modular pipeline: each parser is a plug-in–style module
• Separation of concerns: networking vs. parsing vs. aggregation
• Single process, sequential flow – no multi-threading or microservices
d. Key technologies and libraries
• Python 3.x
• requests (HTTP client)
• BeautifulSoup4, lxml (HTML & XML parsing)
• Standard JSON library for output
Detailed breakdown to capture essential elements
• Recon.py’s central role as the “control hub.”
• Context: user runs CLI → tool → external web servers
• Container: recon.py application
• Components: utils modules, input, output