# rayan70/reconnaissance-web-passive-recon-tool- architecture

> Architecture diagram of the GitHub repository rayan70/reconnaissance-web-passive-recon-tool-, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-12). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rayan70/reconnaissance-web-passive-recon-tool-
- Repository: https://github.com/rayan70/reconnaissance-web-passive-recon-tool-

## Overview

##### Project type and purpose
- This is a **CLI-based open-source Python tool** for **passive web reconnaissance**.
- Purpose: ingest a list of subdomains, scan each via HTTP(S), parse HTML/JS/robots/sitemap/headers, and emit a structured JSON of discovered endpoints.

##### File structure
- Top-level directories:
- data/ (subdomains.txt) – input data
- utils/ – modular parsers and HTTP helpers
- output/ – results.json
- Top-level files:
- recon.py – the orchestrator script
- requirements.txt – dependency manifest
- No separate “frontend” or “backend” – it’s a single-process pipeline with clear separation of concerns via utils.

##### README insights
- Each parser (HTML, JS, robots, sitemap, HTTP headers) is implemented in its own module.
- recon.py coordinates the steps in sequence per subdomain.
- Dependencies: requests, BeautifulSoup4, lxml.

a. Main components
- **Input component**: data/subdomains.txt
- **Orchestrator**: recon.py
- **Parser modules** (in utils/):
- HTTP Utils (http_utils.py)
- HTML Parser (html_parser.py)
- JS Analyzer (js_analyzer.py)
- Robots Parser (robots_parser.py)
- Sitemap Parser (sitemap_parser.py)
- **Data aggregator** (in-memory collector inside recon.py)
- **Output component**: output/results.json
- **External service**: target web servers (HTTP(S) endpoints)
b. Relationships and interactions
1. recon.py reads subdomains.txt
2. For each domain, it invokes HTTP Utils to fetch headers & raw content
3. Passes raw HTML to HTML Parser → extracts links, forms, inline scripts
4. Identifies & downloads .js files → hands to JS Analyzer → extracts API endpoints
5. Fetches /robots.txt → Robots Parser → extracts allow/disallow paths
6. Fetches /sitemap.xml → Sitemap Parser → extracts declared URLs
7. Aggregates all findings in a result object
8. Writes JSON to results.json
9. Interactions with external web servers are one‐way (requests out, responses back)
c. Architectural patterns / design principles
- **Modular pipeline**: each parser is a plug-in–style module
- **Separation of concerns**: networking vs. parsing vs. aggregation
- **Single process, sequential flow** – no multi-threading or microservices
d. Key technologies and libraries
- Python 3.x
- requests (HTTP client)
- BeautifulSoup4, lxml (HTML & XML parsing)
- Standard JSON library for output

##### Detailed breakdown to capture essential elements
- Recon.py’s central role as the “control hub.”
- Context: user runs CLI → tool → external web servers
- Container: recon.py application
- Components: utils modules, input, output

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Stores
    Subdomains["subdomains.txt"]:::datastore
    Results["results.json"]:::datastore

    %% Orchestrator
    Recon["recon.py"]:::orchestrator

    %% Utils Modules
    HTTPUtils["http_utils.py"]:::module
    HTMLParser["html_parser.py"]:::module
    JSAnalyzer["js_analyzer.py"]:::module
    RobotsParser["robots_parser.py"]:::module
    SitemapParser["sitemap_parser.py"]:::module

    %% External Service
    External["External Web Servers"]:::external

    %% Flow
    Subdomains -->|"read domains"| Recon
    Recon -->|"HTTP GET headers & content"| HTTPUtils
    HTTPUtils -->|"response data"| Recon

    Recon -->|"pass HTML"| HTMLParser
    HTMLParser -->|"links, forms, inline scripts"| Recon

    Recon -->|"download .js files"| JSAnalyzer
    JSAnalyzer -->|"API endpoints"| Recon

    Recon -->|"fetch /robots.txt"| RobotsParser
    RobotsParser -->|"allow/disallow paths"| Recon

    Recon -->|"fetch /sitemap.xml"| SitemapParser
    SitemapParser -->|"declared URLs"| Recon

    Recon -->|"aggregate findings"| Recon
    Recon -->|"write JSON"| Results

    Recon -->|"requests"| External
    External -->|"responses"| Recon

    %% Click Events
    click Subdomains "https://github.com/rayan70/reconnaissance-web-passive-recon-tool-/blob/main/RecoWeb/data/subdomains.txt"
    click Recon "https://github.com/rayan70/reconnaissance-web-passive-recon-tool-/blob/main/RecoWeb/recon.py"
    click HTTPUtils "https://github.com/rayan70/reconnaissance-web-passive-recon-tool-/blob/main/RecoWeb/utils/http_utils.py"
    click HTMLParser "https://github.com/rayan70/reconnaissance-web-passive-recon-tool-/blob/main/RecoWeb/utils/html_parser.py"
    click JSAnalyzer "https://github.com/rayan70/reconnaissance-web-passive-recon-tool-/blob/main/RecoWeb/utils/js_analyzer.py"
    click RobotsParser "https://github.com/rayan70/reconnaissance-web-passive-recon-tool-/blob/main/RecoWeb/utils/robots_parser.py"
    click SitemapParser "https://github.com/rayan70/reconnaissance-web-passive-recon-tool-/blob/main/RecoWeb/utils/sitemap_parser.py"
    click Results "https://github.com/rayan70/reconnaissance-web-passive-recon-tool-/blob/main/RecoWeb/output/results.json"
    click Requirements "https://github.com/rayan70/reconnaissance-web-passive-recon-tool-/blob/main/RecoWeb/requirements.txt"

    %% Legend (hidden for cleanliness)

    %% Classes
    classDef datastore fill:#e0ffe0,stroke:#006600,stroke-width:1px;
    classDef orchestrator fill:#cce5ff,stroke:#004085,stroke-width:1px;
    classDef module fill:#cce5ff,stroke:#004085,stroke-width:1px;
    classDef external fill:#ffe5b4,stroke:#cc8400,stroke-width:1px;
```
