# josh1643/totalrecon architecture

> Architecture diagram of the GitHub repository josh1643/totalrecon, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/josh1643/totalrecon
- Repository: https://github.com/josh1643/totalrecon

## Overview

##### Project type and purpose
- This is an open-source Python library with an optional CLI.
- Purpose: passive reconnaissance—scanning text or PDF inputs for subdomains, emails, S3 buckets, then using a locally hosted AI model to summarize findings.

##### File structure
- Top level
- cli.py – CLI entrypoint, parses arguments, invokes library APIs, prints results
- setup.py / pyproject.toml – packaging and dependency declarations
- totalrecon/ package
- extract.py – core extraction logic (regex‐based domain, email, bucket discovery)
- cert.py – specialized certificate parsing or validation functions
- ai.py – wraps the Hugging Face “flan-t5-small” summarization pipeline
- No separate “frontend/backend” folders—monolithic library exposing both CLI and API.

##### README for additional insights
- Describes two primary workflows: extraction and AI summarization
- External dependency: Hugging Face Hub for model download (cached locally on first use)
- Intended users: red teamers, bug bounty hunters, CTF players—focus on offline, lightweight operation

##### Components
a. **User Interface Layer**
- CLI (cli.py) and Python library import interface
b. **Extraction Engine**
- extract.py (text/PDF ingestion module + regex scanners)
- cert.py (certificate‐specific parsing)
c. **AI Summarization Engine**
- ai.py (initializes and invokes the Transformers pipeline)
- Local Model Cache (on-disk storage managed by Hugging Face SDK)
- Hugging Face Hub (remote model repository)
d. **Data Flow and Artifacts**
- Input: plaintext files, PDF files, raw strings
- Output: JSON structure with “domains”, “emails”, “s3_buckets”, “recon_summaries”

##### Relationships and data flow
- User invokes CLI or library call →
Extraction Engine parses input →
Returns raw findings →
AI Engine loads (or reuses cached) model →
Generates human‐readable summary →
CLI/API surfaces combined JSON to user

##### Architectural patterns and design principles
- Modular monolith: clear separation between I/O (CLI), domain logic (extract/cert), and AI services
- Lazy loading / caching: model is downloaded once, reused offline thereafter
- Single responsibility: each module focuses on one aspect of the workflow

##### Detail
- Important functions (e.g., “extract_from_text” in extract.py, “summarize” in ai.py)
- Config files (pyproject.toml) as part of deployment/package management
- Error-handling or validation flow if desired (e.g., PDF parse failures)

## Diagram (Mermaid)

```mermaid
graph TB
    %% UI Layer
    subgraph "User Interface Layer"
        CLI["CLI Entrypoint\n(cli.py)"]:::ui
        API["Library Interface\n(totalrecon/__init__.py)"]:::ui
    end

    %% Core Library
    subgraph "Core Library"
        EXTRACT["Extraction Engine\nextract_from_text()\n(extract.py)"]:::core
        CERT["Certificate Parsing\n(cert.py)"]:::core
        AI["AI Summarization\nsummarize()\n(ai.py)"]:::core
    end

    %% External & Storage
    subgraph "External Dependencies & Storage"
        CACHE[(Local Model Cache)]:::storage
        HUB["Hugging Face Hub"]:::external
        INPUT["Input Files\n(text/PDF/raw)"]:::external
        OUTPUT["Output JSON\n(domains, emails,\n s3_buckets,\n recon_summaries)"]:::external
    end

    %% Meta / Packaging
    subgraph "Project Metadata"
        SETUP["Setup Script\n(setup.py)"]:::meta
        PYPROJECT["Build Config\n(pyproject.toml)"]:::meta
        README["Documentation\n(README.md)"]:::meta
        LICENSE["License\n(LICENSE)"]:::meta
        GITIGNORE["Git Config\n(.gitignore,\n .gitattributes)"]:::meta
    end

    %% Data Flow
    INPUT -->|provides data| CLI
    INPUT -->|provides data| API
    CLI -->|invokes| EXTRACT
    API -->|invokes| EXTRACT
    EXTRACT -->|parses certs| CERT
    EXTRACT -->|returns raw findings| AI
    CERT -->|returns findings| AI
    AI -->|loads model| CACHE
    CACHE -->|fetch on miss| HUB
    AI -->|generates summary| OUTPUT
    EXTRACT -->|returns raw findings| OUTPUT
    CLI -->|displays| OUTPUT
    API -->|returns| OUTPUT

    %% Click Events
    click CLI "https://github.com/josh1643/totalrecon/blob/main/cli.py"
    click API "https://github.com/josh1643/totalrecon/blob/main/totalrecon/__init__.py"
    click EXTRACT "https://github.com/josh1643/totalrecon/blob/main/totalrecon/extract.py"
    click CERT "https://github.com/josh1643/totalrecon/blob/main/totalrecon/cert.py"
    click AI "https://github.com/josh1643/totalrecon/blob/main/totalrecon/ai.py"
    click SETUP "https://github.com/josh1643/totalrecon/blob/main/setup.py"
    click PYPROJECT "https://github.com/josh1643/totalrecon/blob/main/pyproject.toml"
    click README "https://github.com/josh1643/totalrecon/blob/main/README.md"
    click LICENSE "https://github.com/josh1643/totalrecon/tree/main/LICENSE"
    click GITIGNORE "https://github.com/josh1643/totalrecon/blob/main/.gitignore"

    %% Styles
    classDef ui fill:#E0F7FA,stroke:#00796B,stroke-width:1px
    classDef core fill:#E8F5E9,stroke:#388E3C,stroke-width:1px
    classDef storage fill:#ECEFF1,stroke:#546E7A,stroke-width:1px
    classDef external fill:#ECEFF1,stroke:#546E7A,stroke-width:1px,dasharray: 5 5
    classDef meta fill:#FFF3E0,stroke:#FFA000,stroke-width:1px,dasharray: 2 2
```
