# projectdiscovery/httpx architecture

> Architecture diagram of the GitHub repository projectdiscovery/httpx, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-01-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/projectdiscovery/httpx
- Repository: https://github.com/projectdiscovery/httpx

## Overview

##### Main Components Identification

The system should be divided into these primary components:

a. Command Line Interface (CLI)
- Located in cmd/httpx
- Handles user input and flag parsing
- Manages configuration and output formatting

b. Core Engine (Runner)
- Located in runner/
- Orchestrates the HTTP probing operations
- Manages concurrent execution and rate limiting

c. HTTP Processing Module
- Located in common/httpx/
- Handles HTTP/HTTPS requests
- Manages protocol detection and fallback logic

d. Utility Modules
- Located in common/
- Custom extractors (customextract/)
- Header processing (customheader/)
- File operations (fileutil/)
- Hash calculations (hashes/)

##### Component Relationships

**[CLI Layer] → [Runner Layer] → [HTTP Processing Layer]**
- ↓
- [Utility Modules]

##### Specific Elements

a. Input Processing
- Input sources (URLs, hosts, CIDR)
- File input handling
- Raw request processing

b. Core Processing Pipeline
- Request generation
- Concurrent execution
- Rate limiting
- Retry mechanism

c. Feature Modules
- TLS/SSL handling
- HTTP2 support
- Pipeline processing
- Virtual host detection
- Screenshot capability (headless browser)

d. Output Handling
- Multiple format support (JSON, CSV)
- Response storage
- Custom output formatting

##### Important Architectural Patterns to Highlight

a. Pipeline Pattern
- Flow from input → processing → output
- Concurrent processing capability

b. Plugin Architecture
- How different probes can be enabled/disabled
- Extensibility of the system

c. Configuration Management
- How different options affect the processing pipeline
- Where configuration is applied

##### Technical Details

- Rate limiting mechanisms
- Concurrent processing with goroutines
- Protocol handling (HTTP/HTTPS/HTTP2)
- Security features (TLS, custom headers)
- Integration points with external services

**┌─ Input Section ─┐    ┌─ Processing Section ─┐    ┌─ Output Section ─┐**
- File Input    │ → │ - HTTP Processing    │ → │ - JSON Output    │
- URL Input     │    │ - TLS Handling      │    │ - CSV Output     │
- CIDR Input    │    │ - Probe Execution   │    │ - File Storage   │
- ┘    └───────────────────────┘    └────────────────┘

## Diagram (Mermaid)

```mermaid
graph TB
    subgraph CLI
        CLI_Main["CLI Interface"]:::cli
        Config["Configuration Options"]:::config
    end

    subgraph Runner
        Runner_Core["Core Engine"]:::runner
        Health["Health Checker"]:::runner
        API["API Endpoint Handler"]:::runner
        Headless["Headless Browser"]:::runner
    end

    subgraph "HTTP Processing"
        HTTP_Core["HTTP Core"]:::http
        Pipeline["Pipeline Processor"]:::http
        TLS["TLS/SSL Handler"]:::http
        HTTP2["HTTP2 Support"]:::http
        VHost["Virtual Host Detection"]:::http
        Domain["Domain Processor"]:::http
        CSP["Content Security"]:::http
        CDN["CDN Detection"]:::http
        Response["Response Handler"]:::http
    end

    subgraph "Utility Modules"
        Extract["Custom Extractors"]:::util
        Headers["Header Processor"]:::util
        Files["File Operations"]:::util
        Hash["Hash Calculator"]:::util
        HTTPUtil["HTTP Utilities"]:::util
    end

    %% Relationships
    CLI_Main --> Config
    CLI_Main --> Runner_Core
    Runner_Core --> Health
    Runner_Core --> API
    Runner_Core --> Headless
    Runner_Core --> HTTP_Core
    HTTP_Core --> Pipeline
    HTTP_Core --> TLS
    HTTP_Core --> HTTP2
    HTTP_Core --> VHost
    HTTP_Core --> Domain
    HTTP_Core --> CSP
    HTTP_Core --> CDN
    HTTP_Core --> Response
    Runner_Core --> Extract
    Runner_Core --> Headers
    Runner_Core --> Files
    Runner_Core --> Hash
    Runner_Core --> HTTPUtil

    %% Click Events
    click CLI_Main "https://github.com/projectdiscovery/httpx/blob/main/cmd/httpx/httpx.go"
    click Config "https://github.com/projectdiscovery/httpx/blob/main/runner/options.go"
    click Runner_Core "https://github.com/projectdiscovery/httpx/blob/main/runner/runner.go"
    click Health "https://github.com/projectdiscovery/httpx/blob/main/runner/healthcheck.go"
    click API "https://github.com/projectdiscovery/httpx/blob/main/runner/apiendpoint.go"
    click Headless "https://github.com/projectdiscovery/httpx/blob/main/runner/headless.go"
    click HTTP_Core "https://github.com/projectdiscovery/httpx/blob/main/common/httpx/httpx.go"
    click Pipeline "https://github.com/projectdiscovery/httpx/blob/main/common/httpx/pipeline.go"
    click TLS "https://github.com/projectdiscovery/httpx/blob/main/common/httpx/tls.go"
    click HTTP2 "https://github.com/projectdiscovery/httpx/blob/main/common/httpx/http2.go"
    click VHost "https://github.com/projectdiscovery/httpx/blob/main/common/httpx/virtualhost.go"
    click Domain "https://github.com/projectdiscovery/httpx/blob/main/common/httpx/domains.go"
    click CSP "https://github.com/projectdiscovery/httpx/blob/main/common/httpx/csp.go"
    click CDN "https://github.com/projectdiscovery/httpx/blob/main/common/httpx/cdn.go"
    click Response "https://github.com/projectdiscovery/httpx/blob/main/common/httpx/response.go"
    click Extract "https://github.com/projectdiscovery/httpx/blob/main/common/customextract/customextract.go"
    click Headers "https://github.com/projectdiscovery/httpx/blob/main/common/customheader/customheader.go"
    click Files "https://github.com/projectdiscovery/httpx/blob/main/common/fileutil/fileutil.go"
    click Hash "https://github.com/projectdiscovery/httpx/blob/main/common/hashes/hashes.go"
    click HTTPUtil "https://github.com/projectdiscovery/httpx/blob/main/common/httputilz/httputilz.go"

    %% Styles
    classDef cli fill:#2196F3,stroke:#1565C0,color:white
    classDef config fill:#1976D2,stroke:#0D47A1,color:white
    classDef runner fill:#4CAF50,stroke:#2E7D32,color:white
    classDef http fill:#FFC107,stroke:#FFA000,color:black
    classDef util fill:#FF5722,stroke:#E64A19,color:white

    %% Legend
    subgraph Legend
        L1["CLI Components"]:::cli
        L2["Runner Components"]:::runner
        L3["HTTP Components"]:::http
        L4["Utility Components"]:::util
    end
```
