# curiouslearnerdev/trafficeye architecture

> Architecture diagram of the GitHub repository curiouslearnerdev/trafficeye, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-23). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/curiouslearnerdev/trafficeye
- Repository: https://github.com/curiouslearnerdev/trafficeye

## Overview

##### Project type and purpose
- This is an open‑source, modular Python network‑traffic analysis and security‑detection tool (branded “TrafficEye”/“Godzilla”).
- It parses PCAP and log files, applies signature‑ and AI‑driven inspection (e.g., SQLi, XSS, Webshell), extracts binaries, replays sessions, and produces statistics and dashboards.
- Key use cases: blue‑team traffic forensics, pentest replay, automated threat detection, AI‑assisted anomaly analysis.

##### File structure
- Top‑level entrypoints:
- main.py – loads config, dispatches modules
- Godzilla.py – WebShell/malicious‑traffic detection
- binary_extraction.py, core_processing.py, replay_request.py, etc.
- Folders:
- lib/ – helper libraries (cmdline, xdbSearcher, benchmarks, IP DB)
- log_parsing/ – log‑format identification and parsing
- modsec/ – OWASP CRS engine interface + rule files + rule parsers
- custom_extension/ – plug‑in point for bespoke data processing
- history/ – persisted JSON records of past analyses
- Config and data:
- config.yaml – global parameters, regex patterns, rule toggles
- lib/ip2region.xdb – offline IP geolocation DB
- history/trafficeye_data.json – saved analysis state

##### README for additional insights
- Lists core modules and feature set (tshark vs. pyshark, decrypt TLS, AI analysis, replay, stats)
- Notes iterative development of AI module, binary extraction, rule filtering, session utilities

a. Main components
1. Input Layer
– PCAP/PCAPNG ingestion via tshark wrapper (high‑speed parser)
– Log file ingestion (Apache, Nginx, F5, etc.)
– SSL key file for TLS decryption
2. Orchestration & CLI/GUI
– cmdline interface (lib/cmdline.py)
– potential GUI/dashboard layer (uses ico assets)
3. Core Processing Engine (core_processing.py)
– HTTP request/response parsing
– Field extraction, normalization
4. Module Pipeline
– Binary Extraction Module (binary_extraction.py)
– Security Detection Module (Godzilla.py + modsec/)
– AI Analysis Module (ai_analysis_core.py)
– Log Parsing Module (log_parsing/log_identification.py)
– Rule Filtering Module (rule_filtering.py)
– Output Filtering Module (output_filtering.py)
– Replay Module (replay_request.py + session_utils.py)
– URL Statistics Module (url_statistics.py)
5. Persistence & Lookup
– history JSON datastore
– IP geolocation via ip2region.xdb + xdbSearcher
6. Output & Visualization
– CLI reports, JSON, CSV
– Dashboard charts (uses ico images, history data)
b. Relationships & data flows
– Branching: core outputs go in parallel to binary extractor, security detector, AI analyzer, logger
– Module outputs flow into Output Filtering, then into Persistence and Visualization
– Replay Module taps core/formatted sessions via session_utils, sends HTTP back out
– IP lookup service is invoked by statistics/A geo‑tagging subcomponent
c. Architectural patterns & principles
– Plugin pipeline: modules conform to a shared interface, dynamically loaded
– Single Responsibility: each module (parsing, detection, analysis, replay) is isolated
– Configuration‑driven: YAML‑based toggles control modules and regex sets
– Extensibility points (custom_extension/) for user‑defined processors
– Offline lookup service (ip2region) and persistence decoupled from real‑time pipeline
d. Technologies, frameworks, libraries
– Python 3.x as runtime
– tshark/pyshark for PCAP parsing
– ai_analysis_core (likely using external ML/AI APIs or local models)
– OWASP ModSecurity Core Rule Set scripts
– IP2Region XDB for geo lookup
– YAML for config, JSON for history

##### Final advice
- Be as granular as useful: show each major Python script or folder as its own block
- Dynamic flows (parallel branch to AI and modsec) and feedback loops (replay back to network)

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Input Layer"
        PCAP["PCAP/PCAPNG + TLS Decrypt\n(core_processing.py)"]:::ingest
        Logs["Log Files\n(log_parsing/log_identification.py)"]:::ingest
    end
    click PCAP "https://github.com/curiouslearnerdev/trafficeye/blob/master/core_processing.py"
    click Logs "https://github.com/curiouslearnerdev/trafficeye/blob/master/log_parsing/log_identification.py"

    subgraph "Orchestration & CLI/GUI"
        Main["Main Dispatcher\n(main.py)"]:::orchestration
        Config["Configuration\n(config.yaml)"]:::config
        CLI["CLI Interface\n(lib/cmdline.py)"]:::orchestration
        GUI["Dashboard/UI Assets\n(ico/)"]:::orchestration
    end
    click Main "https://github.com/curiouslearnerdev/trafficeye/blob/master/main.py"
    click Config "https://github.com/curiouslearnerdev/trafficeye/blob/master/config.yaml"
    click CLI "https://github.com/curiouslearnerdev/trafficeye/blob/master/lib/cmdline.py"
    click GUI "https://github.com/curiouslearnerdev/trafficeye/tree/master/ico/"

    Main --> PCAP
    Main --> Logs
    Main --> Config
    Main --> CLI

    PCAP -->|"HTTP frames"| Core
    Logs -->|"Parsed logs"| Core

    subgraph "Core Processing Engine"
        Core["Core Engine\n(core_processing.py)"]:::process
    end

    subgraph "Module Pipeline"
        BE["Binary Extraction\n(binary_extraction.py)"]:::process
        SD["Security Detection\n(Godzilla.py + modsec/)"]:::process
        AI["AI Analysis\n(ai_analysis_core.py)"]:::process
        LP["Log Parser\n(log_parsing/log_identification.py)"]:::process
        RF["Rule Filtering\n(rule_filtering.py)"]:::process
        OF["Output Filtering\n(output_filtering.py)"]:::process
        RP["Replay Module\n(replay_request.py)"]:::process
        SU["Session Utils\n(session_utils.py)"]:::process
        US["URL Statistics\n(url_statistics.py)"]:::process
    end
    click BE "https://github.com/curiouslearnerdev/trafficeye/blob/master/binary_extraction.py"
    click SD "https://github.com/curiouslearnerdev/trafficeye/blob/master/Godzilla.py"
    click SD "https://github.com/curiouslearnerdev/trafficeye/blob/master/modsec/modsec_crs.py"
    click SD "https://github.com/curiouslearnerdev/trafficeye/tree/master/modsec/rules/"
    click AI "https://github.com/curiouslearnerdev/trafficeye/blob/master/ai_analysis_core.py"
    click LP "https://github.com/curiouslearnerdev/trafficeye/blob/master/log_parsing/log_identification.py"
    click RF "https://github.com/curiouslearnerdev/trafficeye/blob/master/rule_filtering.py"
    click OF "https://github.com/curiouslearnerdev/trafficeye/blob/master/output_filtering.py"
    click RP "https://github.com/curiouslearnerdev/trafficeye/blob/master/replay_request.py"
    click SU "https://github.com/curiouslearnerdev/trafficeye/blob/master/session_utils.py"
    click US "https://github.com/curiouslearnerdev/trafficeye/blob/master/url_statistics.py"

    Core --> BE
    Core --> SD
    Core --> AI
    Core --> LP

    BE --> RF
    SD --> RF
    AI --> RF
    LP --> RF

    RF --> OF
    OF --> Persist
    OF --> CLIOut

    RP --> Core

    subgraph "Persistence & Lookup"
        Persist["History Store\n(history/trafficeye_data.json)"]:::data
        IPDB["IP Geolocation DB\n(lib/ip2region.xdb)"]:::data
        XDB["XDB Search Utility\n(lib/xdbSearcher.py)"]:::data
    end
    click Persist "https://github.com/curiouslearnerdev/trafficeye/blob/master/history/trafficeye_data.json"
    click IPDB "https://github.com/curiouslearnerdev/trafficeye/blob/master/lib/ip2region.xdb"
    click XDB "https://github.com/curiouslearnerdev/trafficeye/blob/master/lib/xdbSearcher.py"

    AI --> XDB
    SD --> IPDB

    subgraph "Output & Visualization"
        CLIOut["CLI Reports\n(Console/CSV/JSON)"]:::output
        Dashboard["Dashboard Charts"]:::output
    end

    Persist --> Dashboard
    OF --> CLIOut

    classDef ingest fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef orchestration fill:#E0FFFF,stroke:#333,stroke-width:1px
    classDef config fill:#F0E68C,stroke:#333,stroke-width:1px
    classDef process fill:#98FB98,stroke:#333,stroke-width:1px
    classDef data fill:#FFA500,stroke:#333,stroke-width:1px
    classDef output fill:#DDA0DD,stroke:#333,stroke-width:1px
```
