How curiouslearnerdev/trafficeye works
Project type and purpose
• This is an open‑source, modular Python network‑traffic analysis and security‑detection tool (branded “TrafficEye”/“Godzilla”).
• It parses PCAP and log files, applies signature‑ and AI‑driven inspection (e.g., SQLi, XSS, Webshell), extracts binaries, replays sessions, and produces statistics and dashboards.
• Key use cases: blue‑team traffic forensics, pentest replay, automated threat detection, AI‑assisted anomaly analysis.
File structure
• Top‑level entrypoints:
• main.py – loads config, dispatches modules
• Godzilla.py – WebShell/malicious‑traffic detection
• binary_extraction.py, core_processing.py, replay_request.py, etc.
• Folders:
• lib/ – helper libraries (cmdline, xdbSearcher, benchmarks, IP DB)
• log_parsing/ – log‑format identification and parsing
• modsec/ – OWASP CRS engine interface + rule files + rule parsers
• custom_extension/ – plug‑in point for bespoke data processing
• history/ – persisted JSON records of past analyses
• Config and data:
• config.yaml – global parameters, regex patterns, rule toggles
• lib/ip2region.xdb – offline IP geolocation DB
• history/trafficeye_data.json – saved analysis state
README for additional insights
• Lists core modules and feature set (tshark vs. pyshark, decrypt TLS, AI analysis, replay, stats)
• Notes iterative development of AI module, binary extraction, rule filtering, session utilities
a. Main components
1. Input Layer
– PCAP/PCAPNG ingestion via tshark wrapper (high‑speed parser)
– Log file ingestion (Apache, Nginx, F5, etc.)
– SSL key file for TLS decryption
2. Orchestration & CLI/GUI
– cmdline interface (lib/cmdline.py)
– potential GUI/dashboard layer (uses ico assets)
3. Core Processing Engine (core_processing.py)
– HTTP request/response parsing
– Field extraction, normalization
4. Module Pipeline
– Binary Extraction Module (binary_extraction.py)
– Security Detection Module (Godzilla.py + modsec/)
– AI Analysis Module (ai_analysis_core.py)
– Log Parsing Module (log_parsing/log_identification.py)
– Rule Filtering Module (rule_filtering.py)
– Output Filtering Module (output_filtering.py)
– Replay Module (replay_request.py + session_utils.py)
– URL Statistics Module (url_statistics.py)
5. Persistence & Lookup
– history JSON datastore
– IP geolocation via ip2region.xdb + xdbSearcher
6. Output & Visualization
– CLI reports, JSON, CSV
– Dashboard charts (uses ico images, history data)
b. Relationships & data flows
– Branching: core outputs go in parallel to binary extractor, security detector, AI analyzer, logger
– Module outputs flow into Output Filtering, then into Persistence and Visualization
– Replay Module taps core/formatted sessions via session_utils, sends HTTP back out
– IP lookup service is invoked by statistics/A geo‑tagging subcomponent
c. Architectural patterns & principles
– Plugin pipeline: modules conform to a shared interface, dynamically loaded
– Single Responsibility: each module (parsing, detection, analysis, replay) is isolated
– Configuration‑driven: YAML‑based toggles control modules and regex sets
– Extensibility points (custom_extension/) for user‑defined processors
– Offline lookup service (ip2region) and persistence decoupled from real‑time pipeline
d. Technologies, frameworks, libraries
– Python 3.x as runtime
– tshark/pyshark for PCAP parsing
– ai_analysis_core (likely using external ML/AI APIs or local models)
– OWASP ModSecurity Core Rule Set scripts
– IP2Region XDB for geo lookup
– YAML for config, JSON for history
Final advice
• Be as granular as useful: show each major Python script or folder as its own block
• Dynamic flows (parallel branch to AI and modsec) and feedback loops (replay back to network)