How threatrace-detector/threatrace works
Project Type and Purpose
• This is an evaluation‐reproduction pipeline for the THREATRACE intrusion‐detection system.
• It combines data parsing, graph processing (via the GraphChi C++ engine), model training/inference (via PyTorch Geometric and Python), and evaluation.
Top‐Level Components
A. External Data Sources
• DARPA TC, StreamSpot, UnicornSC raw archives
B. Data Ingestion & Parsing (Python scripts in scripts/parse_*.py)
• Reads raw archives, outputs edge‐based graph text (graphchi-cpp-master/graph_data/.../empty.txt)
C. Graph Preprocessing & Sharding (C++ modules in graphchi-cpp-master/src/preprocessing and src/shards)
• Splits graphs into blocks, shards for out‐of‐core processing
D. GraphChi Engine (C++ core in graphchi-cpp-master/src/engine + APIs in src/api)
• Executes user‐defined graph algorithms (feature extraction)
E. Feature & Label Storage (example_models/.../feature.txt, label.txt)
F. Model Training & Inference (Python scripts train_*.py, shell runners run_*.sh)
• Uses PyTorch Geometric to train graph neural networks on extracted features
• Persists model artifacts under models/ or example_models/
G. Testing & Evaluation (Python scripts test_*.py, evaluate_*.py)
• Runs benign/attack flows, compares predictions to ground truth in groundtruth/
H. Results & Metrics (outputs accuracy, ROC, etc.)
Relationships & Data Flow
1. External Data → Parsing Scripts → Graph Data files
2. Graph Data files → Preprocessing & Sharding → Sharded Graph Blocks
3. Sharded Graph Blocks → GraphChi Engine → Extracted Feature + Label files
4. Feature + Label files → Model Training (PyG) → Trained Model files
5. Trained Models + Graph Data → Inference Shell scripts → Prediction logs
6. Prediction logs + Groundtruth → Evaluation scripts → Performance metrics
Architectural Patterns & Design Principles
• Pipeline Architecture: clear linear stages (Ingestion → Preprocessing → Extraction → Training → Evaluation)
• Polyglot Modularization:
• Python layer for orchestration, parsing, ML training
• C++ GraphChi engine for scalable graph operations
• Separation of Concerns:
• Data Layer (graph_data, example_models, groundtruth)
• Compute Layer (GraphChi engine, PyG training)
• Orchestration Layer (Python scripts, shell scripts)