How wangkai-tech23/tflag works
Project type and purpose
• This is a research-oriented machine-learning pipeline for APT (Advanced Persistent Threat) detection on temporal provenance graphs. It implements the “TFLAG” approach, with data preprocessing, model training (GDN/TGAT models), anomaly detection, time‐window generation, and evaluation.
File structure
• Top-level scripts:
• train.py – orchestrates model training
• test.py – runs anomaly detection on held-out data
• anomly_time_windows.py – converts raw anomaly outputs into fixed time windows
• eval.py – computes metrics and summarizes detection performance
• option.py – defines CLI/configuration options
• dataset/ directory:
• build.py – parses raw provenance logs, builds node_map.json, and serializes graph snapshots
• node_map.json – mapping of system entities to graph nodes
• model/ directory:
• gdn.py – implements the Graph Deviation Network
• tgat.py – implements the Temporal Graph Attention Network
• modules/:
• embedding_module.py – node embedding layers
• temporal_attention.py – attention mechanism over time
• time_encoding.py – encodes temporal features
• utils.py – shared helper functions
• util/:
• datasets.py – PyTorch Dataset classes and DataLoader wrappers
• utils.py – evaluation helper functions
• train_log/: stores training logs and metrics
README for additional insights
• Dependencies: PyTorch, PyTorch-Geometric, torch_scatter, torch_sparse, scikit-learn, numpy
• Pipeline order:
1) python train.py
2) python test.py
3) python anomly_time_windows.py
4) python eval.py
• Calls out creation of time_windows/test_data under dataset/
a. Main components
• Data Ingestion & Preparation
• Raw provenance logs → dataset/build.py → serialized graph snapshots + node_map.json
• Data Loader Layer
• util/datasets.py + modules/time_encoding.py → PyTorch DataLoader streams batches of temporal graph data
• Model Layer
• model/tgat.py & model/gdn.py orchestrate forward passes
• modules/embedding_module.py & temporal_attention.py inject embeddings & compute temporal attention
• Training Pipeline
• train.py wires DataLoader → Model → Loss computation → Optimizer → Checkpointing (train_log/)
• Inference Pipeline
• test.py loads checkpoints → runs model on new data → raw anomaly scores
• Post-Processing
• anomly_time_windows.py transforms raw scores into fixed time windows for event grouping
• Evaluation
• eval.py computes precision/recall/F1 using scikit-learn metrics
b. Relationships & Interactions
• train.py outputs Checkpoints → test.py inputs Checkpoints
• test.py outputs Raw Anomaly Scores → anomly_time_windows.py → Windowed Anomalies → eval.py → Final Metrics
c. Architectural patterns & design principles
• Pipeline Stage Separation: clear modules for preprocessing, model, inference, evaluation
• Modular code: data-handling (util/), core model (model/), neural-network building blocks (modules/)
• Use of abstract Dataset/DataLoader pattern for scalability
d. Relevant technologies/frameworks
• Python 3.9, PyTorch 1.10+cu111, PyTorch-Geometric 2.0.4, torch_scatter, torch_sparse, scikit-learn, numpy
Diagrammatic elements
• Components as labeled blocks (e.g., “Data Preprocessing”, “PyTorch DataLoader”, “TGAT Model”, “GDN Model”, “Checkpoint Store”, “Evaluation Module”)
• Blue for data/storage components
• Green for compute/model components
• Orange for orchestration scripts
• Annotations for key libraries/versions near each block (e.g., “PyTorch-Geometric 2.0.4” adjacent to graph neural network blocks)