How fdudsde/magic works
Project Type and Purpose
• MAGIC is a research‐oriented, full ML pipeline for detecting advanced persistent threats (APTs) via masked graph representation learning on system provenance graphs.
• Core purpose: preprocess raw system logs into graph data, train a self-supervised graph encoder, and detect anomalies via outlier detection.
File Structure Analysis
• Top‐level scripts:
• train.py – training the graph representation module from scratch
• eval.py – running quick/standard evaluation using saved checkpoints
• Data directories:
• data/… – compressed .zip graph archives per dataset (StreamSpot, Wget, Trace, Theia, Cadets)
• eval_result/ – KNN distance results used for batched log–level detection
• checkpoints/ – saved model parameters (autoencoder, GAT, etc.)
• Code modules:
• model/ – implementations of graph encoder (autoencoder.py, gat.py), MLP detector, loss function, training & eval wrappers
• utils/ – config loader, dataset loaders, parsers for each raw dataset, pooling utilities
README Insights
• Explains pipeline stages: raw log download → parser scripts → JSON graph files → graphs.pkl → train → eval
• Details on datasets, dependencies (Python, PyTorch, DGL, scikit-learn), and run modes (quick, standard, scratch)
• Includes a high-level architectural figure (figs/model.png) showing masking + GNN + outlier detection
Main Components
a. Data Ingestion & Preprocessing
• Raw Logs (StreamSpot .tsv, Wget .log, DARPA JSON)
• Parser Modules (utils/streamspot_parser.py, wget_parser.py, trace_parser.py)
• Graph Serializer → JSON graph files → graphs.pkl (via utils/loaddata.py)
b. Model Training
• Masked Graph Representation Module (model/autoencoder.py + model/gat.py)
• Training Script (train.py → model/train.py)
• Checkpoint Storage (checkpoints/*.pt)
c. Outlier Detection & Evaluation
• Detector Module (model/mlp.py + model/loss_func.py)
• Evaluation Script (eval.py → model/eval.py)
• Distance Computation & KNN Storage (eval_result/*.pkl)
d. Configuration & Utilities
• Global Config (utils/config.py)
• Common Helpers (utils/utils.py, utils/poolers.py)
Component Interactions & Data Flow
• Raw Logs → Parser → Graph JSON files → Batch Loader → graphs.pkl
• graphs.pkl & config → train.py → model training → checkpoint .pt
• checkpoint & graphs.pkl → eval.py → outlier scores → eval_result
• Optionally, retrain if concept drift detected → update checkpoints
Architectural Patterns & Principles
• Pipeline Pattern: clearly separated ETL → Training → Evaluation stages
• Modular Design: parser, data loader, model, and evaluation are decoupled
• Self-Supervision + Outlier Detection: two-phase learning (masked graph encoder + MLP/KNN)
Technologies & Frameworks
• Python 3.8, PyTorch 1.12.1, DGL 1.0.0 for GNN
• Scikit-learn 1.2.2 for KNN/outlier routines
• Standard packaging (requirements.txt), config via Python module
Detail to Capture
• For each parser, annotate the input format and output (JSON graph files)
• For the training step, indicate hyperparameters come from utils/config.py
• For evaluation, show two modes (quick vs. standard) and KNN distance computation