# fdudsde/magic architecture

> Architecture diagram of the GitHub repository fdudsde/magic, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/fdudsde/magic
- Repository: https://github.com/fdudsde/magic

## Overview

##### Project Type and Purpose
- MAGIC is a research‐oriented, full ML pipeline for detecting advanced persistent threats (APTs) via masked graph representation learning on system provenance graphs.
- Core purpose: preprocess raw system logs into graph data, train a self-supervised graph encoder, and detect anomalies via outlier detection.

##### File Structure Analysis
- Top‐level scripts:
- train.py – training the graph representation module from scratch
- eval.py  – running quick/standard evaluation using saved checkpoints
- Data directories:
- data/… – compressed `.zip` graph archives per dataset (StreamSpot, Wget, Trace, Theia, Cadets)
- eval_result/ – KNN distance results used for batched log–level detection
- checkpoints/ – saved model parameters (autoencoder, GAT, etc.)
- Code modules:
- model/ – implementations of graph encoder (autoencoder.py, gat.py), MLP detector, loss function, training & eval wrappers
- utils/ – config loader, dataset loaders, parsers for each raw dataset, pooling utilities

##### README Insights
- Explains pipeline stages: raw log download → parser scripts → JSON graph files → graphs.pkl → train → eval
- Details on datasets, dependencies (Python, PyTorch, DGL, scikit-learn), and run modes (quick, standard, scratch)
- Includes a high-level architectural figure (`figs/model.png`) showing masking + GNN + outlier detection

##### Main Components
a. Data Ingestion & Preprocessing
- Raw Logs (StreamSpot `.tsv`, Wget `.log`, DARPA JSON)
- Parser Modules (utils/streamspot_parser.py, wget_parser.py, trace_parser.py)
- Graph Serializer → JSON graph files → `graphs.pkl` (via utils/loaddata.py)
b. Model Training
- Masked Graph Representation Module (model/autoencoder.py + model/gat.py)
- Training Script (train.py → model/train.py)
- Checkpoint Storage (checkpoints/*.pt)
c. Outlier Detection & Evaluation
- Detector Module (model/mlp.py + model/loss_func.py)
- Evaluation Script (eval.py → model/eval.py)
- Distance Computation & KNN Storage (eval_result/*.pkl)
d. Configuration & Utilities
- Global Config (utils/config.py)
- Common Helpers (utils/utils.py, utils/poolers.py)

##### Component Interactions & Data Flow
- Raw Logs → Parser → Graph JSON files → Batch Loader → graphs.pkl
- graphs.pkl & config → train.py → model training → checkpoint `.pt`
- checkpoint & graphs.pkl → eval.py → outlier scores → eval_result
- Optionally, retrain if concept drift detected → update checkpoints

##### Architectural Patterns & Principles
- Pipeline Pattern: clearly separated ETL → Training → Evaluation stages
- Modular Design: parser, data loader, model, and evaluation are decoupled
- Self-Supervision + Outlier Detection: two-phase learning (masked graph encoder + MLP/KNN)

##### Technologies & Frameworks
- Python 3.8, PyTorch 1.12.1, DGL 1.0.0 for GNN
- Scikit-learn 1.2.2 for KNN/outlier routines
- Standard packaging (`requirements.txt`), config via Python module

##### Detail to Capture
- For each parser, annotate the input format and output (JSON graph files)
- For the training step, indicate hyperparameters come from utils/config.py
- For evaluation, show two modes (quick vs. standard) and KNN distance computation

## Diagram (Mermaid)

```mermaid
flowchart TD
    subgraph "Data Layer"
        Raw_Cadets["cadets/graphs.zip"]:::data
        Raw_Streamspot["streamspot/graphs.zip"]:::data
        Raw_Theia["theia/graphs.zip"]:::data
        Raw_Trace["trace/graphs.zip"]:::data
        Raw_Wget["wget/graphs.zip"]:::data

        SP["StreamSpot Parser"]:::data
        WP["Wget Parser"]:::data
        TP["Trace Parser"]:::data

        UtilsU["Utility Functions\n(utils/utils.py)"]:::data
        Pool["Graph Poolers\n(utils/poolers.py)"]:::data

        GS["Graph Serializer\n(utils/loaddata.py)"]:::data
        graphs(("graphs.pkl")):::data
    end

    Raw_Streamspot --> SP
    Raw_Wget --> WP
    Raw_Trace --> TP
    Raw_Cadets --> GS
    Raw_Theia --> GS
    SP --> GS
    WP --> GS
    TP --> GS
    UtilsU --> GS
    Pool --> GS
    GS --> graphs

    subgraph "Model Layer"
        CFG["Global Config\n(utils/config.py)"]:::model
        TS["train.py"]:::model
        MW["Training Wrapper\n(model/train.py)"]:::model
        subgraph "Masked Graph Representation"
            AE["Autoencoder\n(model/autoencoder.py)"]:::model
            GAT["GAT\n(model/gat.py)"]:::model
        end
        CP_Cadets(("checkpoint-cadets.pt")):::model
        CP_Streamspot(("checkpoint-streamspot.pt")):::model
        CP_Theia(("checkpoint-theia.pt")):::model
        CP_Trace(("checkpoint-trace.pt")):::model
        CP_Wget(("checkpoint-wget.pt")):::model
    end

    graphs --> TS
    CFG --> TS
    TS --> MW
    MW --> AE
    MW --> GAT
    MW --> CP_Cadets
    MW --> CP_Streamspot
    MW --> CP_Theia
    MW --> CP_Trace
    MW --> CP_Wget

    subgraph "Evaluation Layer"
        ES["eval.py"]:::eval
        EW["Evaluation Wrapper\n(model/eval.py)"]:::eval
        MLP["MLP Detector\n(model/mlp.py)"]:::eval
        LF["Loss Func\n(model/loss_func.py)"]:::eval
        ER_Cadets(("distance_save_cadets.pkl")):::eval
        ER_Theia(("distance_save_theia.pkl")):::eval
        ER_Trace(("distance_save_trace.pkl")):::eval
    end

    graphs --> ES
    CP_Cadets --> ES
    CP_Streamspot --> ES
    CP_Theia --> ES
    CP_Trace --> ES
    CP_Wget --> ES
    ES --> EW
    EW --> MLP
    EW --> LF
    EW --> ER_Cadets
    EW --> ER_Theia
    EW --> ER_Trace

    click Raw_Cadets "https://github.com/fdudsde/magic/blob/main/data/cadets/graphs.zip"
    click Raw_Streamspot "https://github.com/fdudsde/magic/blob/main/data/streamspot/graphs.zip"
    click Raw_Theia "https://github.com/fdudsde/magic/blob/main/data/theia/graphs.zip"
    click Raw_Trace "https://github.com/fdudsde/magic/blob/main/data/trace/graphs.zip"
    click Raw_Wget "https://github.com/fdudsde/magic/blob/main/data/wget/graphs.zip"
    click SP "https://github.com/fdudsde/magic/blob/main/utils/streamspot_parser.py"
    click WP "https://github.com/fdudsde/magic/blob/main/utils/wget_parser.py"
    click TP "https://github.com/fdudsde/magic/blob/main/utils/trace_parser.py"
    click GS "https://github.com/fdudsde/magic/blob/main/utils/loaddata.py"
    click CFG "https://github.com/fdudsde/magic/blob/main/utils/config.py"
    click UtilsU "https://github.com/fdudsde/magic/blob/main/utils/utils.py"
    click Pool "https://github.com/fdudsde/magic/blob/main/utils/poolers.py"
    click TS "https://github.com/fdudsde/magic/blob/main/train.py"
    click MW "https://github.com/fdudsde/magic/blob/main/model/train.py"
    click AE "https://github.com/fdudsde/magic/blob/main/model/autoencoder.py"
    click GAT "https://github.com/fdudsde/magic/blob/main/model/gat.py"
    click CP_Cadets "https://github.com/fdudsde/magic/blob/main/checkpoints/checkpoint-cadets.pt"
    click CP_Streamspot "https://github.com/fdudsde/magic/blob/main/checkpoints/checkpoint-streamspot.pt"
    click CP_Theia "https://github.com/fdudsde/magic/blob/main/checkpoints/checkpoint-theia.pt"
    click CP_Trace "https://github.com/fdudsde/magic/blob/main/checkpoints/checkpoint-trace.pt"
    click CP_Wget "https://github.com/fdudsde/magic/blob/main/checkpoints/checkpoint-wget.pt"
    click ES "https://github.com/fdudsde/magic/blob/main/eval.py"
    click EW "https://github.com/fdudsde/magic/blob/main/model/eval.py"
    click MLP "https://github.com/fdudsde/magic/blob/main/model/mlp.py"
    click LF "https://github.com/fdudsde/magic/blob/main/model/loss_func.py"
    click ER_Cadets "https://github.com/fdudsde/magic/blob/main/eval_result/distance_save_cadets.pkl"
    click ER_Theia "https://github.com/fdudsde/magic/blob/main/eval_result/distance_save_theia.pkl"
    click ER_Trace "https://github.com/fdudsde/magic/blob/main/eval_result/distance_save_trace.pkl"

    classDef data fill:#add8e6,stroke:#000;
    classDef model fill:#90ee90,stroke:#000;
    classDef eval fill:#ffa500,stroke:#000;
```
