# wangkai-tech23/tflag architecture

> Architecture diagram of the GitHub repository wangkai-tech23/tflag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/wangkai-tech23/tflag
- Repository: https://github.com/wangkai-tech23/tflag

## Overview

##### Project type and purpose
- This is a research-oriented machine-learning pipeline for APT (Advanced Persistent Threat) detection on temporal provenance graphs. It implements the “TFLAG” approach, with data preprocessing, model training (GDN/TGAT models), anomaly detection, time‐window generation, and evaluation.

##### File structure
- Top-level scripts:
- train.py – orchestrates model training
- test.py – runs anomaly detection on held-out data
- anomly_time_windows.py – converts raw anomaly outputs into fixed time windows
- eval.py – computes metrics and summarizes detection performance
- option.py – defines CLI/configuration options
- dataset/ directory:
- build.py – parses raw provenance logs, builds node_map.json, and serializes graph snapshots
- node_map.json – mapping of system entities to graph nodes
- model/ directory:
- gdn.py – implements the Graph Deviation Network
- tgat.py – implements the Temporal Graph Attention Network
- modules/:
- embedding_module.py – node embedding layers
- temporal_attention.py – attention mechanism over time
- time_encoding.py – encodes temporal features
- utils.py – shared helper functions
- util/:
- datasets.py – PyTorch Dataset classes and DataLoader wrappers
- utils.py – evaluation helper functions
- train_log/: stores training logs and metrics

##### README for additional insights
- Dependencies: PyTorch, PyTorch-Geometric, torch_scatter, torch_sparse, scikit-learn, numpy
- Pipeline order:
1) python train.py
2) python test.py
3) python anomly_time_windows.py
4) python eval.py
- Calls out creation of time_windows/test_data under dataset/

a. Main components
- Data Ingestion & Preparation
- Raw provenance logs → dataset/build.py → serialized graph snapshots + node_map.json
- Data Loader Layer
- util/datasets.py + modules/time_encoding.py → PyTorch DataLoader streams batches of temporal graph data
- Model Layer
- model/tgat.py & model/gdn.py orchestrate forward passes
- modules/embedding_module.py & temporal_attention.py inject embeddings & compute temporal attention
- Training Pipeline
- train.py wires DataLoader → Model → Loss computation → Optimizer → Checkpointing (train_log/)
- Inference Pipeline
- test.py loads checkpoints → runs model on new data → raw anomaly scores
- Post-Processing
- anomly_time_windows.py transforms raw scores into fixed time windows for event grouping
- Evaluation
- eval.py computes precision/recall/F1 using scikit-learn metrics

b. Relationships & Interactions
- train.py outputs Checkpoints → test.py inputs Checkpoints
- test.py outputs Raw Anomaly Scores → anomly_time_windows.py → Windowed Anomalies → eval.py → Final Metrics

c. Architectural patterns & design principles
- Pipeline Stage Separation: clear modules for preprocessing, model, inference, evaluation
- Modular code: data-handling (util/), core model (model/), neural-network building blocks (modules/)
- Use of abstract Dataset/DataLoader pattern for scalability

d. Relevant technologies/frameworks
- Python 3.9, PyTorch 1.10+cu111, PyTorch-Geometric 2.0.4, torch_scatter, torch_sparse, scikit-learn, numpy

##### Diagrammatic elements
- Components as labeled blocks (e.g., “Data Preprocessing”, “PyTorch DataLoader”, “TGAT Model”, “GDN Model”, “Checkpoint Store”, “Evaluation Module”)
- Blue for data/storage components
- Green for compute/model components
- Orange for orchestration scripts
- Annotations for key libraries/versions near each block (e.g., “PyTorch-Geometric 2.0.4” adjacent to graph neural network blocks)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Layer
    subgraph "Data Layer"
        RawData["Raw Provenance Logs"]:::storage
        Build["Data Preprocessing\n(build.py)"]:::process
        NodeMap["Node Map\n(node_map.json)"]:::storage
        Serialized["Serialized Graph Snapshots"]:::storage
    end

    %% Data Loader Layer
    subgraph "Data Loader Layer"
        DataLoader["PyTorch Dataset/DataLoader"]:::process
        TimeEnc["Time Encoding Module"]:::process
    end

    %% Model Layer
    subgraph "Model Layer"
        Emb["Embeddings Module"]:::compute
        TempAtt["Temporal Attention Module"]:::compute
        GDN["Graph Deviation Network"]:::compute
        TGAT["Temporal GAT Network"]:::compute
    end

    %% Utilities
    subgraph "Utilities"
        ModUtils["Module Utils"]:::process
        GenUtils["General Utils"]:::process
    end

    %% Pipeline Orchestration
    subgraph "Pipeline Orchestration"
        Option["Configuration (option.py)"]:::orchestration
        Train["Training Script (train.py)"]:::orchestration
        TrainLog["Training Logs"]:::storage
        Test["Inference Script (test.py)"]:::orchestration
        Post["Window Generation (anomly_time_windows.py)"]:::orchestration
        Eval["Evaluation Script (eval.py)"]:::orchestration
    end

    %% Connections
    RawData --> Build
    Build --> Serialized
    Build --> NodeMap
    Serialized --> DataLoader
    NodeMap --> DataLoader
    DataLoader --> TimeEnc
    TimeEnc --> Emb
    TimeEnc --> TempAtt
    Emb --> GDN
    TempAtt --> GDN
    Emb --> TGAT
    TempAtt --> TGAT
    DataLoader --> Train
    Option --> Train
    GDN --> Train
    TGAT --> Train
    Train --> TrainLog
    TrainLog --> Test
    Test --> Post
    Post --> Eval

    %% Utility usage
    ModUtils -.-> GDN
    ModUtils -.-> TGAT
    GenUtils -.-> DataLoader
    GenUtils -.-> Eval

    %% Click Events
    click Build "https://github.com/wangkai-tech23/tflag/blob/main/dataset/build.py"
    click NodeMap "https://github.com/wangkai-tech23/tflag/blob/main/dataset/node_map.json"
    click DataLoader "https://github.com/wangkai-tech23/tflag/blob/main/util/datasets.py"
    click TimeEnc "https://github.com/wangkai-tech23/tflag/blob/main/modules/time_encoding.py"
    click GDN "https://github.com/wangkai-tech23/tflag/blob/main/model/gdn.py"
    click TGAT "https://github.com/wangkai-tech23/tflag/blob/main/model/tgat.py"
    click Emb "https://github.com/wangkai-tech23/tflag/blob/main/modules/embedding_module.py"
    click TempAtt "https://github.com/wangkai-tech23/tflag/blob/main/modules/temporal_attention.py"
    click ModUtils "https://github.com/wangkai-tech23/tflag/blob/main/modules/utils.py"
    click GenUtils "https://github.com/wangkai-tech23/tflag/blob/main/util/utils.py"
    click Option "https://github.com/wangkai-tech23/tflag/blob/main/option.py"
    click Train "https://github.com/wangkai-tech23/tflag/blob/main/train.py"
    click TrainLog "https://github.com/wangkai-tech23/tflag/tree/main/train_log/"
    click Test "https://github.com/wangkai-tech23/tflag/blob/main/test.py"
    click Post "https://github.com/wangkai-tech23/tflag/blob/main/anomly_time_windows.py"
    click Eval "https://github.com/wangkai-tech23/tflag/blob/main/eval.py"

    %% Styles
    classDef storage fill:#CCE5FF,stroke:#333,stroke-width:1px;
    classDef process fill:#CCFFCC,stroke:#333,stroke-width:1px;
    classDef compute fill:#CCFFE5,stroke:#333,stroke-width:1px;
    classDef orchestration fill:#FFE5CC,stroke:#333,stroke-width:1px;
```
