# threatrace-detector/threatrace architecture

> Architecture diagram of the GitHub repository threatrace-detector/threatrace, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/threatrace-detector/threatrace
- Repository: https://github.com/threatrace-detector/threatrace

## Overview

##### Project Type and Purpose
- This is an evaluation‐reproduction pipeline for the THREATRACE intrusion‐detection system.
- It combines data parsing, graph processing (via the GraphChi C++ engine), model training/inference (via PyTorch Geometric and Python), and evaluation.

##### Top‐Level Components
A. External Data Sources
- DARPA TC, StreamSpot, UnicornSC raw archives
B. Data Ingestion & Parsing (Python scripts in `scripts/parse_*.py`)
- Reads raw archives, outputs edge‐based graph text (`graphchi-cpp-master/graph_data/.../empty.txt`)
C. Graph Preprocessing & Sharding (C++ modules in `graphchi-cpp-master/src/preprocessing` and `src/shards`)
- Splits graphs into blocks, shards for out‐of‐core processing
D. GraphChi Engine (C++ core in `graphchi-cpp-master/src/engine` + APIs in `src/api`)
- Executes user‐defined graph algorithms (feature extraction)
E. Feature & Label Storage (`example_models/.../feature.txt`, `label.txt`)
F. Model Training & Inference (Python scripts `train_*.py`, shell runners `run_*.sh`)
- Uses PyTorch Geometric to train graph neural networks on extracted features
- Persists model artifacts under `models/` or `example_models/`
G. Testing & Evaluation (Python scripts `test_*.py`, `evaluate_*.py`)
- Runs benign/attack flows, compares predictions to ground truth in `groundtruth/`
H. Results & Metrics (outputs accuracy, ROC, etc.)

##### Relationships & Data Flow
1. External Data → Parsing Scripts → Graph Data files
2. Graph Data files → Preprocessing & Sharding → Sharded Graph Blocks
3. Sharded Graph Blocks → GraphChi Engine → Extracted Feature + Label files
4. Feature + Label files → Model Training (PyG) → Trained Model files
5. Trained Models + Graph Data → Inference Shell scripts → Prediction logs
6. Prediction logs + Groundtruth → Evaluation scripts → Performance metrics

##### Architectural Patterns & Design Principles
- Pipeline Architecture: clear linear stages (Ingestion → Preprocessing → Extraction → Training → Evaluation)
- Polyglot Modularization:
- Python layer for orchestration, parsing, ML training
- C++ GraphChi engine for scalable graph operations
- Separation of Concerns:
- Data Layer (graph_data, example_models, groundtruth)
- Compute Layer (GraphChi engine, PyG training)
- Orchestration Layer (Python scripts, shell scripts)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External Data
    ExternalData["External Data Sources\n(DARPA TC, StreamSpot, UnicornSC)"]:::external
    click ExternalData "https://github.com/threatrace-detector/threatrace/tree/master/graphchi-cpp-master/graph_data/"

    %% Data Stores
    subgraph "Data Stores"
        direction TB
        GraphData["Graph Data Files\n(empty.txt)"]:::data
        FeatureLabel["Feature & Label Files"]:::data
        GroundTruth["Ground Truth Data\n(cadets, fivedirections,\ntheia, trace)"]:::data
        click GraphData "https://github.com/threatrace-detector/threatrace/tree/master/graphchi-cpp-master/graph_data/"
        click FeatureLabel "https://github.com/threatrace-detector/threatrace/blob/master/example_models/darpatc/*/feature.txt"
        click GroundTruth "https://github.com/threatrace-detector/threatrace/blob/master/groundtruth/cadets.txt"
    end

    %% Python Orchestrator
    subgraph "Python Orchestrator" 
        direction TB
        parseDARPA["parse_darpatc.py"]:::python
        parseStream["parse_streamspot.py"]:::python
        parseUnicorn["parse_unicornsc.py"]:::python
        TrainDARPA["train_darpatc.py (PyG)"]:::python
        TrainStream["train_streamspot.py (PyG)"]:::python
        TrainUnicorn["train_unicornsc.py (PyG)"]:::python
        TestEval["Testing & Evaluation Scripts"]:::evaluation
        click parseDARPA "https://github.com/threatrace-detector/threatrace/blob/master/scripts/parse_darpatc.py"
        click parseStream "https://github.com/threatrace-detector/threatrace/blob/master/scripts/parse_streamspot.py"
        click parseUnicorn "https://github.com/threatrace-detector/threatrace/blob/master/scripts/parse_unicornsc.py"
        click TrainDARPA "https://github.com/threatrace-detector/threatrace/blob/master/scripts/train_darpatc.py"
        click TrainStream "https://github.com/threatrace-detector/threatrace/blob/master/scripts/train_streamspot.py"
        click TrainUnicorn "https://github.com/threatrace-detector/threatrace/blob/master/scripts/train_unicornsc.py"
    end

    %% GraphChi Engine
    subgraph "GraphChi Engine" 
        direction TB
        Preprocessing["Preprocessing & Sharding"]:::cpp
        EngineCore["Engine Core\n(graphchi_engine.hpp)"]:::cpp
        API["GraphChi API"]:::cpp
        click Preprocessing "https://github.com/threatrace-detector/threatrace/blob/master/graphchi-cpp-master/src/preprocessing/blocksplitter.cpp"
        click Preprocessing "https://github.com/threatrace-detector/threatrace/blob/master/graphchi-cpp-master/src/preprocessing/sharder_basic.cpp"
        click Preprocessing "https://github.com/threatrace-detector/threatrace/blob/master/graphchi-cpp-master/src/preprocessing/sharder.hpp"
        click Preprocessing "https://github.com/threatrace-detector/threatrace/tree/master/graphchi-cpp-master/src/shards/"
        click EngineCore "https://github.com/threatrace-detector/threatrace/blob/master/graphchi-cpp-master/src/engine/graphchi_engine.hpp"
        click EngineCore "https://github.com/threatrace-detector/threatrace/tree/master/graphchi-cpp-master/src/engine/"
        click API "https://github.com/threatrace-detector/threatrace/tree/master/graphchi-cpp-master/src/api/"
    end

    %% Model Artifacts
    subgraph "Model Artifacts"
        Models["Trained Models\n(models/)"]:::model
        click Models "https://github.com/threatrace-detector/threatrace/tree/master/models/"
    end

    %% Orchestration Utilities
    subgraph "Utilities"
        direction TB
        Setup["setup.py"]:::python
        Monitor["moniter.py"]:::python
        DataProcTrain["data_process_train.py"]:::python
        DataProcTest["data_process_test.py"]:::python
        click Setup "https://github.com/threatrace-detector/threatrace/blob/master/scripts/setup.py"
        click Monitor "https://github.com/threatrace-detector/threatrace/blob/master/scripts/moniter.py"
        click DataProcTrain "https://github.com/threatrace-detector/threatrace/blob/master/scripts/data_process_train.py"
        click DataProcTest "https://github.com/threatrace-detector/threatrace/blob/master/scripts/data_process_test.py"
    end

    %% Shell Scripts
    RunAttackSS["run_attack.sh\n(streamspot)"]:::python
    RunBenignSS["run_benign.sh\n(streamspot)"]:::python
    RunAttackUS["run_attack.sh\n(unicornsc)"]:::python
    RunBenignUS["run_benign.sh\n(unicornsc)"]:::python
    click RunAttackSS "https://github.com/threatrace-detector/threatrace/blob/master/example_models/streamspot/run_attack.sh"
    click RunBenignSS "https://github.com/threatrace-detector/threatrace/blob/master/example_models/streamspot/run_benign.sh"
    click RunAttackUS "https://github.com/threatrace-detector/threatrace/blob/master/example_models/unicornsc/run_attack.sh"
    click RunBenignUS "https://github.com/threatrace-detector/threatrace/blob/master/example_models/unicornsc/run_benign.sh"

    %% Results
    Results["Results & Metrics\n(accuracy, ROC, etc.)"]:::evaluation

    %% Data Flow
    ExternalData -->|"raw archives"| parseDARPA
    ExternalData -->|"raw archives"| parseStream
    ExternalData -->|"raw archives"| parseUnicorn

    parseDARPA -->|"graph text"| GraphData
    parseStream -->|"graph text"| GraphData
    parseUnicorn -->|"graph text"| GraphData

    GraphData -->|"blocksplitter/sharder"| Preprocessing
    Preprocessing -->|"shards"| EngineCore

    EngineCore -->|"GraphChi execution"| FeatureLabel
    API --> EngineCore

    FeatureLabel -->|"features+labels"| TrainDARPA
    FeatureLabel -->|"features+labels"| TrainStream
    FeatureLabel -->|"features+labels"| TrainUnicorn

    TrainDARPA --> Models
    TrainStream --> Models
    TrainUnicorn --> Models

    Models -->|"model files"| RunAttackSS
    Models -->|"model files"| RunBenignSS
    Models -->|"model files"| RunAttackUS
    Models -->|"model files"| RunBenignUS

    RunAttackSS --> PredictionLogs["Prediction Logs"]:::data
    RunBenignSS --> PredictionLogs
    RunAttackUS --> PredictionLogs
    RunBenignUS --> PredictionLogs

    PredictionLogs -->|"with groundtruth"| TestEval
    GroundTruth --> TestEval
    TestEval --> Results

    %% Styles
    classDef python fill:#cce5ff,stroke:#339af0
    classDef cpp fill:#e9ecef,stroke:#6c757d
    classDef data fill:#d4edda,stroke:#28a745
    classDef model fill:#ffe5b4,stroke:#ff7f00
    classDef evaluation fill:#f8d7da,stroke:#dc3545
    classDef external fill:#ffffff,stroke:#333333,stroke-dasharray: 5 5
```
