# qc-tx/node_detection architecture

> Architecture diagram of the GitHub repository qc-tx/node_detection, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-06-21). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/qc-tx/node_detection
- Repository: https://github.com/qc-tx/node_detection

## Overview

##### Project type and purpose
- This is a Python-based machine-learning pipeline for graph-based threat detection. It ingests DARPA OpTC logs (and custom logs), parses them into a graph-format JSON, trains a graph neural network (using PyTorch Geometric), stores models under `models/`, and then runs detection/inference on new logs.

##### File structure
- Top-level:
- LICENSE, README.md
- models/ (pretrained GNN checkpoints + feature/label files)
- parse_log/ (parsers to convert raw logs to JSON graph entries)
- scripts/
– train/ (data-prep, training, validation, evaluation)
– detection/ (run inference on new logs, produce alarms)
– output/ (detection results: alarm.txt, output_time.json)

##### Key insights from README
- Uses PyTorch & PyTorch-Geometric (GNN libs)
- Two main data flows:
a) DARPA OpTC pipeline: parse_darpa.py → graph JSON → train_model.py → models → evaluate_darpatc.py
b) Custom-dataset pipeline: user-formatted JSON → train & detection scripts
- Detection writes to `scripts/output/` and can be evaluated if ground truth exists.

##### Main components and their interactions
a. Data Sources
- DARPA OpTC raw logs (external repo or download)
- Custom JSON logs (user‐supplied)
b. Parsing Layer (parse_log/)
- parse_darpa.py & parse.py: read raw logs → standard JSON graph records
c. Training Layer (scripts/train/)
- data_process_train.py & data_process_validate.py: build train/val datasets
- train_model.py: instantiate GNN (PyG), train, save checkpoints to models/
- evaluate_darpatc.py: evaluate on DARPA‐specific validation set
d. Model Storage
- models/ (checkpoints model_0…model_6, feature/label text files)
e. Inference/Detection Layer (scripts/detection/)
- check_node.py: load a checkpoint from models/, ingest parsed JSON logs, run inference → classified_IOC.txt, alarm.txt
f. Output & Evaluation
- scripts/output/: detection outputs
- scripts/train/groundtruth & evaluate_darpatc.py for labeled‐log evaluation

##### Architectural patterns and design principles
- Pipeline pattern: clear separation of concerns between Parsing → Training → Inference → Evaluation.
- Stateless scripts communicate via file artifacts (JSON, .txt, model checkpoints).
- Modularization: each script folder is a self-contained stage.

##### Technologies & frameworks
- Python 3.6, numpy, pandas, argparse, subprocess, psutil
- PyTorch 1.9.1 and PyTorch-Geometric (scatter, cluster, sparse, spline_conv)
- File-based I/O for inter-stage data passing
- Optional MySQL connection in detection (`mysql_con.py`) for IOC lookups

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Data Sources
    subgraph "External Data Sources"
        DARPA["DARPA OpTC Raw Logs"]:::input
        CustomJSON["Custom JSON Logs"]:::input
    end

    %% Parsing Layer
    subgraph "Parsing Layer"
        ParseCore["parse.py"]:::process
        ParseDARPA["parse_darpa.py"]:::process
        ParseDARPA_TC["parse_darpatc.py"]:::process
    end

    ParsedJSON["Parsed JSON Graph"]:::io

    %% Training Layer
    subgraph "Training Layer"
        DataPrepTrain["data_process_train.py"]:::process
        DataPrepValidate["data_process_validate.py"]:::process
        TrainModel["train_model.py"]:::process
        EvaluateDARPA_TC["evaluate_darpatc.py"]:::process
    end

    %% Ground Truth Storage
    subgraph "Ground Truth Files"
        GT_ID["groundtruth_id.txt"]:::storage
        GT_Node["groundtruth_nodeId.txt"]:::storage
        GT_UUID["groundtruth_uuid.txt"]:::storage
        ID_to_UUID["id_to_uuid.txt"]:::storage
        GT_Script["ground_id_to_uuid.py"]:::process
    end

    %% Model Storage
    subgraph "Model Storage"
        ModelsDir["models/"]:::storage
    end

    %% Inference/Detection Layer
    subgraph "Inference/Detection Layer"
        CheckNode["check_node.py"]:::process
        ParseIOC["parse_IOC.py"]:::process
        IOCtxt["IOC.txt"]:::storage
        ClassifiedIOC["classified_IOC.txt"]:::io
        NewClassifiedIOC["new_classified_IOC.txt"]:::io
        IDtoUUID_Det["id_to_uuid.txt"]:::storage
        AptType["apt_type.py"]:::process
        DataPrepTest["data_process_test.py"]:::process
        MySQLCon["mysql_con.py"]:::process
    end

    %% Output & Evaluation
    subgraph "Output & Evaluation"
        Alarm["alarm.txt"]:::output
        OutputTxt["output.txt"]:::output
        OutputTime["output_time.json"]:::output
    end

    %% Data Flows
    DARPA -->|"raw logs"| ParseDARPA
    DARPA -->|"raw logs"| ParseDARPA_TC
    CustomJSON -->|"raw logs"| ParseCore

    ParseCore -->|"graph JSON"| ParsedJSON
    ParseDARPA -->|"graph JSON"| ParsedJSON
    ParseDARPA_TC -->|"graph JSON"| ParsedJSON

    ParsedJSON -->|"train data"| DataPrepTrain
    ParsedJSON -->|"validation data"| DataPrepValidate

    DataPrepTrain -->|"processed data"| TrainModel
    DataPrepValidate -->|"processed data"| TrainModel

    TrainModel -->|"checkpoints"| ModelsDir
    EvaluateDARPA_TC -->|"evaluation"| ModelsDir

    GT_ID --> GT_Script
    GT_Node --> GT_Script
    GT_UUID --> GT_Script
    ID_to_UUID --> GT_Script
    GT_Script --> EvaluateDARPA_TC

    ParsedJSON -->|"test data"| CheckNode
    ModelsDir -->|"load checkpoint"| CheckNode

    CheckNode --> ClassifiedIOC
    ParseIOC --> ClassifiedIOC
    CheckNode --> NewClassifiedIOC
    IDtoUUID_Det --> CheckNode
    DataPrepTest --> CheckNode
    AptType --> CheckNode
    MySQLCon --> CheckNode

    ClassifiedIOC --> Alarm
    ClassifiedIOC --> OutputTxt
    ClassifiedIOC --> OutputTime

    %% Click Events
    click ParseCore "https://github.com/qc-tx/node_detection/blob/master/parse_log/parse.py"
    click ParseDARPA "https://github.com/qc-tx/node_detection/blob/master/parse_log/parse_darpa.py"
    click ParseDARPA_TC "https://github.com/qc-tx/node_detection/blob/master/parse_log/parse_darpatc.py"

    click DataPrepTrain "https://github.com/qc-tx/node_detection/blob/master/scripts/train/data_process_train.py"
    click DataPrepValidate "https://github.com/qc-tx/node_detection/blob/master/scripts/train/data_process_validate.py"
    click TrainModel "https://github.com/qc-tx/node_detection/blob/master/scripts/train/train_model.py"
    click EvaluateDARPA_TC "https://github.com/qc-tx/node_detection/blob/master/scripts/train/evaluate_darpatc.py"
    click GT_ID "https://github.com/qc-tx/node_detection/blob/master/scripts/train/groundtruth/groundtruth_id.txt"
    click GT_Node "https://github.com/qc-tx/node_detection/blob/master/scripts/train/groundtruth/groundtruth_nodeId.txt"
    click GT_UUID "https://github.com/qc-tx/node_detection/blob/master/scripts/train/groundtruth/groundtruth_uuid.txt"
    click ID_to_UUID "https://github.com/qc-tx/node_detection/blob/master/scripts/train/groundtruth/id_to_uuid.txt"
    click GT_Script "https://github.com/qc-tx/node_detection/blob/master/scripts/train/groundtruth/ground_id_to_uuid.py"

    click ModelsDir "https://github.com/qc-tx/node_detection/tree/master/models/"

    click CheckNode "https://github.com/qc-tx/node_detection/blob/master/scripts/detection/check_node.py"
    click ParseIOC "https://github.com/qc-tx/node_detection/blob/master/scripts/detection/parse_IOC.py"
    click IOCtxt "https://github.com/qc-tx/node_detection/blob/master/scripts/detection/IOC.txt"
    click ClassifiedIOC "https://github.com/qc-tx/node_detection/blob/master/scripts/detection/classified_IOC.txt"
    click NewClassifiedIOC "https://github.com/qc-tx/node_detection/blob/master/scripts/detection/new_classified_IOC.txt"
    click IDtoUUID_Det "https://github.com/qc-tx/node_detection/blob/master/scripts/detection/id_to_uuid.txt"
    click AptType "https://github.com/qc-tx/node_detection/blob/master/scripts/detection/apt_type.py"
    click DataPrepTest "https://github.com/qc-tx/node_detection/blob/master/scripts/detection/data_process_test.py"
    click MySQLCon "https://github.com/qc-tx/node_detection/blob/master/scripts/detection/mysql_con.py"

    click Alarm "https://github.com/qc-tx/node_detection/blob/master/scripts/output/alarm.txt"
    click OutputTxt "https://github.com/qc-tx/node_detection/blob/master/scripts/output/output.txt"
    click OutputTime "https://github.com/qc-tx/node_detection/blob/master/scripts/output/output_time.json"

    %% Styles
    classDef input fill:#ccf,stroke:#333,stroke-width:1px
    classDef process fill:#9cf,stroke:#333,stroke-width:1px
    classDef io fill:#cfc,stroke:#333,stroke-width:1px
    classDef storage fill:#ff9,stroke:#333,stroke-width:1px
    classDef output fill:#afa,stroke:#333,stroke-width:1px
```
