How qc-tx/node_detection works
Project type and purpose
• This is a Python-based machine-learning pipeline for graph-based threat detection. It ingests DARPA OpTC logs (and custom logs), parses them into a graph-format JSON, trains a graph neural network (using PyTorch Geometric), stores models under models/, and then runs detection/inference on new logs.
File structure
• Top-level:
• LICENSE, README.md
• models/ (pretrained GNN checkpoints + feature/label files)
• parse_log/ (parsers to convert raw logs to JSON graph entries)
• scripts/
– train/ (data-prep, training, validation, evaluation)
– detection/ (run inference on new logs, produce alarms)
– output/ (detection results: alarm.txt, output_time.json)
Key insights from README
• Uses PyTorch & PyTorch-Geometric (GNN libs)
• Two main data flows:
a) DARPA OpTC pipeline: parse_darpa.py → graph JSON → train_model.py → models → evaluate_darpatc.py
b) Custom-dataset pipeline: user-formatted JSON → train & detection scripts
• Detection writes to scripts/output/ and can be evaluated if ground truth exists.
Main components and their interactions
a. Data Sources
• DARPA OpTC raw logs (external repo or download)
• Custom JSON logs (user‐supplied)
b. Parsing Layer (parse_log/)
• parse_darpa.py & parse.py: read raw logs → standard JSON graph records
c. Training Layer (scripts/train/)
• data_process_train.py & data_process_validate.py: build train/val datasets
• train_model.py: instantiate GNN (PyG), train, save checkpoints to models/
• evaluate_darpatc.py: evaluate on DARPA‐specific validation set
d. Model Storage
• models/ (checkpoints model_0…model_6, feature/label text files)
e. Inference/Detection Layer (scripts/detection/)
• check_node.py: load a checkpoint from models/, ingest parsed JSON logs, run inference → classified_IOC.txt, alarm.txt
f. Output & Evaluation
• scripts/output/: detection outputs
• scripts/train/groundtruth & evaluate_darpatc.py for labeled‐log evaluation
Architectural patterns and design principles
• Pipeline pattern: clear separation of concerns between Parsing → Training → Inference → Evaluation.
• Stateless scripts communicate via file artifacts (JSON, .txt, model checkpoints).
• Modularization: each script folder is a self-contained stage.
Technologies & frameworks
• Python 3.6, numpy, pandas, argparse, subprocess, psutil
• PyTorch 1.9.1 and PyTorch-Geometric (scatter, cluster, sparse, spline_conv)
• File-based I/O for inter-stage data passing
• Optional MySQL connection in detection (mysql_con.py) for IOC lookups