How mandycheen/tomnet works
Project type and purpose
• TomNet is an open-source machine-learning framework for few-shot IoT malware classification under open-set conditions.
• It leverages function-call graphs (FCGs), Graph Neural Networks (GCN, GraphSAGE, GIN, GAT) and a label-propagation algorithm to classify unseen malware families from limited samples.
File structure
• Top-level scripts:
• RunTraining.py – orchestrates a training run using config/config.json
• RunEval.py – runs evaluation on saved checkpoints
• config/ – JSON experiment settings (dataset paths, model hyperparameters, open-set flags)
• preprocessing/genGpickle.py – converts raw FCG dumps (dot, JSON) to NetworkX .gpickle graphs with opcode node-attributes
• dataset/raw_csv/ – CSV metadata listing FCG samples, families and CPU architectures
• scripts/ – core pipeline modules:
• loadDataset.py – Dataset and DataLoader for few-shot episodes
• fcgVectorize.py – Word2Vec opcode embedding on each node
• models.py – GNN encoder definitions (GCN, GraphSAGE, etc.)
• trainModule.py – end-to-end training and testing loops
• loss.py – few-shot and open-set loss functions
• dataset.py – samplers for k-way n-shot episodes
• train_utils.py, utils.py – auxiliary utilities
• Auto-generated at runtime: embeddings/, checkpoints/, dataset/split/, log files
Key insights from README
• Data flows: raw_csv → preprocessing → .gpickle FCGs → Word2Vec embedding → GNN encoder → label-propagation → classification
• Supports open-set via entropy losses and AUROC metrics
• Configuration drives dataset selection, few-shot parameters, relation network architecture and open-set toggles
Components
a. Main components
1. Data Ingestion Layer
– Raw CSV metadata + raw FCG dumps
– preprocessing/genGpickle.py
2. Storage
– dataset/raw_csv/
– dataset/{name}/{arch}/{family}/*.gpickle
– embeddings/ (Word2Vec models, .npy)
– checkpoints/ (trained model weights)
3. Embedding Module
– scripts/fcgVectorize.py (gensim Word2Vec)
4. Dataset & Loader
– scripts/loadDataset.py & scripts/dataset.py (few-shot sampler)
5. Model & Training
– scripts/models.py (GNN architectures)
– scripts/trainModule.py (forward, backward)
– scripts/loss.py (few-shot + open-set losses)
– scripts/train_utils.py
6. Evaluation Module
– RunEval.py (loads checkpoints, computes metrics, AUROC)
7. Configuration
– config/config.json drives all modules
8. External Libraries
– PyTorch, PyTorch-Geometric, NetworkX, Gensim, Pandas, NumPy, Scikit-Learn
b. Relationships and interactions
• Preprocessing outputs .gpickle FCGs into dataset storage
• Embedding module reads .gpickle, writes embeddings to embeddings/
• Loader reads gpickle + embeddings → constructs few-shot batches
• Trainer reads batches + config → feeds to GNN encoder, loss → writes checkpoints
• Eval reads checkpoints + test data → generates evaluation logs
c. Architectural patterns
– Pipeline pattern: sequential data preprocessing → embedding → training → evaluation
– Modular design: each concern (preprocessing, embedding, dataset, model, loss, utils) in its own module
– Config-driven execution: all paths and hyperparameters controlled by JSON
– Episodic few-shot pattern: custom samplers for k-way n-shot episodes
d. Relevant technologies
– Python 3.8+, PyTorch & PyTorch-Geometric for GNNs, NetworkX for graph I/O, Gensim for Word2Vec, standard data libs
Detail
• Both closed-set and open-set flows if desired (e.g., conditional branches for open-set training in trainModule)
• Ephemeral artifacts (split files, logs) where relevant
• Don’t shy from splitting large modules (e.g., separate Label-Propagation submodule in trainModule)