# mandycheen/tomnet architecture

> Architecture diagram of the GitHub repository mandycheen/tomnet, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-08). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mandycheen/tomnet
- Repository: https://github.com/mandycheen/tomnet

## Overview

##### Project type and purpose
- TomNet is an open-source machine-learning framework for few-shot IoT malware classification under open-set conditions.
- It leverages function-call graphs (FCGs), Graph Neural Networks (GCN, GraphSAGE, GIN, GAT) and a label-propagation algorithm to classify unseen malware families from limited samples.

##### File structure
- Top-level scripts:
- RunTraining.py – orchestrates a training run using config/config.json
- RunEval.py – runs evaluation on saved checkpoints
- config/ – JSON experiment settings (dataset paths, model hyperparameters, open-set flags)
- preprocessing/genGpickle.py – converts raw FCG dumps (dot, JSON) to NetworkX .gpickle graphs with opcode node-attributes
- dataset/raw_csv/ – CSV metadata listing FCG samples, families and CPU architectures
- scripts/ – core pipeline modules:
- loadDataset.py – Dataset and DataLoader for few-shot episodes
- fcgVectorize.py – Word2Vec opcode embedding on each node
- models.py – GNN encoder definitions (GCN, GraphSAGE, etc.)
- trainModule.py – end-to-end training and testing loops
- loss.py – few-shot and open-set loss functions
- dataset.py – samplers for k-way n-shot episodes
- train_utils.py, utils.py – auxiliary utilities
- Auto-generated at runtime: embeddings/, checkpoints/, dataset/split/, log files

##### Key insights from README
- Data flows: raw_csv → preprocessing → .gpickle FCGs → Word2Vec embedding → GNN encoder → label-propagation → classification
- Supports open-set via entropy losses and AUROC metrics
- Configuration drives dataset selection, few-shot parameters, relation network architecture and open-set toggles

##### Components
a. Main components
1. Data Ingestion Layer
– Raw CSV metadata + raw FCG dumps
– preprocessing/genGpickle.py
2. Storage
– dataset/raw_csv/
– dataset/{name}/{arch}/{family}/*.gpickle
– embeddings/ (Word2Vec models, .npy)
– checkpoints/ (trained model weights)
3. Embedding Module
– scripts/fcgVectorize.py (gensim Word2Vec)
4. Dataset & Loader
– scripts/loadDataset.py & scripts/dataset.py (few-shot sampler)
5. Model & Training
– scripts/models.py (GNN architectures)
– scripts/trainModule.py (forward, backward)
– scripts/loss.py (few-shot + open-set losses)
– scripts/train_utils.py
6. Evaluation Module
– RunEval.py (loads checkpoints, computes metrics, AUROC)
7. Configuration
– config/config.json drives all modules
8. External Libraries
– PyTorch, PyTorch-Geometric, NetworkX, Gensim, Pandas, NumPy, Scikit-Learn
b. Relationships and interactions
- Preprocessing outputs .gpickle FCGs into dataset storage
- Embedding module reads .gpickle, writes embeddings to embeddings/
- Loader reads gpickle + embeddings → constructs few-shot batches
- Trainer reads batches + config → feeds to GNN encoder, loss → writes checkpoints
- Eval reads checkpoints + test data → generates evaluation logs
c. Architectural patterns
– Pipeline pattern: sequential data preprocessing → embedding → training → evaluation
– Modular design: each concern (preprocessing, embedding, dataset, model, loss, utils) in its own module
– Config-driven execution: all paths and hyperparameters controlled by JSON
– Episodic few-shot pattern: custom samplers for k-way n-shot episodes
d. Relevant technologies
– Python 3.8+, PyTorch & PyTorch-Geometric for GNNs, NetworkX for graph I/O, Gensim for Word2Vec, standard data libs

##### Detail
- Both closed-set and open-set flows if desired (e.g., conditional branches for open-set training in trainModule)
- Ephemeral artifacts (split files, logs) where relevant
- Don’t shy from splitting large modules (e.g., separate Label-Propagation submodule in trainModule)

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Sources
    RawCSV["Raw CSV Metadata"]:::io
    RawFCG["Raw FCG Dumps"]:::io

    %% Data Ingestion
    Preprocessing["FCG Preprocessor"]:::process
    click Preprocessing "https://github.com/mandycheen/tomnet/blob/main/preprocessing/genGpickle.py"

    %% Storage Cylinders
    RawCSVStorage[("dataset/raw_csv/")]:::storage
    click RawCSVStorage "https://github.com/mandycheen/tomnet/tree/main/dataset/raw_csv/"
    GpickleStorage[("GPickle FCGs")]:::storage
    EmbeddingsStorage[("embeddings/")]:::storage
    click EmbeddingsStorage "https://github.com/mandycheen/tomnet/tree/main/embeddings/"
    CheckpointsStorage[("checkpoints/")]:::storage
    click CheckpointsStorage "https://github.com/mandycheen/tomnet/tree/main/checkpoints/"
    SplitStorage[("dataset/split/")]:::storage
    click SplitStorage "https://github.com/mandycheen/tomnet/tree/main/dataset/split/"

    %% Embedding Module
    Embedding["Word2Vec Embedding"]:::process
    click Embedding "https://github.com/mandycheen/tomnet/blob/main/scripts/fcgVectorize.py"

    %% Data Loading Subgraph
    subgraph "Data Loading"
        Loader["Dataset Loader"]:::process
        click Loader "https://github.com/mandycheen/tomnet/blob/main/scripts/loadDataset.py"
        Sampler["Few-Shot Sampler"]:::process
        click Sampler "https://github.com/mandycheen/tomnet/blob/main/scripts/dataset.py"
    end

    %% Model & Training Subgraph
    subgraph "Model & Training"
        Models["GNN Architectures"]:::process
        click Models "https://github.com/mandycheen/tomnet/blob/main/scripts/models.py"
        TrainModule["Training Loop"]:::process
        click TrainModule "https://github.com/mandycheen/tomnet/blob/main/scripts/trainModule.py"
        Loss["Loss Functions"]:::process
        click Loss "https://github.com/mandycheen/tomnet/blob/main/scripts/loss.py"
        Utils["Training Utilities"]:::process
        click Utils "https://github.com/mandycheen/tomnet/blob/main/scripts/train_utils.py"
    end

    %% Evaluation Module
    Evaluation["Evaluation"]:::process
    click Evaluation "https://github.com/mandycheen/tomnet/blob/main/RunEval.py"

    %% Configuration and Orchestration
    Config["config/config.json"]:::process
    click Config "https://github.com/mandycheen/tomnet/blob/main/config/config.json"
    RunTrain["RunTraining.py"]:::io
    click RunTrain "https://github.com/mandycheen/tomnet/blob/main/RunTraining.py"
    RunEvalScript["RunEval.py"]:::io
    click RunEvalScript "https://github.com/mandycheen/tomnet/blob/main/RunEval.py"

    %% External Libraries
    subgraph "External Libraries"
        PyTorch["PyTorch / PyG"]:::external
        NetworkX["NetworkX"]:::external
        Gensim["Gensim"]:::external
        Pandas["Pandas"]:::external
        NumPy["NumPy"]:::external
        Sklearn["Scikit-Learn"]:::external
    end

    %% Data Flow
    RawCSV --> RawCSVStorage
    RawFCG --> Preprocessing
    RawCSVStorage --> Preprocessing
    Preprocessing --> GpickleStorage

    GpickleStorage --> Embedding
    Embedding --> EmbeddingsStorage

    GpickleStorage --> Loader
    EmbeddingsStorage --> Loader
    Loader --> Sampler
    Sampler --> TrainModule

    TrainModule --> CheckpointsStorage
    TrainModule --> SplitStorage

    CheckpointsStorage --> Evaluation
    SplitStorage --> Evaluation

    %% Orchestration and Config
    RunTrain --> Preprocessing
    RunTrain --> Embedding
    RunTrain --> Loader
    RunTrain --> TrainModule
    RunEvalScript --> Evaluation

    Config --> Preprocessing
    Config --> Embedding
    Config --> Loader
    Config --> TrainModule
    Config --> Evaluation

    %% External deps integration
    Preprocessing --> NetworkX
    Embedding --> Gensim
    Loader --> PyTorch
    TrainModule --> PyTorch
    Models --> PyTorch
    Loss --> PyTorch
    Loader --> NumPy
    Loader --> Pandas
    Evaluation --> Sklearn

    %% Styling
    classDef storage fill:#BBDEFB,stroke:#2196F3,stroke-width:2px
    classDef process fill:#C8E6C9,stroke:#2E7D32,stroke-width:2px
    classDef external fill:#FFE0B2,stroke:#EF6C00,stroke-width:2px
    classDef io fill:#E1BEE7,stroke:#8E24AA,stroke-width:2px
```
