How ryderling/malgraph works
Main Components
• Data Preprocessing Module
• Located mainly in the samples folder (PreProcess.py and sample files). This module interfaces with IDA Pro to parse disassembled executable files and converts the raw data (using the function parse_json_list_2_pyg_object()) into PyTorch Geometric data objects.
• Vocabulary Builder
• Found in the src/utils/Vocabulary.py file. This component is vital for constructing and saving the vocabulary of external function names across the training dataset.
• Configuration Management
• The configs/default.yaml file contains hyper-parameters and settings for both preprocessing and model training. This file drives how the training process is initialized.
• Model Architecture
• In the src/models folder, the HierarchicalGraphModel.py provides the core model implementation, capturing the hierarchical structure (inter-function call graph plus intra-function control flow graphs).
• Model Training & Execution
• The src/DistTrainModel.py script is the entry point for training the model. It utilizes the preprocessed data, configured parameters, and the defined model architecture.
• Auxiliary Utilities
• Under src/utils, additional helper modules (like FunctionHelpers.py, ParameterClasses.py, PreProcessedDataset.py, RealBatch.py) facilitate data manipulation, batching, and parameter management.
Relationships and Data Flow
• The Configuration Management component (configs/default.yaml) should be displayed adjacent to both the preprocessing and training modules, emphasizing that hyper-parameters are shared across the pipeline.
• Finally, the Model architecture (HierarchicalGraphModel) is central to processing the graph data, with training routines (DistTrainModel.py) orchestrating the training process.
Architectural Patterns and Technologies
• Clearly separate concerns between data ingestion (preprocessing), data transformation (vocabulary building and dataset preparations), and the core model training.
• That the system employs a hierarchical model design, where data is represented on multiple abstraction levels (inter-function and intra-function graphs).
• Use of PyTorch Geometric for graph data management and neural computations—a critical dependency in the design.
• The design implicitly follows a modular structure, making it easier to enhance or replace individual components (e.g., upgrading preprocessing or experimenting with different graph models).