How guanxinjie124/treevul works
Project Type & Purpose
• This is a research replication and experimental package designed to predict fine-grained vulnerability types from security patch commits. It uses tree-structured CWE information to enhance vulnerability classification.
• It supports several model types: deep learning models (TreeVul and its two variants, TreeVul-t and TreeVul-h) and neural baselines (Bi-LSTM and CodeBERT) as well as multiple machine learning baselines.
Main High-Level Components
a. Data Layer:
• Data Sources: JSON datasets (e.g., dataset_cleaned.json, dataset_cleaned_level3.json, test_set.json, train_set.json, validation_set.json, and various CVE/CWE files in the data/ directory).
• Preprocessing & Utility Modules: Files such as utils.py, process_edit.py, and reader modules (reader_treevul_hunk.py, reader_cwe.py, and those in Baseline and Baseline_ml directories) are responsible for parsing, cleaning, and preparing the data.
• CWE Tree Construction: Files like cwe_tree.json, cwe_path.json, and valid_cwes.json along with callbacks (callbacks.py) that prepare and configure the CWE tree structure.
b. Model Components:
• TreeVul Models: Located in TreeVul/ (model_treevul.py) and its variants—TreeVul-t and TreeVul-h—each with their own configuration files (e.g., config_treevul.json, config_ablation_notree.json, and config_ablation_noedit.json).
• Neural Baseline Models: In the Baseline directory, you have Bi-LSTM (model_baseline_bilstm.py, config_baseline_bilstm.json) and CodeBERT (model_baseline_codebert.py, config_baseline_codebert.json). These utilize tokenizers and dataset readers specific to each model.
• Machine Learning Baselines: In the Baseline_ml directory, ml_baseline.py and associated readers/tokenizers support alternative approaches (SVM, RF, LR, XGB, KNN).
c. Training & Evaluation Layer:
• Training Engine: The training is facilitated using custom modules such as custom_trainer.py and custom_train.py that wrap AllenNLP’s training procedures. They support gradient descent and incorporate callbacks.
• Callbacks & Custom Modules: callbacks.py, custom_modules.py, and custom_metric.py (present in multiple directories) ensure proper integration of CWE tree preparations, metric computation, and module reusability.
• Command Interfaces & Configuration: Various configuration files (e.g., test_config.json, test_config_tree.json) provide fine-grained control over training settings. Execution commands using AllenNLP and environment settings (CUBLAS_WORKSPACE_CONFIG) are included in the README.
• Evaluation Scripts: predict.py and cal_metrics.py manage evaluation, supporting the differences among the models (with or without tree structure).
d. External Dependencies:
• Frameworks & Libraries: PyTorch, AllenNLP, HuggingFace Transformers (for CodeBERT), and AllenNLP’s configuration & training workflows are core. The README details adoption of CUDA settings and specific package versions.
• Pretrained Models & Embeddings: The CodeBERT model is auto-downloaded, and GloVe embeddings are used for the Bi-LSTM baseline.
Relationships & Data Flow
• Data Ingestion: Data is loaded from the data folder → processed by utility functions (in utils.py, process_edit.py) → fed into appropriate dataset readers (reader_treevul_hunk.py, reader_baseline*, reader_baseline_ml.py).
• Model Training: Each type of model (TreeVul variants, neural baselines, ML baselines) picks its own configuration and training pipeline:
• The deep learning models are set up via AllenNLP’s training command with custom trainers and callbacks.
• Machine learning models, on the other hand, are initiated via separate scripts (e.g., ml_baselines.py).
• Model Evaluation: Post-training, predict.py and cal_metrics.py are used to generate prediction results and calculate evaluation metrics.
• Integration of CWE Knowledge: A key aspect is that the CWE tree structure (constructed and maintained by callbacks and reader_cwe.py) informs both the model architecture (e.g., hierarchical/chain-based designs) and the embedding processes (via custom_PTM_embedder.py).
Architectural Patterns & Design Principles
• Layered Modular Design: The separation between data preprocessing, model definitions, training controllers, and evaluation modules makes the system modular.
• Extensibility: Multiple directories (TreeVul, Baseline, Baseline_ml) allow side-by-side experiments with varied approaches (tree-structured deep learning vs. standard neural and ML baselines).
• Reusability: Common tools (utils.py, custom_modules.py, custom_metric.py) and configuration reuse across components highlight good modularization.
• Callback & Configuration Driven: The use of callbacks and JSON configuration files emphasize a flexible, configuration-driven workflow governed by AllenNLP.