How dessertlab/evil works
Project Type & Purpose
• This is a research and experimental repository focused on exploiting software via natural language. In essence, EVIL demonstrates how Neural Machine Translation (NMT) techniques (using models like Seq2Seq and CodeBERT) can automatically generate exploits – producing both assembly shellcodes (decoders) and Python encoders – from natural language descriptions.
• The project is structured to support data collection, preprocessing, model training, evaluation, and replication of experiments from the referenced paper.
Analyzing the File Structure
• You have directories and files that correspond to different components:
• “datasets”: Contains separate folders for “encoder” (Python code exploits) and “decoder” (assembly/shellcode exploits) datasets. These datasets are pre-split into training, dev, and test sets.
• “model”: Holds scripts (e.g., model.py, run.py, fine_tune scripts) and utilities for model creation and fine-tuning as well as evaluation preparation (like bleu.py for computing BLEU scores).
• “xnmt”: A rich subdirectory, indicating that the XNMT framework is leveraged heavily for NMT tasks. This folder contains submodules for model components (encoders, decoders, attenders), experiment management, training regimens, and even Cython acceleration modules.
• “utils”: Contains preprocessing scripts (under utils/preproc) that convert data between formats (e.g., json_to_seq2seq, seq2seq_output_to_code) and evaluation utilities (e.g., bleu_score.py, codegen_eval.py) that support performance assessment.
• Script files (e.g., CodeBERT_Launch.sh, Seq2Seq_Launch.sh, evaluate.sh) represent entry points to run experiments, select models, and orchestrate different workflows.
• Other support files (INSTALL.md, README.md, requirements, environment YAML files) indicate dependencies, instructions, and environmental setup.
Key Components
a. Data Input Layer:
• Datasets: Highlight two primary branches – the “encoder” (Python code with natural language descriptions) and “decoder” (assembly code with shells/decoders) datasets. These feed directly into the subsequent pipeline.
b. Preprocessing Module:
• Preprocessing scripts (under the “utils/preproc” folder) transform raw dataset files into an appropriate format for the NMT models. They might include canonicalization and conversion steps (e.g., OurCanonical.py, json_to_seq2seq.py).
c. Model & Training Module:
• Two major experiment types:
– CodeBERT-based experiments (as launched via CodeBERT_Launch.sh)
– Seq2Seq experiments (invoked by Seq2Seq_Launch.sh)
• The “model” folder contains scripts (model.py, run.py) that coordinate training or inference, while “xnmt” holds the broader NMT framework implementation (including model parts, training strategies, and experimental configurations).
• Architectural details such as the encoder-decoder structures, attention mechanisms, and layers organized in XNMT.
d. Evaluation & Post-Processing Module:
• Evaluation is facilitated through scripts like evaluate.sh and utilities under “utils/eval” (which include BLEU scoring and performance metrics) to measure the quality of generated exploits.
• Additional postprocessing (e.g., converting outputs into executable or interpretable code formats) happens via preprocessing utilities in “utils/preproc” (like seq2seq_output_to_code.py).
e. Experiment Orchestration & Supporting Components:
• The launching scripts (CodeBERT_Launch.sh, Seq2Seq_Launch.sh) orchestrate the overall experiments.
• Configuration and dependency management files (requirements.txt, environment YAMLs) ensure that the execution environment is configured appropriately.
• “notebooks” allow for exploratory analysis and likely serve as experimental diagnostics/visualizations of the training or evaluation processes.
• “Appendix” holds additional details on test sets (compiled statistics and external information) that further validate the experiments.
Essential Relationships & Data Flow
• Flow starts with the “datasets” that feed into the “preprocessing” module.
• Preprocessed data is directed to the “Model & Training Module” where the NMT models (implemented with a combination of CodeBERT or Seq2Seq architectures within the XNMT framework) are trained and fine-tuned.
• Post training, the outputs (i.e., generated exploits) are sent to the “Evaluation” module where BLEU scores and other sanity checks are computed.
• Experiment orchestration scripts glue all these components together, ensuring a smooth execution from data ingestion to evaluation.
Architectural Patterns & Technologies
• The architecture follows a modular design with clear separations: data ingestion, preprocessing, model training, and evaluation.
• It leverages the XNMT framework extensively for sequence modeling and NMT tasks.
• Use of shell scripts indicates a command-line driven orchestration pattern.
• Integration of pre-trained models (as seen in “model/pretrained_models”) suggests an approach that builds on transfer learning (especially for CodeBERT experiments).