How mohammadi-ali/metamorphasm works
This repository is an offline research benchmark pipeline, not an application service. Its architecture centers on one shared dataset-generation subsystem and several model-specific experiment subsystems that all consume the same benchmark artifacts.
1. Top-level purpose
- The repo evaluates whether LLMs can generate obfuscated assembly code.
- The core output is the MetamorphASM Dataset (MAD) plus experiments across multiple model families.
2. Dataset creation subsystem
- MetamorphASM_dataset_creation/ is the data-engineering boundary of the project.
- It contains scripts for transforming assembly into obfuscated variants, including:
- dead_code_Insertion.py
- register_substitution.py
- control_flow.py
- shuffle.py
- Obfuscation.py
- Assembly_separation.py
- Code_Linearization.py
- make_dataset.py
- This layer is responsible for taking source assembly material and producing benchmark-ready examples.
3. Benchmark outputs
- The dataset folder includes CSVs under Data(200 Samples)/ for the three main obfuscation types:
- dead_code_dataset.csv
- register_substitution_dataset.csv
- control_flow_change_dataset.csv
- There is also a mini_dataset.zip and several path/config artifacts, indicating prepared subsets and intermediate outputs.
4. Obfuscation task decomposition
- The benchmark is organized around three transformation targets:
- dead code insertion
- register substitution
- control-flow change
- These are the main semantic boundaries that the evaluation code and prompts are built around.
5. Prompt and few-shot resources
- few_shots/ stores shared few-shot CSV examples used for prompting.
- Some model-specific folders also include their own few-shot files, suggesting prompt templates are adapted per model family and per task.
6. Model experiment layer: notebook-driven workflows
- CodeT5/ and GPT3.5_4/ are notebook-centric experiment folders.
- These contain Jupyter notebooks that appear to run generation, transformation, and analysis experiments interactively rather than through a single CLI pipeline.
7. Model experiment layer: script-driven workflows
- Starcoder2-7b/, Codegemma_Codellama/, and Llama_3.1/ use Python scripts for task execution.
- These folders likely wrap prompting, model invocation, and result collection in a more automated batch style than the notebook-based folders.
8. Per-model isolation
- Each model family has its own execution boundary and dependencies:
- Starcoder2-7b/ includes DeadCode.py, Transpose.py, substitution.py, and score.py
- Codegemma_Codellama/ includes core.py, process_tasks.py, task_prompts.py, and setup.py
- Llama_3.1/ includes Llama_Experiment.py
- This indicates the repo is structured as multiple adapters over a shared benchmark, not one shared runtime.
9. Scoring and evaluation
- Starcoder2-7b/score.py suggests automated scoring/assessment of generated outputs.
- The README also mentions information-theoretic metrics and manual human review, so evaluation is split between automated metrics and qualitative validation.
10. Model/runtime dependencies
- requirements.txt appears in multiple experiment folders, indicating separate environment requirements per model stack.
- Codegemma_Codellama/setup.py and deadcode_package.sh suggest packaging and local execution support for that model family.
11. Data flow across the repo
- The intended flow is:
- raw assembly data → dataset creation scripts → obfuscated benchmark CSVs → prompt/few-shot inputs → model-specific experiment scripts/notebooks → generated outputs → scoring/manual review.
- The central reusable asset is the dataset; the model folders are consumers of that asset.
12. What to draw in an architecture diagram
- Draw one shared Dataset Generation block feeding multiple Model Experiment blocks.
- Show the few_shots resources as an input to the prompting layer.
- Show Evaluation/Scoring as a downstream stage that consumes model outputs.
- Keep model folders separate, because each one has its own runner and dependency boundary.
13. Scope note
- Despite the README mentioning more model families, the visible repo content mainly exposes experiment paths for GPT, CodeT5, Starcoder2, CodeGemma/CodeLlama, and LLaMA 3.1, plus the dataset-construction pipeline.
Main entry points: Data(200 Samples) Benchmark CSVs · few_shots Few-shot pool · Obfuscation.py Obfuscation ops · make_dataset.py Dataset pipeline
Components 18
Dataset generation
Transforms raw assembly material into benchmark-ready obfuscated datasets.
- Dataset pipeline · Shared assembly-to-benchmark generation flow.
MetamorphASM_dataset_creation/make_dataset.py - Obfuscation ops · Core obfuscation transformations for the benchmark tasks.
MetamorphASM_dataset_creation/Obfuscation.py - Assembly prep · Assembly splitting and linearization before obfuscation.
- Dead code · Dead-code insertion task implementation.
MetamorphASM_dataset_creation/dead_code_Insertion.py - Register swap · Register substitution task implementation.
MetamorphASM_dataset_creation/register_substitution.py - Control flow · Control-flow change task implementation.
MetamorphASM_dataset_creation/control_flow.py - Benchmark CSVs · Prepared datasets for the three main obfuscation targets.
MetamorphASM_dataset_creation/Data(200 Samples)
Shared inputs
Reusable prompt and few-shot resources consumed by multiple experiments.
- Few-shot pool · Shared few-shot examples used across model families.
few_shots - Llama shots · Model-specific few-shot files for LLaMA experiments.
Llama_3.1/Few_shot_dead_code.csv
Notebook experiments
Interactive model workflows run through Jupyter notebooks.
- CodeT5 notebooks · Notebook-based experiments for CodeT5 generation and analysis.
CodeT5 - GPT-4 notebooks · Notebook-driven MetamorphASM experiments for GPT-3.5/GPT-4.
GPT3.5_4/Metamorphis_GPT3_5_and_GPT4.ipynb - GPT-3.5/GPT-4 notes · Experiment README and setup notes for the GPT workflow.
GPT3.5_4/README.md
Script experiments
Automated model adapters, runners, and scoring scripts.
- StarCoder runner · Scripted task execution for StarCoder2-7b.
starcoder2-7b/DeadCode.py - StarCoder scoring · Automated scoring for generated outputs.
starcoder2-7b/score.py - CodeGemma adapter · Batch prompting and task processing for CodeGemma/CodeLlama.
Codegemma_Codellama/core.py - Task prompts · Task-specific prompt definitions for the CodeGemma/CodeLlama workflow.
Codegemma_Codellama/task_prompts.py - LLaMA experiment · Scripted experiment runner for LLaMA 3.1.
Llama_3.1/Llama_Experiment.py
Other components
- Repository · MetamorphASM benchmark and model evaluation workspace.
README.md
Connections 20
- Repository → Dataset pipeline: hosts · Repository centers on the shared dataset-generation subsystem.
- Dataset pipeline → Assembly prep: prepares · Raw assembly is separated and linearized before transformation.
- Assembly prep → Obfuscation ops: feeds · Preprocessed assembly enters the obfuscation layer.
- Obfuscation ops → Dead code: includes · Dead-code insertion is one of the core benchmark transformations.
- Obfuscation ops → Register swap: includes · Register substitution is one of the core benchmark transformations.
- Obfuscation ops → Control flow: includes · Control-flow change is one of the core benchmark transformations.
- Dataset pipeline → Benchmark CSVs: produces · Dataset scripts emit benchmark-ready CSV artifacts.
- Benchmark CSVs → Few-shot pool: supports · Shared few-shot examples are used to prompt experiments.
- Few-shot pool → CodeT5 notebooks: consumed by · CodeT5 notebooks reuse shared prompt resources.
- Few-shot pool → GPT-4 notebooks: consumed by · GPT-3.5/GPT-4 notebooks reuse shared prompt resources.
- Few-shot pool → StarCoder runner: consumed by · StarCoder scripts use shared few-shot examples.
- Few-shot pool → CodeGemma adapter: consumed by · CodeGemma/CodeLlama adapters use shared few-shot examples.
- Few-shot pool → LLaMA experiment: consumed by · LLaMA experiments use shared few-shot examples.
- Benchmark CSVs → CodeT5 notebooks: evaluates · Notebook experiments consume the benchmark CSVs.
- Benchmark CSVs → StarCoder runner: evaluates · Scripted model adapters consume the benchmark CSVs.
- Benchmark CSVs → CodeGemma adapter: evaluates · Scripted model adapters consume the benchmark CSVs.
- Benchmark CSVs → LLaMA experiment: evaluates · Scripted model adapters consume the benchmark CSVs.
- StarCoder runner → StarCoder scoring: scores · Generated outputs are assessed by the StarCoder scoring stage.
- CodeGemma adapter → Task prompts: builds · Prompt templates are assembled inside the CodeGemma workflow.
- GPT-4 notebooks → GPT-3.5/GPT-4 notes: documents · README captures the GPT experiment setup and usage.