How hongthanh909/pyvul-pilot works
This repository curates the PyVul vulnerability benchmark and evaluates machine-learning and rule-based vulnerability detectors. The main workflow gathers vulnerability reports and fixing commits, extracts changed functions, labels and filters function pairs, then uses benchmark data to train or evaluate detectors and compare findings with reported CWEs. The README documents additional collection and detector-running capabilities whose scripts are not present in the supplied tree; those are represented only where paths are supported. Source excerpts are partial, so uncertain internal wiring is omitted.
Main entry points: dataset PyVul Benchmark · clone_checkout_commit.py Commit Snapshots · openai_binary.py OpenAI Finetuning · check_codeql_results.py CodeQL Result Analysis
Components 26
Report Collection
- Git Advisory Fetch
benchmark_curation/data_collection/get_git_advisories.py - GitHub Advisory Sources
benchmark_curation/data_collection/get_github_adv_src.py - Huntr Spider
benchmark_curation/data_collection/huntr_spider.py - Snyk Spider
benchmark_curation/data_collection/snyk_spider_for_src.py - Commit Snapshots
benchmark_curation/data_collection/clone_checkout_commit.py - Function Extraction
benchmark_curation/data_collection/collect_functions_from_commits.py
Benchmark Curation
- Finetune Data
dataset/finetune_data - GPT Cleansing
benchmark_curation/data_cleansing/gpt_cleansing.py - GPT Filter Results
benchmark_curation/data_cleansing/read_chatgpt_filter_results.py - PyVul Benchmark
dataset
ML Evaluation
- OpenAI Finetuning
vd_evaluation/ml_vd_evaluation/finetune/openai_binary.py - CWE Finetuning
vd_evaluation/ml_vd_evaluation/finetune/openai_bycwe_binary.py - OpenAI Evaluation
vd_evaluation/ml_vd_evaluation/finetune/eval_openai.py - OpenAI Zero Shot
vd_evaluation/ml_vd_evaluation/zeroshot/zero_shot.py - CodeQwen Zero Shot
vd_evaluation/ml_vd_evaluation/zeroshot/zero_shot_codeqwen.py
Rule Evaluation
- CodeQL Result Analysis
vd_evaluation/rule_vd_evaluation/check_codeql_results.py - Pysa Result Analysis
vd_evaluation/rule_vd_evaluation/check_pysa_results.py - Bandit Result Analysis
vd_evaluation/rule_vd_evaluation/check_bandit_results.py
Other components
- Researcher
- Advisory Sources
- GitHub Repositories
- Huntr Reports
- Snyk Reports
- OpenAI API
- Local Model API
- Detector Outputs
Connections 33
- Advisory Sources → Git Advisory Fetch: provides reports
- Advisory Sources → GitHub Advisory Sources: provides advisories
- Huntr Reports → Huntr Spider: provides reports
- Snyk Reports → Snyk Spider: provides reports
- Git Advisory Fetch → Commit Snapshots: identifies fixes
- GitHub Advisory Sources → Commit Snapshots: identifies fixes
- Huntr Spider → Commit Snapshots: provides fix links
- Snyk Spider → Commit Snapshots: provides fix links
- GitHub Repositories → Commit Snapshots: provides commits
- Commit Snapshots → Function Extraction: checks out commits
- Function Extraction → Finetune Data: extracts function pairs
- Finetune Data → GPT Cleansing: supplies samples
- GPT Cleansing → GPT Filter Results: produces labels
- GPT Filter Results → PyVul Benchmark: filters and labels
- PyVul Benchmark → OpenAI Finetuning: supplies training data
- PyVul Benchmark → CWE Finetuning: supplies training data
- PyVul Benchmark → OpenAI Evaluation: supplies evaluation data
- PyVul Benchmark → OpenAI Zero Shot: supplies test data
- PyVul Benchmark → CodeQwen Zero Shot: supplies test data
- OpenAI Finetuning → OpenAI API: uploads and trains
- CWE Finetuning → OpenAI API: uploads and trains
- OpenAI Evaluation → OpenAI API: evaluates model
- OpenAI Zero Shot → OpenAI API: requests predictions
- CodeQwen Zero Shot → Local Model API: requests predictions
- Detector Outputs → CodeQL Result Analysis: reads findings
- Detector Outputs → Pysa Result Analysis: reads findings
- Detector Outputs → Bandit Result Analysis: reads findings
- PyVul Benchmark → CodeQL Result Analysis: provides CWE context
- PyVul Benchmark → Pysa Result Analysis: provides CWE context
- PyVul Benchmark → Bandit Result Analysis: provides CWE context
- Researcher → Commit Snapshots: reproduces snapshots
- Researcher → OpenAI Finetuning: runs experiments
- Researcher → CodeQL Result Analysis: reviews detector results