# hongthanh909/pyvul-pilot architecture

> Architecture diagram of the GitHub repository hongthanh909/pyvul-pilot, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hongthanh909/pyvul-pilot
- Repository: https://github.com/hongthanh909/pyvul-pilot

## Overview

This repository curates the PyVul vulnerability benchmark and evaluates machine-learning and rule-based vulnerability detectors. The main workflow gathers vulnerability reports and fixing commits, extracts changed functions, labels and filters function pairs, then uses benchmark data to train or evaluate detectors and compare findings with reported CWEs. The README documents additional collection and detector-running capabilities whose scripts are not present in the supplied tree; those are represented only where paths are supported. Source excerpts are partial, so uncertain internal wiring is omitted.

Main entry points: [`dataset`](https://github.com/hongthanh909/pyvul-pilot/tree/main/dataset) (PyVul Benchmark), [`benchmark_curation/data_collection/clone_checkout_commit.py`](https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/clone_checkout_commit.py) (Commit Snapshots), [`vd_evaluation/ml_vd_evaluation/finetune/openai_binary.py`](https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/finetune/openai_binary.py) (OpenAI Finetuning), [`vd_evaluation/rule_vd_evaluation/check_codeql_results.py`](https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/rule_vd_evaluation/check_codeql_results.py) (CodeQL Result Analysis)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_collection["Report Collection"]
  node_gitadvisory["Git Advisory Fetch"]
  node_githubsrc["GitHub Advisory Sources"]
  node_huntrspider["Huntr Spider<br/>[huntr_spider.py]"]
  node_snykspider["Snyk Spider"]
  node_snapshot["Commit Snapshots"]
  node_functionextract["Function Extraction"]
end

subgraph group_curation["Benchmark Curation"]
  node_finetunedata["Finetune Data"]
  node_gptclean["GPT Cleansing<br/>[gpt_cleansing.py]"]
  node_gptresults["GPT Filter Results"]
  node_benchmark["PyVul Benchmark"]
end

subgraph group_ml["ML Evaluation"]
  node_fineopenai["OpenAI Finetuning<br/>[openai_binary.py]"]
  node_cweopenai["CWE Finetuning"]
  node_evalopenai["OpenAI Evaluation<br/>[eval_openai.py]"]
  node_zeroshot["OpenAI Zero Shot<br/>[zero_shot.py]"]
  node_codeqwen["CodeQwen Zero Shot"]
end

subgraph group_rules["Rule Evaluation"]
  node_codeqlcheck["CodeQL Result Analysis"]
  node_pysacheck["Pysa Result Analysis"]
  node_banditcheck["Bandit Result Analysis"]
end

node_researcher(("Researcher"))
node_advisories(("Advisory Sources"))
node_github(("GitHub Repositories"))
node_huntr(("Huntr Reports"))
node_snyk(("Snyk Reports"))
node_openai{{"OpenAI API"}}
node_ollama{{"Local Model API"}}
node_ruleoutputs["Detector Outputs"]

node_advisories -->|"provides reports"| node_gitadvisory
node_advisories -->|"provides advisories"| node_githubsrc
node_huntr -->|"provides reports"| node_huntrspider
node_snyk -->|"provides reports"| node_snykspider
node_gitadvisory -->|"identifies fixes"| node_snapshot
node_githubsrc -->|"identifies fixes"| node_snapshot
node_huntrspider -->|"provides fix links"| node_snapshot
node_snykspider -->|"provides fix links"| node_snapshot
node_github -->|"provides commits"| node_snapshot
node_snapshot -->|"checks out commits"| node_functionextract
node_functionextract -->|"extracts function pairs"| node_finetunedata
node_finetunedata -->|"supplies samples"| node_gptclean
node_gptclean -->|"produces labels"| node_gptresults
node_gptresults -->|"filters and labels"| node_benchmark
node_benchmark -->|"supplies training data"| node_fineopenai
node_benchmark -->|"supplies training data"| node_cweopenai
node_benchmark -->|"supplies evaluation data"| node_evalopenai
node_benchmark -->|"supplies test data"| node_zeroshot
node_benchmark -->|"supplies test data"| node_codeqwen
node_fineopenai -.->|"uploads and trains"| node_openai
node_cweopenai -.->|"uploads and trains"| node_openai
node_evalopenai -.->|"evaluates model"| node_openai
node_zeroshot -.->|"requests predictions"| node_openai
node_codeqwen -.->|"requests predictions"| node_ollama
node_ruleoutputs -->|"reads findings"| node_codeqlcheck
node_ruleoutputs -->|"reads findings"| node_pysacheck
node_ruleoutputs -->|"reads findings"| node_banditcheck
node_benchmark -->|"provides CWE context"| node_codeqlcheck
node_benchmark -->|"provides CWE context"| node_pysacheck
node_benchmark -->|"provides CWE context"| node_banditcheck
node_researcher -->|"reproduces snapshots"| node_snapshot
node_researcher -->|"runs experiments"| node_fineopenai
node_researcher -->|"reviews detector results"| node_codeqlcheck

click node_gitadvisory "https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/get_git_advisories.py"
click node_githubsrc "https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/get_github_adv_src.py"
click node_huntrspider "https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/huntr_spider.py"
click node_snykspider "https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/snyk_spider_for_src.py"
click node_snapshot "https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/clone_checkout_commit.py"
click node_functionextract "https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/collect_functions_from_commits.py"
click node_finetunedata "https://github.com/hongthanh909/pyvul-pilot/tree/main/dataset/finetune_data"
click node_gptclean "https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_cleansing/gpt_cleansing.py"
click node_gptresults "https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_cleansing/read_chatgpt_filter_results.py"
click node_benchmark "https://github.com/hongthanh909/pyvul-pilot/tree/main/dataset"
click node_fineopenai "https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/finetune/openai_binary.py"
click node_cweopenai "https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/finetune/openai_bycwe_binary.py"
click node_evalopenai "https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/finetune/eval_openai.py"
click node_zeroshot "https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/zeroshot/zero_shot.py"
click node_codeqwen "https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/zeroshot/zero_shot_codeqwen.py"
click node_codeqlcheck "https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/rule_vd_evaluation/check_codeql_results.py"
click node_pysacheck "https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/rule_vd_evaluation/check_pysa_results.py"
click node_banditcheck "https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/rule_vd_evaluation/check_bandit_results.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_gitadvisory,node_githubsrc,node_huntrspider,node_snykspider,node_snapshot,node_functionextract toneBlue
class node_finetunedata,node_gptclean,node_gptresults,node_benchmark toneAmber
class node_fineopenai,node_cweopenai,node_evalopenai,node_zeroshot,node_codeqwen,node_openai,node_ollama toneMint
class node_codeqlcheck,node_pysacheck,node_banditcheck toneRose
class node_researcher,node_advisories,node_github,node_huntr,node_snyk,node_ruleoutputs toneIndigo
```

## Components

### Report Collection

- [Git Advisory Fetch](https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/get_git_advisories.py): `benchmark_curation/data_collection/get_git_advisories.py`
- [GitHub Advisory Sources](https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/get_github_adv_src.py): `benchmark_curation/data_collection/get_github_adv_src.py`
- [Huntr Spider](https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/huntr_spider.py): `benchmark_curation/data_collection/huntr_spider.py`
- [Snyk Spider](https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/snyk_spider_for_src.py): `benchmark_curation/data_collection/snyk_spider_for_src.py`
- [Commit Snapshots](https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/clone_checkout_commit.py): `benchmark_curation/data_collection/clone_checkout_commit.py`
- [Function Extraction](https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_collection/collect_functions_from_commits.py): `benchmark_curation/data_collection/collect_functions_from_commits.py`

### Benchmark Curation

- [Finetune Data](https://github.com/hongthanh909/pyvul-pilot/tree/main/dataset/finetune_data): `dataset/finetune_data`
- [GPT Cleansing](https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_cleansing/gpt_cleansing.py): `benchmark_curation/data_cleansing/gpt_cleansing.py`
- [GPT Filter Results](https://github.com/hongthanh909/pyvul-pilot/blob/main/benchmark_curation/data_cleansing/read_chatgpt_filter_results.py): `benchmark_curation/data_cleansing/read_chatgpt_filter_results.py`
- [PyVul Benchmark](https://github.com/hongthanh909/pyvul-pilot/tree/main/dataset): `dataset`

### ML Evaluation

- [OpenAI Finetuning](https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/finetune/openai_binary.py): `vd_evaluation/ml_vd_evaluation/finetune/openai_binary.py`
- [CWE Finetuning](https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/finetune/openai_bycwe_binary.py): `vd_evaluation/ml_vd_evaluation/finetune/openai_bycwe_binary.py`
- [OpenAI Evaluation](https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/finetune/eval_openai.py): `vd_evaluation/ml_vd_evaluation/finetune/eval_openai.py`
- [OpenAI Zero Shot](https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/zeroshot/zero_shot.py): `vd_evaluation/ml_vd_evaluation/zeroshot/zero_shot.py`
- [CodeQwen Zero Shot](https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/ml_vd_evaluation/zeroshot/zero_shot_codeqwen.py): `vd_evaluation/ml_vd_evaluation/zeroshot/zero_shot_codeqwen.py`

### Rule Evaluation

- [CodeQL Result Analysis](https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/rule_vd_evaluation/check_codeql_results.py): `vd_evaluation/rule_vd_evaluation/check_codeql_results.py`
- [Pysa Result Analysis](https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/rule_vd_evaluation/check_pysa_results.py): `vd_evaluation/rule_vd_evaluation/check_pysa_results.py`
- [Bandit Result Analysis](https://github.com/hongthanh909/pyvul-pilot/blob/main/vd_evaluation/rule_vd_evaluation/check_bandit_results.py): `vd_evaluation/rule_vd_evaluation/check_bandit_results.py`

### Other components

- **Researcher**
- **Advisory Sources**
- **GitHub Repositories**
- **Huntr Reports**
- **Snyk Reports**
- **OpenAI API**
- **Local Model API**
- **Detector Outputs**

## Connections

- Advisory Sources → Git Advisory Fetch: provides reports
- Advisory Sources → GitHub Advisory Sources: provides advisories
- Huntr Reports → Huntr Spider: provides reports
- Snyk Reports → Snyk Spider: provides reports
- Git Advisory Fetch → Commit Snapshots: identifies fixes
- GitHub Advisory Sources → Commit Snapshots: identifies fixes
- Huntr Spider → Commit Snapshots: provides fix links
- Snyk Spider → Commit Snapshots: provides fix links
- GitHub Repositories → Commit Snapshots: provides commits
- Commit Snapshots → Function Extraction: checks out commits
- Function Extraction → Finetune Data: extracts function pairs
- Finetune Data → GPT Cleansing: supplies samples
- GPT Cleansing → GPT Filter Results: produces labels
- GPT Filter Results → PyVul Benchmark: filters and labels
- PyVul Benchmark → OpenAI Finetuning: supplies training data
- PyVul Benchmark → CWE Finetuning: supplies training data
- PyVul Benchmark → OpenAI Evaluation: supplies evaluation data
- PyVul Benchmark → OpenAI Zero Shot: supplies test data
- PyVul Benchmark → CodeQwen Zero Shot: supplies test data
- OpenAI Finetuning → OpenAI API: uploads and trains
- CWE Finetuning → OpenAI API: uploads and trains
- OpenAI Evaluation → OpenAI API: evaluates model
- OpenAI Zero Shot → OpenAI API: requests predictions
- CodeQwen Zero Shot → Local Model API: requests predictions
- Detector Outputs → CodeQL Result Analysis: reads findings
- Detector Outputs → Pysa Result Analysis: reads findings
- Detector Outputs → Bandit Result Analysis: reads findings
- PyVul Benchmark → CodeQL Result Analysis: provides CWE context
- PyVul Benchmark → Pysa Result Analysis: provides CWE context
- PyVul Benchmark → Bandit Result Analysis: provides CWE context
- Researcher → Commit Snapshots: reproduces snapshots
- Researcher → OpenAI Finetuning: runs experiments
- Researcher → CodeQL Result Analysis: reviews detector results