How eshe0922/reposvul works
Based on our analysis of the provided file tree and README, this project—ReposVul—is a data collection framework for repository-level vulnerability detection and analysis. It integrates multiple modules that (a) crawl raw data from different sources, (b) untangle vulnerability-fixing patches using both LLMs and static analysis tools, (c) extract multi-granularity dependency information from codebases (e.g., inter-procedural call relationships), and (d) apply trace-based filtering to ensure that only up-to-date patches are included in the final dataset.
Main Components
• Raw Data Crawling Module:
• This part is responsible for crawling vulnerability entries and patches.
• Within the “Raw_Data_Crawling” directory, note the presence of subdirectories for different platforms (android, chrome, github). Each has its own run scripts or merge logic.
• Vulnerability Untangling Module:
• Found under the “Vulnerability_Untangling_Module” folder.
• It has two subcomponents:
▪ LLM-based analysis (under the “llm” subdirectory, including a CSV file and evaluation script).
▪ Static Analysis (under the “static” subdirectory with various scripts for tools like cppcheck, flawfinder, rats, and semgrep).
• Multi-granularity Dependency Extraction Module:
• Located in “Multi-granularity_Dependency_Extraction_Module”.
• The Python files (with naming patterns “process_*” and “parse_getout_nearfunc_*”) indicate that this module handles dependency parsing/extraction for different programming languages (C, C++, Python, and their “plus” variants).
• Trace-Based Filtering Module:
• Found in “Trace-based_Filtering_Module”.
• Contains subdirectories (android, chrome, github) with a “window.py” script in each, suggesting that commit trace information (e.g., file path trace or commit history) is used to filter outdated patches.
Relationships and Interactions
• Data Flow:
• The process likely begins with the Raw Data Crawling Module collecting raw patches and vulnerability data.
• This raw data then feeds into the Vulnerability Untangling Module where both static analysis and LLM decisions help filter and label vulnerability-related changes.
• The Multi-granularity Dependency Extraction Module then analyzes the patched code to extract relationships at various granularity levels (repository, file, function, and line levels).
• Finally, the Trace-based Filtering Module examines commit history and file path traces to weed out outdated patches.
• Interactions:
• There are clearly defined “internal” data exchanges between submodules (e.g., vulnerability metadata produced by the untangling module is used by the dependency extraction module).
Architectural Patterns and Key Technologies
• The project adopts a pipeline architecture where each module represents a processing stage.
• The design is modular and language-agnostic since modules (especially in dependency extraction) support multiple programming languages.
• Integration of external tools (static analyzers like cppcheck, flawfinder, etc.) and Large Language Models for contextual decisions is a central architectural design choice.
• Multi-granularity Dependency Extraction Module (indicate distinct processes for each language and mark “process” and “parse” functions)
• Trace-Based Filtering Module (each platform’s filtering process, e.g., android window, chrome window, github window)
• Rectangles for processing modules,
• Diamonds for decision or filtering stages,