How jetbrains-research/coqpilot works
Project Type and Purpose
• Type: A full-fledged VS Code extension.
• Purpose: To automate the writing and refinement of Coq proofs by integrating LLM services (such as OpenAI, LM Studio, Grazie, and a predefined tactics service) with Coq’s tooling (via coq-lsp for typechecking and proof validation). It also includes benchmarking features to evaluate generated proofs.
Main Components
a. User Interface and VS Code Extension Layer:
– Contains the extension code (see src/extension), responsible for interacting with the user (commands like “perform completion under cursor,” selection-based completions, etc.), displaying logs, and managing settings.
– Handles VS Code–specific integration (e.g., UI components, status indicators, document editors).
b. Core Engine and Business Logic:
– Core proof generation logic (found in src/core and src/benchmark) that parses Coq files, finds “admit.” statements, and prepares contexts for LLM communication.
– Experimentation modules (under src/benchmark/framework/experiment) that manage multi-round fixing and iterative proof generation.
– Context theorems ranker (under src/core/contextTheoremRanker) selecting the relevant parts of a Coq file as context for LLM generation.
c. LLM Service Integration Modules:
– The LLM layer (located in src/llm and its subdirectories) encapsulates various services (OpenAI, LM Studio, Grazie, and a predefined tactics service).
– For each service, there are specific modules (e.g., src/llm/llmServices/openai, src/llm/llmServices/grazie) that handle the API integration, parameter resolution, and processing of generated completions.
– Model configuration settings and multi-round profiles (as detailed in the README) allow dynamic selection and ordering of models.
d. Coq-LSP Integration and Parsing:
– The integration with coq-lsp (see src/coqLsp) allows parsing of Coq source files and typechecking of generated proofs.
– There are dedicated parsers (in src/coqParser and parts of src/benchmark/framework/parseDataset) that process Coq files and extract proof obligations.
e. Subprocess and Utility Modules:
– Several utilities for subprocess management (in src/benchmark/framework/subprocessCalls) handle spawning Coq compilation processes, executing external commands, and managing inter-process communication.
– Generic utility libraries (in src/utils and under various “utils” directories in different modules) provide logging, file and JSON operations, error handling, and asynchronous scheduling.
f. Benchmarking and Reporting:
– Benchmark components (roots in src/benchmark and dataset directories) allow testing the proof generation performance, managing different datasets and experiments.
– Reporting modules (in src/benchmark/framework/reportBuilders) transform experiment outcomes into JSON or other report formats.
Relationships and Interactions
– The VS Code Extension (UI) accepts user commands (e.g., “prove all admits”), then channels these requests to the Core Engine.
– The Core Engine parses the active Coq file, extracts contexts, and dispatches individual proof generation tasks.
– Each generation task is forwarded to the LLM Integration Layer where the request is sent (potentially through multiple LLM services as fallback) and the service responses are validated.
– Coq proofs are verified using coq-lsp integration (often via subprocess calls) to ensure type-correctness.
– The benchmarking module may interact with both the core logic and LLM modules to collect performance data and generate reports.
– Utility modules (logging, async scheduling, file processing) support all other components with cross-cutting functionality.
Architectural Patterns and Design Principles
– Modular and layered architecture: Each set of functionalities (UI, core logic, LLM services, parsing, benchmarking) is well encapsulated.
– Separation of concerns: The user interface is separated from core proof-generation logic, which in turn is decoupled from LLM-specific integrations.
– Extensibility: The design supports adding or removing LLM services by configuring the associated models, as visible with multiple “models” arrays in the settings.
– Communication through well-defined interfaces: For example, internal protocols for IPC in subprocess calls are defined in src/benchmark/framework/utils/subprocessUtils/ipc.