How microsoft/monitors4codegen works
Project Type and Purpose
• This is a research-oriented Python library and artifact repository for “Monitor-Guided Decoding (MGD)” of code language models, combining static analysis (via LSP) with code generation.
• It is not a full-stack web app, but rather a modular toolkit and evaluation harness:
• Core Python package (monitors4codegen) containing MGD orchestrator, monitor implementations, and LSP client wrapper (multilspy).
• Datasets (Java repos and prompts).
• Evaluation scripts to score generated code and produce tables/figures.
A. Datasets & Artifacts
• datasets/PragmaticCode (JSON of repo file contents, CSV of repos)
• datasets/DotPrompts (CSV of prompt examples)
• inference_results (model outputs CSV)
B. Core Library (src/monitors4codegen)
1. MGD Orchestrator
• hf_gen.py: integrates with HuggingFace models as a LogitsProcessor
• openai_gen.py: wraps OpenAI API calls under MGD
2. Monitor Interface (monitor.py) & Implementations (monitors/*.py)
• Dereferences, NumArgs, Typestate, Switch-Enum, ClassInstantiation, JointMonitors
3. LSP Client Layer (multilspy)
• LanguageServer / SyncLanguageServer classes, protocol handlers, configs
• Language-specific adapters (Eclipse JDTLS, Jedi, Omnisharp, rust-analyzer directories)
C. External Services & Tools
• HuggingFace Transformers models
• OpenAI API
• Language Servers (Java, Python, C#, Rust)
D. Evaluation Pipeline
• evaluation_scripts/eval_results.py and eval_utils.py
• Input: inference_results/*.csv + datasets/PragmaticCode/fileContentsByRepo.json
• Output: metrics (CR, NIM, ISM, PM), figures, Report.md
E. Results & Reports
• results and results_sample directories (CSV/MD tables, figure PNGs)
Relationships & Data Flow
• Static Analysis Flow:
1. MGD Orchestrator → calls multilspy → starts Language Server → issues LSP requests (completion, hover, definitions) → receives type/symbol info → Monitor uses this to build token masks
2. MGD Orchestrator → injects masks into LM generation (HF or OpenAI) → produces code output
• Evaluation Flow:
1. Read inference CSV → for each sample, compute metrics using eval_utils.py (compilation check uses repo context JSON) → generate plots (matplotlib/Plotly) → dump tables & figures → assemble Report.md
• Artifact Flow:
• Datasets feed both the MGD process (for context) and evaluation scripts.
• Inference results are consumed by evaluation scripts to reproduce paper figures.
Architectural Patterns & Design Principles
• Modular Separation:
• Core MGD logic decoupled from LM backends (HF vs OpenAI) via adapter classes.
• Monitor implementations adhere to a common interface (Monitor).
• LSP communications abstracted into multilspy.
• Pipeline/Orchestration: Data-driven pipeline connecting static analysis → decoding → evaluation.
• Extensibility: New monitors or language servers can be plugged in by adding new classes under monitors or language_servers.