How hkust-nlp/codeio works
Project Type and Purpose
• This project implements a multi-stage pipeline that pre-processes raw code data, builds structured messages, performs inference via APIs (e.g., deepseek-chat, GPT series models), parses and verifies generated outputs, and finally assembles a dataset in the desired format.
• As indicated in the README and file structure, the goal is to transform code reasoning into natural langauge through a carefully orchestrated series of operations.
File Structure
• Top-Level Directories:
• data: Contains the raw input files (e.g., rawcode_1k.jsonl and the related raw code messages).
• processed_data: Hosts pre-processed versions of the dataset at various stages (e.g., unified code format, parsed outputs, verified inference results, and the final demo data).
• scripts: Contains the pipeline_check.sh bash script for verification steps.
• src: Contains all core Python scripts that execute each stage of the pipeline:
• build_transform_msg.py – Converts raw code to message format.
• batched_api_inference.py – Performs batched API calls for inference.
• parse_gen_ios.py – Parses outputs to generate I/O pairs.
• build_codeio_msg.py and build_codeio_rev_msg.py – Constructs messages for both initial inference and revision steps.
• assemble_codeio_demo.py – Aggregates and finalizes the output data.
• Additional utility scripts (e.g., codeio_utils.py, utils.py) provide auxiliary functionalities.
• Configuration and Environment:
• Files like requirements.txt and environment.yaml indicate dependency management.
• The existence of multiple JSONL files across directories shows how intermediate results are stored.
Incorporate README Insights
• The README provides a comprehensive guide to the data processing pipeline, including steps for data preparation, inference, multi-turn revision (CodeI/O++), and verification.
• The instructions detail external dependencies like different models (deepseek-chat, GPT variants) that are invoked via API calls.
a. Main Components to Include:
• Input Data Source (raw code files from the data directory).
• Message Builder (build_transform_msg.py).
• Batched Inference Module (batched_api_inference.py) with external API integrations (DeepSeek API, GPT models, etc.).
• Code Parsing and I/O Pair Generation (parse_gen_ios.py, build_codeio_msg.py).
• Verification Process (pipeline_check.sh script).
• Revision Module for multi-turn operations (build_codeio_rev_msg.py and a second round of batched inference).
• Final Assembly (assemble_codeio_demo.py) producing the final data set.
• Supporting Utilities (codeio_utils.py, utils.py).
b. Relationships and Data Flow:
• Data flows sequentially from raw input files → message building → inference → parsing → I/O pair extraction → verification → revision (if needed) → final data assembly.
• Interactions with external APIs (e.g., DeepSeek) midway in the pipeline.
c. Architectural Patterns and Design Principles:
• Modular design: Each pipeline stage is handled by a dedicated script, promoting separation of concerns.
• Batch processing and parallel processing: Notice the use of parameters like “--num_process” and “--num_thread” indicating parallel execution.
• Verification loop: The design includes a built-in validation stage (pipeline_check.sh) that re-checks inference outputs.
• Multi-turn iteration: The revision phase demonstrates a feedback loop to refine outputs (CodeI/O++).
d. Technologies, Frameworks, and Libraries:
• Python is the primary language, with environment dependencies defined via requirements.txt/environment.yaml.
• External API calls (DeepSeek and GPT-style APIs) and potential integration with local inference frameworks like vllm or sglang.
• Shell scripting for process management and verification (pipeline_check.sh).
Final Advice
• Script’s role and show how intermediate data files (from the data and processed_data directories) move between these stages.
• Remember that the objective is to reflect the linear (and iterative) processing nature of the data pipeline, from raw data ingestion to the final assembled dataset.
• Adding notes on areas where external dependencies or potential scalability concerns (parallel processing parameters) are relevant.