# hkust-nlp/codeio architecture

> Architecture diagram of the GitHub repository hkust-nlp/codeio, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/hkust-nlp/codeio
- Repository: https://github.com/hkust-nlp/codeio

## Overview

##### Project Type and Purpose
- This project implements a multi-stage pipeline that pre-processes raw code data, builds structured messages, performs inference via APIs (e.g., deepseek-chat, GPT series models), parses and verifies generated outputs, and finally assembles a dataset in the desired format.
- As indicated in the README and file structure, the goal is to transform code reasoning into natural langauge through a carefully orchestrated series of operations.

##### File Structure
- Top-Level Directories:
- data: Contains the raw input files (e.g., rawcode_1k.jsonl and the related raw code messages).
- processed_data: Hosts pre-processed versions of the dataset at various stages (e.g., unified code format, parsed outputs, verified inference results, and the final demo data).
- scripts: Contains the pipeline_check.sh bash script for verification steps.
- src: Contains all core Python scripts that execute each stage of the pipeline:
- build_transform_msg.py – Converts raw code to message format.
- batched_api_inference.py – Performs batched API calls for inference.
- parse_gen_ios.py – Parses outputs to generate I/O pairs.
- build_codeio_msg.py and build_codeio_rev_msg.py – Constructs messages for both initial inference and revision steps.
- assemble_codeio_demo.py – Aggregates and finalizes the output data.
- Additional utility scripts (e.g., codeio_utils.py, utils.py) provide auxiliary functionalities.
- Configuration and Environment:
- Files like requirements.txt and environment.yaml indicate dependency management.
- The existence of multiple JSONL files across directories shows how intermediate results are stored.

##### Incorporate README Insights
- The README provides a comprehensive guide to the data processing pipeline, including steps for data preparation, inference, multi-turn revision (CodeI/O++), and verification.
- The instructions detail external dependencies like different models (deepseek-chat, GPT variants) that are invoked via API calls.

a. Main Components to Include:
- Input Data Source (raw code files from the data directory).
- Message Builder (build_transform_msg.py).
- Batched Inference Module (batched_api_inference.py) with external API integrations (DeepSeek API, GPT models, etc.).
- Code Parsing and I/O Pair Generation (parse_gen_ios.py, build_codeio_msg.py).
- Verification Process (pipeline_check.sh script).
- Revision Module for multi-turn operations (build_codeio_rev_msg.py and a second round of batched inference).
- Final Assembly (assemble_codeio_demo.py) producing the final data set.
- Supporting Utilities (codeio_utils.py, utils.py).

b. Relationships and Data Flow:
- Data flows sequentially from raw input files → message building → inference → parsing → I/O pair extraction → verification → revision (if needed) → final data assembly.
- Interactions with external APIs (e.g., DeepSeek) midway in the pipeline.

c. Architectural Patterns and Design Principles:
- Modular design: Each pipeline stage is handled by a dedicated script, promoting separation of concerns.
- Batch processing and parallel processing: Notice the use of parameters like “--num_process” and “--num_thread” indicating parallel execution.
- Verification loop: The design includes a built-in validation stage (pipeline_check.sh) that re-checks inference outputs.
- Multi-turn iteration: The revision phase demonstrates a feedback loop to refine outputs (CodeI/O++).

d. Technologies, Frameworks, and Libraries:
- Python is the primary language, with environment dependencies defined via requirements.txt/environment.yaml.
- External API calls (DeepSeek and GPT-style APIs) and potential integration with local inference frameworks like vllm or sglang.
- Shell scripting for process management and verification (pipeline_check.sh).

##### Final Advice
- Script’s role and show how intermediate data files (from the data and processed_data directories) move between these stages.
- Remember that the objective is to reflect the linear (and iterative) processing nature of the data pipeline, from raw data ingestion to the final assembled dataset.
- Adding notes on areas where external dependencies or potential scalability concerns (parallel processing parameters) are relevant.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Main Pipeline Nodes
    A["Data Input (rawcode_1k.jsonl)"]:::input
    B["Message Builder (build_transform_msg.py)"]:::process
    C["Batched Inference (batched_api_inference.py)"]:::process

    subgraph "Parsing & I/O Generation"
        D["Parsing (parse_gen_ios.py)"]:::process
        E["I/O Generation (build_codeio_msg.py)"]:::process
    end

    F["Verification (pipeline_check.sh)"]:::verification

    subgraph "Revision Stage"
        G["Revision Module (build_codeio_rev_msg.py)"]:::revision
        H["Batched Inference (Revision Round)"]:::process
    end

    I["Final Assembly (assemble_codeio_demo.py)"]:::final

    %% Supporting Utilities as separate nodes
    J["Supporting Utility: codeio_utils.py"]:::utility
    K["Supporting Utility: utils.py"]:::utility

    %% Main Process Flow
    A --> B
    B --> C
    C --> D
    D --> E
    E --> F
    F --> G
    G --> H
    H --> I

    %% Dotted links from Supporting Utilities to all main nodes
    J --- A
    J --- B
    J --- C
    J --- D
    J --- E
    J --- F
    J --- G
    J --- H
    J --- I

    K --- A
    K --- B
    K --- C
    K --- D
    K --- E
    K --- F
    K --- G
    K --- H
    K --- I

    %% Click Events
    click A "https://github.com/hkust-nlp/codeio/blob/master/data/rawcode_1k.jsonl"
    click B "https://github.com/hkust-nlp/codeio/blob/master/src/build_transform_msg.py"
    click C "https://github.com/hkust-nlp/codeio/blob/master/src/batched_api_inference.py"
    click D "https://github.com/hkust-nlp/codeio/blob/master/src/parse_gen_ios.py"
    click E "https://github.com/hkust-nlp/codeio/blob/master/src/build_codeio_msg.py"
    click F "https://github.com/hkust-nlp/codeio/blob/master/scripts/pipeline_check.sh"
    click G "https://github.com/hkust-nlp/codeio/blob/master/src/build_codeio_rev_msg.py"
    click I "https://github.com/hkust-nlp/codeio/blob/master/src/assemble_codeio_demo.py"
    click J "https://github.com/hkust-nlp/codeio/blob/master/src/codeio_utils.py"
    click K "https://github.com/hkust-nlp/codeio/blob/master/src/utils.py"

    %% Styles
    classDef input fill:#f9e79f,stroke:#d35400,stroke-width:2px;
    classDef process fill:#aed6f1,stroke:#2874a6,stroke-width:2px;
    classDef verification fill:#f5b7b1,stroke:#c0392b,stroke-width:2px;
    classDef revision fill:#d7bde2,stroke:#8e44ad,stroke-width:2px;
    classDef final fill:#abebc6,stroke:#27ae60,stroke-width:2px;
    classDef utility fill:#e8daef,stroke:#8e44ad,stroke-width:2px;
```
