# aixcoder-plugin/aixcoder-7b architecture

> Architecture diagram of the GitHub repository aixcoder-plugin/aixcoder-7b, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/aixcoder-plugin/aixcoder-7b
- Repository: https://github.com/aixcoder-plugin/aixcoder-7b

## Overview

##### Project Type and Purpose
- This repository is an open-source code-centric large language model (LLM) implementation and toolkit for both training and inference of the aiXcoder-7B code model.
- It provides:
- Model definition modules (megatron_mini) with tensor-parallel support
- A lightweight Hugging Face–style inference wrapper (hf_mini)
- Fine-tuning integration via PEFT (finetune.py + requirements_peft.txt)
- Ready-to-run training and inference scripts (sess_megatron.py, sess_huggingface.py)
- Goals: pretrain / fine-tune on code datasets, perform fast inference, support quantization.

##### File Structure Analysis
- Top-level:
- aiXcoder_extended_dataset: JSONL files for training / evaluation
- assets: static assets (charts, language lists)
- finetune.py: entry point for PEFT-based fine-tuning
- sess_megatron.py: script for Megatron-style inference (tensor parallel)
- sess_huggingface.py: script for HF Transformers inference
- requirements.txt / requirements_peft.txt: dependencies
- Packages:
- megatron_mini: core model definition, parallel_state, tensor_parallel layers, transformer, llama_model
- hf_mini: minimal wrappers for data filtering, input formatting, and utility functions
- Config / License: Docker implied by README, Apache-2.0 license.

##### Key Components
A. Data Layer
- aiXcoder_extended_dataset
- External datasets (bigcode/the-stack, Hugging Face Hub)
- Preprocessing modules: hf_mini/filter.py, megatron_mini/filter.py
B. Model Definition Layer
- megatron_mini/core: parallel state manager, tensor_parallel utilities
- megatron_mini/model: transformer blocks, LLaMA model wrapper
- hf_mini/utils: input wrapper, tokenization helpers
C. Training & Fine-Tuning Layer
- finetune.py: orchestrates data loading, model loading (via Hugging Face + PEFT), FIM data construction
- requirements_peft.txt → PEFT, accelerate
D. Inference Layer
- sess_megatron.py: loads model_dir, uses megatron_mini initialize, runs TestInference
- sess_huggingface.py: loads from Hugging Face Hub, uses hf_mini.input_wrapper and transformers API
- bitsandbytes / flash_attention integration
E. External Services & Artifacts
- Model weights on Hugging Face
- Docker container (pytorch/pytorch:2.1.0-cuda…)
- FlashAttention, BitsAndBytes for quantization
- VS Code / JetBrains plugins (consumers of the model)

##### Relationships and Data Flow
1. Raw Data (aiXcoder_extended_dataset & external repos)
→ Filter & preprocess (filter.py modules)
→ Produce training / FIM batches.
2. Training Pipeline (finetune.py)
→ Initialize model via megatron_mini or HF + PEFT
→ Run structured FIM + autoregressive tasks
→ Output: fine-tuned model weights (local dir or HF Hub)
3. Inference Pipeline
- Megatron Path: sess_megatron.py
← model_dir → megatron_mini.initialize → TestInference.run_infer
- HF Path: sess_huggingface.py
← HF Hub → AutoModelForCausalLM + AutoTokenizer → hf_mini.input_wrapper → generate()
4. Quantization Option
→ BitsAndBytesConfig injection into HF load_pretrained → reduced memory footprint
5. Delivery & Consumption
→ Developers invoke CLI or Python API
→ VS Code / JetBrains plugin integrate API calls via sess_huggingface.py

##### Architectural Patterns & Principles
- Modular separation: data processing, model definition, training, inference
- Parallelism: tensor and pipeline parallel via megatron_mini
- FIM (Fill-In-The-Middle) structured pretraining → specialized data pipeline
- Extensibility: dual-backends (Megatron vs HF), quantization hooks (bitsandbytes)

##### Final Recommendations
- Essential modules and how they interact.
- Do not over-decompose; group related files under their package/component.
- Maintain clarity: each layer (Data, Model, Training, Inference, External) is visually distinct.
- Arrowheads indicate direction (e.g., Preprocess → Training; Weights → Inference).
- Technologies (PyTorch, Transformers, PEFT, Megatron, BitsAndBytes).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Layer
    subgraph "Data Layer" 
        direction TB
        D1[/"aiXcoder_extended_dataset"/]:::data
        D2(External Datasets):::data
        D3["hf_mini/filter.py"]:::data
        D4["megatron_mini/filter.py"]:::data
    end

    %% Model Definition Layer
    subgraph "Model Definition Layer"
        direction TB
        M1["megatron_mini/initialize.py"]:::model
        M2["parallel_state.py"]:::model
        subgraph "Tensor Parallel Utils"
            direction TB
            M3["layers.py"]:::model
            M4["mappings.py"]:::model
            M5["random.py"]:::model
            M6["utils.py"]:::model
        end
        M7["megatron_mini/core/utils.py"]:::model
        subgraph "Model Definitions"
            direction TB
            M8["transformer.py"]:::model
            M9["llama_model.py"]:::model
            M10["module.py"]:::model
        end
        M11["hf_mini/utils.py"]:::model
    end

    %% Training & Fine-Tuning Layer
    subgraph "Training & Fine-Tuning Layer"
        direction TB
        T1["finetune.py"]:::train
        T2[/requirements_peft.txt/]:::train
    end

    %% Inference Layer
    subgraph "Inference Layer"
        direction TB
        subgraph "Megatron Backend"
            direction TB
            I1["sess_megatron.py"]:::infer
            I2["megatron_mini/initialize.py"]:::infer
        end
        subgraph "Hugging Face Backend"
            direction TB
            I3["sess_huggingface.py"]:::infer
            I4["hf_mini/utils.py"]:::infer
            I5["hf_mini/filter.py"]:::infer
        end
    end

    %% External Services & Artifacts
    subgraph "External Services"
        direction TB
        E1((HF Hub)):::external
        E2[Docker Container]:::external
        E3[FlashAttention]:::external
        E4[BitsAndBytes]:::external
        E5[VS Code/JetBrains Plugins]:::external
    end

    %% Connections
    D1 -->|"raw JSONL"| D3
    D2 -->|"raw data"| D4
    D3 -->|"filtered JSONL"| T1
    D4 -->|"filtered JSONL"| T1

    T1 -->|"trained weights"| E1
    T1 -->|"trained weights"| I1
    T1 -->|"trained weights"| I3

    M1 --> M2
    M2 --> M3
    M3 --> M4
    M4 --> M5
    M5 --> M6
    M6 --> M7
    M7 --> M8
    M8 --> M9
    M9 --> M10

    I1 -->|"init model"| M1
    I3 -->|"load HF model"| I4
    I4 -->|"wrapped input"| I3
    I3 --> E5

    E1 -->|"weights download"| I3
    E2 -->|"runtime env"| I1
    E3 -->|"acceleration"| I1
    E4 -->|"quantization"| I3

    %% Click Events
    click D1 "https://github.com/aixcoder-plugin/aixcoder-7b/tree/main/aiXcoder_extended_dataset/"
    click D3 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/hf_mini/filter.py"
    click D4 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/filter.py"
    click M1 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/initialize.py"
    click M2 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/core/parallel_state.py"
    click M3 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/core/tensor_parallel/layers.py"
    click M4 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/core/tensor_parallel/mappings.py"
    click M5 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/core/tensor_parallel/random.py"
    click M6 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/core/tensor_parallel/utils.py"
    click M7 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/core/utils.py"
    click M8 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/model/transformer.py"
    click M9 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/model/llama_model.py"
    click M10 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/model/module.py"
    click M11 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/hf_mini/utils.py"
    click T1 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/finetune.py"
    click T2 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/requirements_peft.txt"
    click I1 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/sess_megatron.py"
    click I2 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/megatron_mini/initialize.py"
    click I3 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/sess_huggingface.py"
    click I4 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/hf_mini/utils.py"
    click I5 "https://github.com/aixcoder-plugin/aixcoder-7b/blob/main/hf_mini/filter.py"

    %% Styles
    classDef data fill:#CFE2F3,stroke:#2B6CB0,color:#1A365D
    classDef model fill:#D1E8D1,stroke:#2F855A,color:#22543D
    classDef train fill:#FEEBC8,stroke:#DD6B20,color:#7B341E
    classDef infer fill:#E9D8FD,stroke:#9F7AEA,color:#553C9A
    classDef external fill:#E2E8F0,stroke:#4A5568,color:#2D3748
```
