# salesforce/codegen architecture

> Architecture diagram of the GitHub repository salesforce/codegen, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-10-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/salesforce/codegen
- Repository: https://github.com/salesforce/codegen

## Overview

CodeGen is a family of pretrained code-generation language models, released across three generations. The principal documented workflow is to load a checkpoint and tokenizer, encode a prompt, generate tokens, and decode a completion; the CodeGen1 sample script additionally truncates and prints the result. The graph also preserves the documented CodeGen2 infill capability and the CodeGen1 training example. CodeGen2 and CodeGen2.5 implementation details are not present in the sampled source, so their links are limited to README-documented relationships.

Main entry points: [`codegen1/jaxformer/hf/sample.py`](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/sample.py) (Sampling CLI), [`codegen1/jaxformer/hf/codegen/modeling_codegen.py`](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py) (Transformer block), [`codegen2`](https://github.com/salesforce/codegen/tree/main/codegen2) (CodeGen2 checkpoints), [`codegen1`](https://github.com/salesforce/codegen/tree/main/codegen1) (CodeGen1 checkpoints)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_usage["Generation workflow"]
  node_sample["Sampling CLI<br/>[sample.py]"]
  node_tokenizer["GPT-2 tokenizer<br/>[sample.py]"]
  node_completion["Decoded completion<br/>[sample.py]"]
end

subgraph group_model["Model runtime"]
  node_config["Model configuration"]
  node_attention["Causal attention"]
  node_mlp["Transformer MLP"]
  node_block["Transformer block"]
  node_transformer["CodeGen transformer"]
  node_lm["Causal LM head"]
end

subgraph group_releases["Model releases and training"]
  node_codegen1[("CodeGen1 checkpoints")]
  node_codegen2[("CodeGen2 checkpoints")]
  node_codegen25[("CodeGen2.5 checkpoints")]
  node_training["DeepSpeed training example<br/>[train_deepspeed.py]"]
end

node_user(("Model user"))

node_user -->|"provides prompt"| node_sample
node_sample -->|"encodes prompt"| node_tokenizer
node_sample -->|"generates tokens"| node_lm
node_sample -->|"decodes tokens"| node_tokenizer
node_sample -->|"truncates and prints"| node_completion
node_config -->|"configures"| node_transformer
node_block -->|"invokes"| node_attention
node_block -->|"invokes"| node_mlp
node_transformer -->|"runs layers"| node_block
node_lm -->|"runs transformer"| node_transformer
node_sample -->|"loads checkpoint"| node_codegen1
node_user -->|"loads and generates"| node_codegen2
node_user -->|"loads and generates"| node_codegen25
node_codegen2 -->|"supports infill generation"| node_completion
node_training -->|"loads model"| node_codegen1

click node_sample "https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/sample.py"
click node_tokenizer "https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/sample.py"
click node_completion "https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/sample.py"
click node_config "https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/configuration_codegen.py"
click node_attention "https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py"
click node_mlp "https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py"
click node_block "https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py"
click node_transformer "https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py"
click node_lm "https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py"
click node_codegen1 "https://github.com/salesforce/codegen/tree/main/codegen1"
click node_codegen2 "https://github.com/salesforce/codegen/tree/main/codegen2"
click node_codegen25 "https://github.com/salesforce/codegen/tree/main/codegen25"
click node_training "https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/train_deepspeed.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_sample,node_tokenizer,node_completion,node_user toneBlue
class node_config,node_attention,node_mlp,node_block,node_transformer,node_lm toneAmber
class node_codegen1,node_codegen2,node_codegen25,node_training toneMint
```

## Components

### Generation workflow

- [Sampling CLI](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/sample.py): `codegen1/jaxformer/hf/sample.py`
- [GPT-2 tokenizer](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/sample.py): `codegen1/jaxformer/hf/sample.py`
- [Decoded completion](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/sample.py): `codegen1/jaxformer/hf/sample.py`

### Model runtime

- [Model configuration](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/configuration_codegen.py): `codegen1/jaxformer/hf/codegen/configuration_codegen.py`
- [Causal attention](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py): `codegen1/jaxformer/hf/codegen/modeling_codegen.py`
- [Transformer MLP](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py): `codegen1/jaxformer/hf/codegen/modeling_codegen.py`
- [Transformer block](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py): `codegen1/jaxformer/hf/codegen/modeling_codegen.py`
- [CodeGen transformer](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py): `codegen1/jaxformer/hf/codegen/modeling_codegen.py`
- [Causal LM head](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/codegen/modeling_codegen.py): `codegen1/jaxformer/hf/codegen/modeling_codegen.py`

### Model releases and training

- [CodeGen1 checkpoints](https://github.com/salesforce/codegen/tree/main/codegen1): `codegen1`
- [CodeGen2 checkpoints](https://github.com/salesforce/codegen/tree/main/codegen2): `codegen2`
- [CodeGen2.5 checkpoints](https://github.com/salesforce/codegen/tree/main/codegen25): `codegen25`
- [DeepSpeed training example](https://github.com/salesforce/codegen/blob/main/codegen1/jaxformer/hf/train_deepspeed.py): `codegen1/jaxformer/hf/train_deepspeed.py`

### Other components

- **Model user**

## Connections

- Model user → Sampling CLI: provides prompt (evidence: `codegen1/jaxformer/hf/sample.py`)
- Sampling CLI → GPT-2 tokenizer: encodes prompt (evidence: `codegen1/jaxformer/hf/sample.py`)
- Sampling CLI → Causal LM head: generates tokens (evidence: `codegen1/jaxformer/hf/sample.py`)
- Sampling CLI → GPT-2 tokenizer: decodes tokens (evidence: `codegen1/jaxformer/hf/sample.py`)
- Sampling CLI → Decoded completion: truncates and prints (evidence: `codegen1/jaxformer/hf/sample.py`)
- Model configuration → CodeGen transformer: configures (evidence: `codegen1/jaxformer/hf/codegen/modeling_codegen.py`)
- Transformer block → Causal attention: invokes (evidence: `codegen1/jaxformer/hf/codegen/modeling_codegen.py`)
- Transformer block → Transformer MLP: invokes (evidence: `codegen1/jaxformer/hf/codegen/modeling_codegen.py`)
- CodeGen transformer → Transformer block: runs layers (evidence: `codegen1/jaxformer/hf/codegen/modeling_codegen.py`)
- Causal LM head → CodeGen transformer: runs transformer (evidence: `codegen1/jaxformer/hf/codegen/modeling_codegen.py`)
- Sampling CLI → CodeGen1 checkpoints: loads checkpoint (evidence: `codegen1/jaxformer/hf/sample.py`)
- Model user → CodeGen2 checkpoints: loads and generates (evidence: `README.md`)
- Model user → CodeGen2.5 checkpoints: loads and generates (evidence: `README.md`)
- CodeGen2 checkpoints → Decoded completion: supports infill generation (evidence: `README.md`)
- DeepSpeed training example → CodeGen1 checkpoints: loads model (evidence: `codegen1/jaxformer/hf/train_deepspeed.py`)
