# neuphonic/neucodec architecture

> Architecture diagram of the GitHub repository neuphonic/neucodec, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/neuphonic/neucodec
- Repository: https://github.com/neuphonic/neucodec

## Overview

##### Project type and purpose
- This is a Python library and model repository (“neucodec”) implementing a lightweight neural audio codec.
- It provides encode/decode APIs for speech at 0.8 kbps, an ONNX export pathway, and integration with Speech-LM workflows.

##### File structure
- Top‐level packaging and CI:
- .github/workflows/publish.yaml (CI to publish pypi package)
- pyproject.toml (build metadata)
- LICENSE, README.md, .gitignore, pre-commit config
- Core library under neucodec/:
- activations.py, model.py, module.py (defines network blocks)
- codec_encoder*.py (BigCodec-style audio encoder, distillation variant)
- codec_decoder_vocos.py (upsampling decoder to 24 kHz)
- distill_layers.py (teacher-student compression layers)
- Subpackages for specialized processing:
- alias_free_torch/ (anti-alias filters, resampling)
- tconv/ (transposed-conv upsampling blocks)
- ONNX support under onnx/:
- export_onnx.py (conversion script)
- onnx_ops.py (custom ops or wrappers)
- Tests under tests/ for model‐level validation.

##### README for additional insights
- Describes dual encoder strategy (audio + semantic via Wav2Vec2-BERT)
- FSQ quantisation component producing token vectors
- Upsampling decoder raising sample rate 16→24 kHz
- External dependencies: PyTorch, torchaudio, librosa, onnxruntime, HF Hub models

##### Main components and interactions
a. Data Ingestion
- User code / CLI
- torchaudio / librosa loader
b. Preprocessing
- alias_free_torch.filter (anti-alias filters)
- alias_free_torch.resample (resample input to 16 kHz)
c. Encoding
- codec_encoder.py (audio encoder network)
- semantic encoder (Wav2Vec2-BERT, external HF module)
- distill_layers.py (teacher-student distillation path)
- activations.py (nonlinearities)
- FSQ module (quantisation logic in model.py)
d. Bottleneck / Code output
- FSQ codes (tensor of discrete tokens)
e. Decoding
- codec_decoder_vocos.py (initial decoder network)
- tconv/ base and t_first (upsample from 16 k to 24 k)
f. ONNX Export Path
- export_onnx.py invokes model graph export
- onnx_ops.py defines any custom operator support
g. Packaging & Distribution
- pyproject.toml, publish.yaml for PyPI
- tests/ for CI validation

##### Architectural patterns and design principles
- Modular separation of preprocessing, encoding, quantisation, decoding
- Plugin-style subpackages for alias-free processing and t-conv upsampling
- Clear dependency flow: data → preprocess → encode → quantise → decode → output
- Optional export branch to ONNX runtime
- CI/CD for packaging and quality gates

##### Final advice
- Be exhaustive: include every major Python module as a component group
- Dual paths (training vs. distillation, PyTorch vs. ONNX runtime)
- Keep labels concise, show data rates and sample-rate transitions where relevant

## Diagram (Mermaid)

```mermaid
graph TB
    %% Data Ingestion
    subgraph "Data Ingestion"
        Librosa["librosa / torchaudio"]:::ext
    end

    %% Preprocessing
    subgraph "Preprocessing"
        Filter["alias_free_torch/filter.py"]:::preprocess
        Resample["alias_free_torch/resample.py"]:::preprocess
    end

    %% Core Codec
    subgraph "Core Codec"
        subgraph "Encoding"
            AudioEnc["codec_encoder.py"]:::core
            AudioEncDistill["codec_encoder_distill.py"]:::core
            DistillLayers["distill_layers.py"]:::core
            SemanticEnc["bs_roformer5.py"]:::core
        end
        subgraph "Activation & Modules"
            Activation["activations.py"]:::core
            Module["module.py"]:::core
            Model["model.py"]:::core
        end
        FSQ["FSQ Tokens"]:::core
        subgraph "Decoding"
            Decoder["codec_decoder_vocos.py"]:::core
            TconvBase["tconv/base.py"]:::core
            TconvFirst["tconv/t_first.py"]:::core
        end
    end

    %% ONNX Export
    subgraph "ONNX Export"
        ExportONNX["export_onnx.py"]:::export
        ONNXOps["onnx_ops.py"]:::export
    end

    %% Packaging & CI/CD
    subgraph "Packaging & CI/CD"
        Pyproject["pyproject.toml"]:::packaging
        Publish[".github/workflows/publish.yaml"]:::packaging
    end

    %% Tests
    subgraph "Tests / Validation"
        TestModels["tests/test_models.py"]:::test
    end

    %% External Dependencies
    HFModel["HF Wav2Vec2-BERT"]:::ext

    %% Data Flow
    Librosa -->|Waveform| Filter -->|Filtered Wave| Resample -->|16 kHz Tensor| AudioEnc
    Resample -->|16 kHz Tensor| AudioEncDistill
    HFModel -->|Semantic Features| SemanticEnc
    AudioEnc --> Activation --> Model
    AudioEncDistill --> DistillLayers --> Model
    SemanticEnc --> Model
    Module --> Model
    Model -->|FSQ Tokens| FSQ
    FSQ --> Decoder
    Decoder --> TconvBase --> TconvFirst -->|24 kHz Waveform| Output["Decoded Audio"]
    FSQ -->|Model Graph| ExportONNX --> ONNXOps

    %% Packaging & Tests
    Publish --> Pyproject
    Pyproject --> TestModels

    %% Click Events
    click Filter "https://github.com/neuphonic/neucodec/blob/main/neucodec/alias_free_torch/filter.py"
    click Resample "https://github.com/neuphonic/neucodec/blob/main/neucodec/alias_free_torch/resample.py"
    click AudioEnc "https://github.com/neuphonic/neucodec/blob/main/neucodec/codec_encoder.py"
    click AudioEncDistill "https://github.com/neuphonic/neucodec/blob/main/neucodec/codec_encoder_distill.py"
    click DistillLayers "https://github.com/neuphonic/neucodec/blob/main/neucodec/distill_layers.py"
    click SemanticEnc "https://github.com/neuphonic/neucodec/blob/main/neucodec/bs_roformer5.py"
    click Activation "https://github.com/neuphonic/neucodec/blob/main/neucodec/activations.py"
    click Module "https://github.com/neuphonic/neucodec/blob/main/neucodec/module.py"
    click Model "https://github.com/neuphonic/neucodec/blob/main/neucodec/model.py"
    click Decoder "https://github.com/neuphonic/neucodec/blob/main/neucodec/codec_decoder_vocos.py"
    click TconvBase "https://github.com/neuphonic/neucodec/blob/main/neucodec/tconv/base.py"
    click TconvFirst "https://github.com/neuphonic/neucodec/blob/main/neucodec/tconv/t_first.py"
    click ExportONNX "https://github.com/neuphonic/neucodec/blob/main/onnx/export_onnx.py"
    click ONNXOps "https://github.com/neuphonic/neucodec/blob/main/onnx/onnx_ops.py"
    click Pyproject "https://github.com/neuphonic/neucodec/blob/main/pyproject.toml"
    click Publish "https://github.com/neuphonic/neucodec/blob/main/.github/workflows/publish.yaml"
    click TestModels "https://github.com/neuphonic/neucodec/blob/main/tests/test_models.py"

    %% Styles
    classDef core fill:#ADD8E6,stroke:#000,stroke-width:1px
    classDef preprocess fill:#90EE90,stroke:#000,stroke-width:1px
    classDef export fill:#FFA500,stroke:#000,stroke-width:1px
    classDef packaging fill:#FFD700,stroke:#000,stroke-width:1px
    classDef ext fill:#D3D3D3,stroke:#000,stroke-width:1px
    classDef test fill:#CCCCCC,stroke:#000,stroke-width:1px
```
