How neuphonic/neucodec works
Project type and purpose
• This is a Python library and model repository (“neucodec”) implementing a lightweight neural audio codec.
• It provides encode/decode APIs for speech at 0.8 kbps, an ONNX export pathway, and integration with Speech-LM workflows.
File structure
• Top‐level packaging and CI:
• .github/workflows/publish.yaml (CI to publish pypi package)
• pyproject.toml (build metadata)
• LICENSE, README.md, .gitignore, pre-commit config
• Core library under neucodec/:
• activations.py, model.py, module.py (defines network blocks)
• codec_encoder*.py (BigCodec-style audio encoder, distillation variant)
• codec_decoder_vocos.py (upsampling decoder to 24 kHz)
• distill_layers.py (teacher-student compression layers)
• Subpackages for specialized processing:
• alias_free_torch/ (anti-alias filters, resampling)
• tconv/ (transposed-conv upsampling blocks)
• ONNX support under onnx/:
• export_onnx.py (conversion script)
• onnx_ops.py (custom ops or wrappers)
• Tests under tests/ for model‐level validation.
README for additional insights
• Describes dual encoder strategy (audio + semantic via Wav2Vec2-BERT)
• FSQ quantisation component producing token vectors
• Upsampling decoder raising sample rate 16→24 kHz
• External dependencies: PyTorch, torchaudio, librosa, onnxruntime, HF Hub models
Main components and interactions
a. Data Ingestion
• User code / CLI
• torchaudio / librosa loader
b. Preprocessing
• alias_free_torch.filter (anti-alias filters)
• alias_free_torch.resample (resample input to 16 kHz)
c. Encoding
• codec_encoder.py (audio encoder network)
• semantic encoder (Wav2Vec2-BERT, external HF module)
• distill_layers.py (teacher-student distillation path)
• activations.py (nonlinearities)
• FSQ module (quantisation logic in model.py)
d. Bottleneck / Code output
• FSQ codes (tensor of discrete tokens)
e. Decoding
• codec_decoder_vocos.py (initial decoder network)
• tconv/ base and t_first (upsample from 16 k to 24 k)
f. ONNX Export Path
• export_onnx.py invokes model graph export
• onnx_ops.py defines any custom operator support
g. Packaging & Distribution
• pyproject.toml, publish.yaml for PyPI
• tests/ for CI validation
Architectural patterns and design principles
• Modular separation of preprocessing, encoding, quantisation, decoding
• Plugin-style subpackages for alias-free processing and t-conv upsampling
• Clear dependency flow: data → preprocess → encode → quantise → decode → output
• Optional export branch to ONNX runtime
• CI/CD for packaging and quality gates
Final advice
• Be exhaustive: include every major Python module as a component group
• Dual paths (training vs. distillation, PyTorch vs. ONNX runtime)
• Keep labels concise, show data rates and sample-rate transitions where relevant