# justinlovelace/sesd architecture

> Architecture diagram of the GitHub repository justinlovelace/sesd, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/justinlovelace/sesd
- Repository: https://github.com/justinlovelace/sesd

## Overview

##### Project type and purpose
- This is a research-oriented deep learning project implementing “Sample-Efficient Speech Diffusion (SESD)”, a latent diffusion model for text-to-speech (TTS) synthesis.
- It trains a U-Audio Transformer (U-AT) diffusion model in the latent space of a pretrained audio autoencoder, conditioned on character-level language model representations.
- Key use cases: training on modest-scale speech corpora (e.g., LibriSpeech), synthesizing intelligible TTS with minimal data, evaluating via WER metrics.

##### File structure
- Top-level directories:
- audio_datasets: data loaders & preprocessing utilities (LibriSpeech, forced aligner helpers).
- diffusion: core diffusion training logic (noise schedules, time samplers, optimizer).
- models: model definitions
– modules/: building blocks (convolutions, transformers, normalization, position embeddings)
– unet.py: U-Audio Transformer backbone
– transformer_wrapper.py: higher-level interface to transformer block
- neural_codec: wrapper around EnCodec autoencoder for latent representation I/O.
- evaluation: scripts to compute word error rate (evaluate_transcript.py).
- scripts/train & scripts/sample: shell scripts to train and run inference.
- train_audio_diffusion.py & compute_wer.py: entrypoints for training and WER computation.
- data/: bundled aligned LibriSpeech archive.
- utils/: miscellaneous utilities.
- requirements.txt, README.md, LICENSE, .gitignore.

- Patterns & architecture hints:
- Data ingestion → audio autoencoding → latent diffusion training → decoding → evaluation.
- Separation of concerns: data, model definition, diffusion algorithm, training loop, inference, evaluation.

##### README
- Clearly states core components: latent diffusion model (U-AT), pretrained audio autoencoder, scheduler, optimizer, training & sampling scripts.
- Mentions external dependencies: Montreal Forced Aligner, LibriSpeech from HuggingFace.

##### Detail
- Every module folder: e.g., models/modules → show that UNet is composed of conv, norm, transformer blocks.
- Scheduler and optimizer modules as subcomponents of the diffusion engine.
- Evaluation path: generated audio → ASR/WER computation.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Preparation
    subgraph "Data Preparation"
        DataArchive["Aligned LibriSpeech Archive"]:::external
        DataLoader["Data Loader & Preprocessor"]:::data
    end

    %% Core Model Components
    subgraph "Neural Codec"
        NeuralCodec["Neural Codec (EnCodec wrapper)"]:::model
    end

    subgraph "Latent Diffusion Model"
        UNet["U-Audio Transformer (UNet)"]:::model
        TFWrapper["Transformer Wrapper"]:::model
        subgraph "Modules"
            Blocks["blocks.py"]:::model
            Conv["conv.py"]:::model
            Norm["norm.py"]:::model
            Position["position.py"]:::model
            TransformerMod["transformer.py"]:::model
        end
    end

    subgraph "Diffusion Engine"
        Schedule["noise_schedule.py"]:::model
        Sampler["time_sampler.py"]:::model
        Optimizer["optimizer.py"]:::model
    end

    %% Orchestration
    subgraph "Training"
        TrainEntry["train_audio_diffusion.py"]:::orchestration
        TrainScript["scripts/train/train.sh"]:::orchestration
    end

    subgraph "Inference"
        SampleScript["scripts/sample/sample_16_ls_testclean.sh"]:::orchestration
    end

    subgraph "Evaluation"
        ComputeWER["compute_wer.py"]:::orchestration
        EvalTranscript["evaluate_transcript.py"]:::orchestration
    end

    %% Utilities & Checkpoints
    Utils["Utilities"]:::data
    Checkpoint["Checkpoint & Logging"]:::external

    %% Data Flow
    DataArchive -->|".tar.gz"| DataLoader
    DataLoader -->|".wav, text"| NeuralCodec
    NeuralCodec -->|"latent tensor"| UNet
    TFWrapper --> UNet
    Blocks --> UNet
    Conv --> UNet
    Norm --> UNet
    Position --> UNet
    TransformerMod --> TFWrapper
    UNet -->|"denoising step"| Schedule
    Schedule --> Sampler
    Sampler --> Optimizer
    Optimizer -->|"update weights"| UNet

    %% Training Control Flow
    UNet -->|".step()"| TrainEntry
    Schedule --> TrainEntry
    TrainEntry --> TrainScript
    TrainEntry --> Checkpoint

    %% Inference Flow
    Checkpoint --> SampleScript
    SampleScript --> UNet
    UNet --> NeuralCodec
    NeuralCodec -->|"waveform output"| Evaluation

    %% Evaluation Flow
    SampleScript --> ComputeWER
    ComputeWER --> EvalTranscript

    %% Utilities
    Utils --> TrainEntry
    Utils --> SampleScript
    Utils --> ComputeWER

    %% Click Events
    click DataLoader "https://github.com/justinlovelace/sesd/tree/main/audio_datasets/"
    click DataArchive "https://github.com/justinlovelace/sesd/blob/main/data/aligned_librispeech.tar.gz"
    click NeuralCodec "https://github.com/justinlovelace/sesd/blob/main/neural_codec/encodec_wrapper.py"
    click UNet "https://github.com/justinlovelace/sesd/blob/main/models/unet.py"
    click TFWrapper "https://github.com/justinlovelace/sesd/blob/main/models/transformer_wrapper.py"
    click Blocks "https://github.com/justinlovelace/sesd/blob/main/models/modules/blocks.py"
    click Conv "https://github.com/justinlovelace/sesd/blob/main/models/modules/conv.py"
    click Norm "https://github.com/justinlovelace/sesd/blob/main/models/modules/norm.py"
    click Position "https://github.com/justinlovelace/sesd/blob/main/models/modules/position.py"
    click TransformerMod "https://github.com/justinlovelace/sesd/blob/main/models/modules/transformer.py"
    click Schedule "https://github.com/justinlovelace/sesd/blob/main/diffusion/noise_schedule.py"
    click Sampler "https://github.com/justinlovelace/sesd/blob/main/diffusion/time_sampler.py"
    click Optimizer "https://github.com/justinlovelace/sesd/blob/main/diffusion/optimizer.py"
    click TrainEntry "https://github.com/justinlovelace/sesd/blob/main/train_audio_diffusion.py"
    click TrainScript "https://github.com/justinlovelace/sesd/blob/main/scripts/train/train.sh"
    click SampleScript "https://github.com/justinlovelace/sesd/blob/main/scripts/sample/sample_16_ls_testclean.sh"
    click ComputeWER "https://github.com/justinlovelace/sesd/blob/main/compute_wer.py"
    click EvalTranscript "https://github.com/justinlovelace/sesd/blob/main/evaluation/evaluate_transcript.py"
    click Utils "https://github.com/justinlovelace/sesd/blob/main/utils/utils.py"

    %% Styles
    classDef data fill:#e0f7e9,stroke:#027e46,color:#027e46
    classDef model fill:#e1e8fc,stroke:#1a4580,color:#1a4580
    classDef orchestration fill:#ffe8cc,stroke:#b35f00,color:#b35f00
    classDef external fill:#f0f0f0,stroke:#999999,color:#666666
```
