# swapnoneelbarua/music-therapy-llm architecture

> Architecture diagram of the GitHub repository swapnoneelbarua/music-therapy-llm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-07-06). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/swapnoneelbarua/music-therapy-llm
- Repository: https://github.com/swapnoneelbarua/music-therapy-llm

## Overview

1. **Overall shape**
   - This is a Python research codebase organized as a pipeline of stable artifacts, not a single monolith.
   - The main flow is: raw audio dataset → indexed metadata → validated/preprocessed segments and features → human annotations and consensus labels → CNN benchmarks → downstream therapist/RAG modules.

2. **Primary source package boundaries**
   - The implementation lives under `src/indian_music_therapy_ai/`, split by concern:
     - `dataset` for discovery/indexing/preparation
     - `processing` for audio preprocessing and feature extraction
     - `annotations` for the annotation platform and consensus workflow
     - `training` and `benchmark` for model experimentation
     - `therapist` for retrieval/reasoning/recommendation logic
     - `cli`, `config`, and `utils` for orchestration and shared infrastructure
   - This separation matters because each stage writes artifacts that later stages reuse without schema churn.

3. **Configuration and command layer**
   - The repository is YAML-driven: `configs/*.yaml` define dataset, preprocessing, annotation, training, benchmark, and evaluation settings.
   - The CLI entrypoint is `icmta` via `src/indian_music_therapy_ai/cli/main.py`, which runs phase-specific commands like indexing, preprocessing, annotation, consensus, and benchmarking.
   - `utils/logging.py` and `config/loader.py` suggest consistent logging and centralized config loading across phases.

4. **Dataset ingestion and indexing**
   - Raw audio is expected under `dataset/esraj/` and `dataset/sarod/`, recursively discovered by the dataset subsystem.
   - `dataset/indexer.py`, `dataset/audio_probe.py`, and `dataset/preparation.py` generate a canonical `metadata/metadata.csv` plus validation and summary artifacts.
   - The key boundary here is that all later processing depends on this indexed metadata rather than scanning raw files again.

5. **Preprocessing and feature generation**
   - The `processing` package handles audio QC and transformation: validation, resampling, mono conversion, normalization, silence trimming, padding, segmentation, and feature extraction.
   - Outputs are staged under `data_processed/` and related directories: audio segments, manifests, spectrograms, and tensor/feature stores.
   - Feature types are explicitly separated into their own directories (`features/mfcc`, `features/chroma`, `features/energy`, `features/mel_spectrogram`, etc.), which supports reuse by both CNN training and analysis.

6. **Reusable manifests and experiment inputs**
   - Dataset splits live under `dataset/splits/` and benchmark-specific normalized manifests under `experiments/benchmark/normalized_manifests/`.
   - This indicates a deliberate boundary between raw annotation/processing outputs and model-ready train/val/test manifests.
   - The benchmark suite consumes these manifests to ensure comparisons across backbones are repeatable.

7. **Annotation platform and consensus workflow**
   - `annotations/` contains an offline SQLite-backed annotation system with GUI support (`gui.py`), session/database management, segment selection, playback, validation, and export.
   - The presence of `consensus.py`, `export.py`, and `validation.py` shows a pipeline from individual annotator records to consensus labels and quality checks.
   - Artifacts include annotator statistics, agreement reports, publication tables, and `consensus_labels.csv`, which become another downstream dataset boundary.

8. **Benchmarking and training**
   - `training/` contains model/dataset/trainer abstractions for CNN experimentation, while `benchmark/runner.py` wraps repeated runs across architectures.
   - The benchmark outputs under `experiments/benchmark/<model>/` include configs, history, predictions, confusion matrices, metrics, and TensorBoard logs.
   - Multiple backbones are represented in the repo state: `simplecnn`, `squeezenet`, `mobilenetv2`, `resnet18`, and `resnet50`, reinforcing that this is a comparative research suite.

9. **Therapist / retrieval / reasoning layer**
   - The `therapist` package is a later-stage application layer that combines knowledge, retrieval (`rag.py`), recommendation logic, and local SLM integration (`ollama_slm.py`, `slm.py`).
   - The README’s architecture suggests the model stack is not the end product; CNN emotion classification feeds a RAG + reasoning layer that ultimately produces music recommendations.
   - This is the clearest “top-of-pipeline” boundary: learned emotion recognition is treated as an input to decision support, not the final output.

10. **Interface and runtime surfaces**
   - `app.py` and `.agents/skills/developing-with-streamlit` indicate a Streamlit-facing interface for interactive use, while the annotation GUI uses PySide6.
   - This repo therefore has multiple presentation surfaces: a command-line orchestration layer, a desktop annotation tool, and a browser-style app interface.
   - Those interfaces are thin shells over the same underlying artifact pipeline.

11. **Persistent storage layout**
   - The repo is artifact-heavy and stores state in versioned folders: `metadata/`, `data_processed/`, `dataset/splits/`, `annotations/`, `experiments/`, `models/`, `figures/`, `results/`, and `database/`.
   - SQLite and vector-store placeholders show planned persistence boundaries for annotation records and future retrieval infrastructure.
   - The diagrams should reflect that most modules read/write files rather than communicating through a live service mesh.

12. **Documentation, audits, and tests**
   - `docs/` includes architecture, protocol, and phase-specific design docs, which appear to formalize the pipeline into stages.
   - `scripts/` provides dataset audits and validation checks, likely used to enforce consistency across phases.
   - `tests/` covers dataset indexing, preprocessing, annotations, training, AI therapist behavior, and benchmark execution, mirroring the repository’s subsystem boundaries.

Main entry points: [`src/indian_music_therapy_ai/therapist/orchestrator.py`](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/therapist/orchestrator.py) (Therapist), [`src/indian_music_therapy_ai/processing/pipeline.py`](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/processing/pipeline.py) (Processing), [`src/indian_music_therapy_ai/training/trainer.py`](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/training/trainer.py) (Training), [`metadata/metadata.csv`](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/metadata/metadata.csv) (Metadata)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_grp_core["Core pipeline"]
  node_config_loader["Config loader<br/>yaml config<br/>[loader.py]"]
  node_dataset_indexer["Indexing<br/>dataset discovery<br/>[indexer.py]"]
  node_audio_probe["Audio probe<br/>audio inspection<br/>[audio_probe.py]"]
  node_preparation["Preparation<br/>dataset prep<br/>[preparation.py]"]
  node_processing_pipeline["Processing<br/>audio pipeline<br/>[pipeline.py]"]
  node_audio_ops["Audio ops<br/>signal transforms<br/>[audio.py]"]
  node_feature_extractor["Features<br/>feature extraction<br/>[features.py]"]
  node_annotation_db[("Annotation DB<br/>sqlite store<br/>[database.py]")]
  node_consensus["Consensus<br/>label fusion<br/>[consensus.py]"]
  node_annotation_export["Export<br/>[export.py]"]
  node_training_framework["Training<br/>[trainer.py]"]
  node_benchmark_runner["Benchmark<br/>experiment runner<br/>[runner.py]"]
  node_therapist_orchestrator["Therapist<br/>decision layer<br/>[orchestrator.py]"]
  node_rag["RAG<br/>retrieval layer<br/>[rag.py]"]
  node_recommendation["Recommend<br/>[recommendation.py]"]
end

subgraph group_grp_ui["Interfaces"]
  node_cli_main(("CLI<br/>entrypoint<br/>[main.py]"))
  node_app(("App<br/>[app.py]"))
  node_annotation_gui(("Annotator UI<br/>desktop gui<br/>[gui.py]"))
end

subgraph group_grp_data["Artifact stores"]
  node_metadata_csv["Metadata<br/>indexed dataset<br/>[metadata.csv]"]
  node_processed_data[("Processed data<br/>staged artifacts")]
  node_feature_store[("Feature store<br/>feature artifacts")]
  node_annotation_artifacts["Labels<br/>consensus dataset"]
  node_split_manifests["Splits<br/>model inputs<br/>[train_manifest.csv]"]
  node_benchmark_outputs[("Benchmark runs<br/>experiment results")]
end

subgraph group_grp_docs["Docs and scripts"]
  node_docs["Docs<br/>[architecture.md]"]
  node_scripts["Scripts<br/>[scan_dataset.py]"]
end

node_cli_main -->|"loads"| node_config_loader
node_app -->|"uses"| node_therapist_orchestrator
node_annotation_gui -->|"writes"| node_annotation_db
node_dataset_indexer -->|"inspects"| node_audio_probe
node_dataset_indexer -->|"builds"| node_metadata_csv
node_preparation -->|"reads"| node_metadata_csv
node_preparation -->|"stages"| node_processed_data
node_processing_pipeline -->|"coordinates"| node_audio_ops
node_processing_pipeline -->|"generates"| node_feature_extractor
node_processing_pipeline -->|"writes"| node_processed_data
node_feature_extractor -->|"writes"| node_feature_store
node_consensus -->|"produces"| node_annotation_artifacts
node_annotation_export -->|"exports"| node_annotation_artifacts
node_training_framework -->|"consumes"| node_split_manifests
node_training_framework -->|"trains on"| node_feature_store
node_benchmark_runner -->|"wraps"| node_training_framework
node_benchmark_runner -->|"writes"| node_benchmark_outputs
node_therapist_orchestrator -->|"orchestrates"| node_rag
node_therapist_orchestrator -->|"produces"| node_recommendation
node_therapist_orchestrator -->|"consumes"| node_benchmark_outputs
node_docs -->|"describes"| node_cli_main
node_scripts -->|"audits"| node_metadata_csv
node_scripts -->|"checks"| node_annotation_artifacts

click node_cli_main "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/cli/main.py"
click node_app "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/app.py"
click node_annotation_gui "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/annotations/gui.py"
click node_config_loader "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/config/loader.py"
click node_dataset_indexer "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/dataset/indexer.py"
click node_audio_probe "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/dataset/audio_probe.py"
click node_preparation "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/dataset/preparation.py"
click node_processing_pipeline "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/processing/pipeline.py"
click node_audio_ops "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/processing/audio.py"
click node_feature_extractor "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/processing/features.py"
click node_annotation_db "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/annotations/database.py"
click node_consensus "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/annotations/consensus.py"
click node_annotation_export "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/annotations/export.py"
click node_training_framework "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/training/trainer.py"
click node_benchmark_runner "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/benchmark/runner.py"
click node_therapist_orchestrator "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/therapist/orchestrator.py"
click node_rag "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/therapist/rag.py"
click node_recommendation "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/therapist/recommendation.py"
click node_metadata_csv "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/metadata/metadata.csv"
click node_processed_data "https://github.com/swapnoneelbarua/music-therapy-llm/tree/main/data_processed"
click node_feature_store "https://github.com/swapnoneelbarua/music-therapy-llm/tree/main/features"
click node_annotation_artifacts "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/annotations/consensus_labels.csv"
click node_split_manifests "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/dataset/splits/train_manifest.csv"
click node_benchmark_outputs "https://github.com/swapnoneelbarua/music-therapy-llm/tree/main/experiments/benchmark"
click node_docs "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/docs/architecture.md"
click node_scripts "https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/scripts/scan_dataset.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_config_loader,node_dataset_indexer,node_audio_probe,node_preparation,node_processing_pipeline,node_audio_ops,node_feature_extractor,node_annotation_db,node_consensus,node_annotation_export,node_training_framework,node_benchmark_runner,node_therapist_orchestrator,node_rag,node_recommendation toneBlue
class node_cli_main,node_app,node_annotation_gui toneAmber
class node_metadata_csv,node_processed_data,node_feature_store,node_annotation_artifacts,node_split_manifests,node_benchmark_outputs toneMint
class node_docs,node_scripts toneRose
```

## Components

### Core pipeline

Source packages that implement the repository's staged workflow

- [Config loader](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/config/loader.py): yaml config · Centralized configuration loading for pipeline phases · `src/indian_music_therapy_ai/config/loader.py`
- [Indexing](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/dataset/indexer.py): dataset discovery · Discovers raw audio and builds canonical metadata · `src/indian_music_therapy_ai/dataset/indexer.py`
- [Audio probe](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/dataset/audio_probe.py): audio inspection · Inspects source audio and captures validation details · `src/indian_music_therapy_ai/dataset/audio_probe.py`
- [Preparation](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/dataset/preparation.py): dataset prep · Validates and prepares indexed dataset artifacts · `src/indian_music_therapy_ai/dataset/preparation.py`
- [Processing](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/processing/pipeline.py): audio pipeline · Coordinates preprocessing, segmentation, and feature extraction · `src/indian_music_therapy_ai/processing/pipeline.py`
- [Audio ops](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/processing/audio.py): signal transforms · Validation, resampling, normalization, trimming, and segmentation · `src/indian_music_therapy_ai/processing/audio.py`
- [Features](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/processing/features.py): feature extraction · Builds reusable audio feature representations · `src/indian_music_therapy_ai/processing/features.py`
- [Annotation DB](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/annotations/database.py): sqlite store · Session and record persistence for annotation workflow · `src/indian_music_therapy_ai/annotations/database.py`
- [Consensus](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/annotations/consensus.py): label fusion · Aggregates annotator records into consensus labels · `src/indian_music_therapy_ai/annotations/consensus.py`
- [Export](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/annotations/export.py): annotation export · Produces reports and downstream label artifacts · `src/indian_music_therapy_ai/annotations/export.py`
- [Training](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/training/trainer.py): cnn training · Dataset and trainer abstractions for model experimentation · `src/indian_music_therapy_ai/training/trainer.py`
- [Benchmark](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/benchmark/runner.py): experiment runner · Runs repeatable comparisons across CNN backbones · `src/indian_music_therapy_ai/benchmark/runner.py`
- [Therapist](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/therapist/orchestrator.py): decision layer · Orchestrates retrieval, reasoning, and recommendation outputs · `src/indian_music_therapy_ai/therapist/orchestrator.py`
- [RAG](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/therapist/rag.py): retrieval layer · Retrieval and context assembly for therapy decisions · `src/indian_music_therapy_ai/therapist/rag.py`
- [Recommend](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/therapist/recommendation.py): music recommendation · Maps model and knowledge signals to recommendations · `src/indian_music_therapy_ai/therapist/recommendation.py`

### Interfaces

Command, desktop, and app surfaces

- [CLI](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/cli/main.py): entrypoint · Main command-line orchestrator for phase-specific runs · `src/indian_music_therapy_ai/cli/main.py`
- [App](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/app.py): streamlit app · Interactive application surface over the artifact pipeline · `app.py`
- [Annotator UI](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/src/indian_music_therapy_ai/annotations/gui.py): desktop gui · Offline annotation interface with playback and segment review · `src/indian_music_therapy_ai/annotations/gui.py`

### Artifact stores

Persistent dataset, feature, annotation, and experiment outputs

- [Metadata](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/metadata/metadata.csv): indexed dataset · Canonical indexed metadata for all later stages · `metadata/metadata.csv`
- [Processed data](https://github.com/swapnoneelbarua/music-therapy-llm/tree/main/data_processed): staged artifacts · Validated audio, segments, manifests, spectrograms, and tensors · `data_processed`
- [Feature store](https://github.com/swapnoneelbarua/music-therapy-llm/tree/main/features): feature artifacts · Reusable feature families for training and analysis · `features`
- [Labels](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/annotations/consensus_labels.csv): consensus dataset · Annotator records, consensus labels, and reports · `annotations/consensus_labels.csv`
- [Splits](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/dataset/splits/train_manifest.csv): model inputs · Train, validation, and test manifests for experiments · `dataset/splits/train_manifest.csv`
- [Benchmark runs](https://github.com/swapnoneelbarua/music-therapy-llm/tree/main/experiments/benchmark): experiment results · Per-model metrics, predictions, and reports · `experiments/benchmark`

### Docs and scripts

Operational documentation and validation utilities

- [Docs](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/docs/architecture.md): design docs · Architecture and phase-specific repository documentation · `docs/architecture.md`
- [Scripts](https://github.com/swapnoneelbarua/music-therapy-llm/blob/main/scripts/scan_dataset.py): validation scripts · Audit and dataset consistency utilities · `scripts/scan_dataset.py`

## Connections

- CLI → Config loader: loads · CLI resolves YAML-driven phase configuration
- App → Therapist: uses · Interactive app calls the top-level therapy workflow
- Annotator UI → Annotation DB: writes · GUI persists annotator sessions and segment decisions
- Indexing → Audio probe: inspects · Indexing uses audio probing to validate source files
- Indexing → Metadata: builds · Creates the canonical indexed metadata artifact
- Preparation → Metadata: reads · Preparation consumes indexed metadata rather than raw scans
- Preparation → Processed data: stages · Produces validated prepared artifacts for later phases
- Processing → Audio ops: coordinates · Pipeline delegates signal transforms and segmentation
- Processing → Features: generates · Pipeline emits reusable feature representations
- Processing → Processed data: writes · Stores manifests, segments, spectrograms, and tensors
- Features → Feature store: writes · Feature families are stored as reusable artifacts
- Consensus → Labels: produces · Consensus workflow turns annotator records into labels
- Export → Labels: exports · Export writes reports and publication tables from labels
- Training → Splits: consumes · Model training uses split manifests as stable inputs
- Training → Feature store: trains on · Training reuses extracted feature artifacts
- Benchmark → Training: wraps · Benchmarking repeatedly invokes the training stack
- Benchmark → Benchmark runs: writes · Benchmark runs emit configs, metrics, predictions, and reports
- Therapist → RAG: orchestrates · Top-level therapist flow combines retrieval and reasoning
- Therapist → Recommend: produces · Orchestrator returns music recommendation outputs
- Therapist → Benchmark runs: consumes · Therapy layer can depend on learned classifier outputs
- Docs → CLI: describes · Documentation formalizes the pipeline entrypoints and phases
- Scripts → Metadata: audits · Utility scripts validate dataset indexing and exclusions
- Scripts → Labels: checks · Utility scripts inspect annotation and consensus consistency