swapnoneelbarua/music-therapy-llm

1. Overall shape

- This is a Python research codebase organized as a pipeline of stable artifacts, not a single monolith.

- The main flow is: raw audio dataset → indexed metadata → validated/preprocessed segments and features → human annotations and consensus labels → CNN benchmarks → downstream therapist/RAG modules.

2. Primary source package boundaries

- The implementation lives under src/indian_music_therapy_ai/, split by concern:

- dataset for discovery/indexing/preparation

- processing for audio preprocessing and feature extraction

- annotations for the annotation platform and consensus workflow

- training and benchmark for model experimentation

- therapist for retrieval/reasoning/recommendation logic

- cli, config, and utils for orchestration and shared infrastructure

- This separation matters because each stage writes artifacts that later stages reuse without schema churn.

3. Configuration and command layer

- The repository is YAML-driven: configs/*.yaml define dataset, preprocessing, annotation, training, benchmark, and evaluation settings.

- The CLI entrypoint is icmta via src/indian_music_therapy_ai/cli/main.py, which runs phase-specific commands like indexing, preprocessing, annotation, consensus, and benchmarking.

- utils/logging.py and config/loader.py suggest consistent logging and centralized config loading across phases.

4. Dataset ingestion and indexing

- Raw audio is expected under dataset/esraj/ and dataset/sarod/, recursively discovered by the dataset subsystem.

- dataset/indexer.py, dataset/audio_probe.py, and dataset/preparation.py generate a canonical metadata/metadata.csv plus validation and summary artifacts.

- The key boundary here is that all later processing depends on this indexed metadata rather than scanning raw files again.

5. Preprocessing and feature generation

- The processing package handles audio QC and transformation: validation, resampling, mono conversion, normalization, silence trimming, padding, segmentation, and feature extraction.

- Outputs are staged under data_processed/ and related directories: audio segments, manifests, spectrograms, and tensor/feature stores.

- Feature types are explicitly separated into their own directories (features/mfcc, features/chroma, features/energy, features/mel_spectrogram, etc.), which supports reuse by both CNN training and analysis.

6. Reusable manifests and experiment inputs

- Dataset splits live under dataset/splits/ and benchmark-specific normalized manifests under experiments/benchmark/normalized_manifests/.

- This indicates a deliberate boundary between raw annotation/processing outputs and model-ready train/val/test manifests.

- The benchmark suite consumes these manifests to ensure comparisons across backbones are repeatable.

7. Annotation platform and consensus workflow

- annotations/ contains an offline SQLite-backed annotation system with GUI support (gui.py), session/database management, segment selection, playback, validation, and export.

- The presence of consensus.py, export.py, and validation.py shows a pipeline from individual annotator records to consensus labels and quality checks.

- Artifacts include annotator statistics, agreement reports, publication tables, and consensus_labels.csv, which become another downstream dataset boundary.

8. Benchmarking and training

- training/ contains model/dataset/trainer abstractions for CNN experimentation, while benchmark/runner.py wraps repeated runs across architectures.

- The benchmark outputs under experiments/benchmark/<model>/ include configs, history, predictions, confusion matrices, metrics, and TensorBoard logs.

- Multiple backbones are represented in the repo state: simplecnn, squeezenet, mobilenetv2, resnet18, and resnet50, reinforcing that this is a comparative research suite.

9. Therapist / retrieval / reasoning layer

- The therapist package is a later-stage application layer that combines knowledge, retrieval (rag.py), recommendation logic, and local SLM integration (ollama_slm.py, slm.py).

- The README’s architecture suggests the model stack is not the end product; CNN emotion classification feeds a RAG + reasoning layer that ultimately produces music recommendations.

- This is the clearest “top-of-pipeline” boundary: learned emotion recognition is treated as an input to decision support, not the final output.

10. Interface and runtime surfaces

- app.py and .agents/skills/developing-with-streamlit indicate a Streamlit-facing interface for interactive use, while the annotation GUI uses PySide6.

- This repo therefore has multiple presentation surfaces: a command-line orchestration layer, a desktop annotation tool, and a browser-style app interface.

- Those interfaces are thin shells over the same underlying artifact pipeline.

11. Persistent storage layout

- The repo is artifact-heavy and stores state in versioned folders: metadata/, data_processed/, dataset/splits/, annotations/, experiments/, models/, figures/, results/, and database/.

- SQLite and vector-store placeholders show planned persistence boundaries for annotation records and future retrieval infrastructure.

- The diagrams should reflect that most modules read/write files rather than communicating through a live service mesh.

12. Documentation, audits, and tests

- docs/ includes architecture, protocol, and phase-specific design docs, which appear to formalize the pipeline into stages.

- scripts/ provides dataset audits and validation checks, likely used to enforce consistency across phases.

- tests/ covers dataset indexing, preprocessing, annotations, training, AI therapist behavior, and benchmark execution, mirroring the repository’s subsystem boundaries.