# felixbur/nkululeko architecture

> Architecture diagram of the GitHub repository felixbur/nkululeko, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/felixbur/nkululeko
- Repository: https://github.com/felixbur/nkululeko

## Overview

##### Project Type and Purpose
- This is not a web app but a **CLI-driven, modular machine-learning experiment framework** in Python. It orchestrates the flow from raw audio datasets (CSV, WAV) through feature extraction, model training/optimization, evaluation, and report generation. Users control everything via INI-style configuration files, and the library provides pluggable components for data loading, feature extractors, models, augmenters, optimizers, and reporting.

##### High-Level File Structure Analysis
- **nkululeko/**: core library and CLI entry-points
- `nkululeko/nkululeko.py`, `ensemble.py`, `multidb.py`, etc. – top-level commands
- `runmanager.py`, `experiment.py`, `modelrunner.py`, `optimizationrunner.py` – orchestration engines
- `data/`, `feat_extract/`, `augmenting/`, `models/`, `reporting/`, `segmenting/` – plugin bins
- `flags.py`, `glob_conf.py` – configuration & factory logic
- **data/**: raw dataset definitions and processing scripts (one subfolder per corpus)
- **examples/** & **meta/**: sample INI files and demo scripts
- **scripts/**: convenience scripts to launch pipelines

##### Main Components & Their Relationships
- `python -m nkululeko.nkululeko`, `ensemble`, `multidb`, `demo`, `explore`, `augment`, `predict`, `segment`, `resample`, `optim`, `flags`
↳ parse `--config exp.ini` and hand off to RunManager
- Modules: `flags.py`, `glob_conf.py`, `nkuluflag.py`
↳ read INI, resolve sections ([DATA], [FEATS], [MODEL], etc.), create factory objects
- Classes: `RunManager`, `Experiment`, `ModelRunner`, `OptimizationRunner`
↳ orchestrate pipeline steps
- `nkululeko/data/dataset.py`, `dataset_csv.py`, plus project-specific `data/.../process_database.py`
↳ read CSVs, apply splitting strategies, return in-memory datasets
- `nkululeko/feat_extract/feats_*.py`, `transformer_feature_extractor.py`
↳ wrappers around Praat, opensmile, wav2vec2, CLAP, etc.
- `augmenting/`, `resample.py`
↳ random splicing, synthetic sampling (SMOTE), resampling rates
- `nkululeko/models/model_*.py` (SVM, XGB, KNN, MLP, CNN, GMM, etc.)
↳ scikit-learn & torch/tensorflow wrappers
h. **Optimization**
- `optim.py`, `OptimizationRunner`
↳ grid search, hyperparameter sweeps
- `reporting/report.py`, `latex_writer.py`, `plots.py`
↳ confusion matrix, t-SNE, feature importance, PDF/LaTeX reports
j. **Utilities & Core**
- `constants.py`, `cacheddataset.py`, `file_checker.py`, logging, stats, file I/O

##### Architectural Patterns & Design Principles
- **Plugin/Flexible Factory**: all extractors, models, metrics are loaded by name from config.
- **Pipeline Separation**: clear stages (Data → Features → [Augment] → Model → Report).
- **Config-Driven**: no hard-coded parameters; the INI drives the flow.
- **Modular & Testable**: each plugin has its own tests under `nkululeko/*/tests` and `tests/`.
- **CLI Frontend / Core Engine**: entry-points delegate to the same core pipeline.

## Diagram (Mermaid)

```mermaid
graph TB
    %% Legend
    subgraph "Legend"
        CLI["CLI Commands"]:::cli
        CONFIG["Config Loader"]:::config
        ENGINE["Core Engine"]:::engine
        PLUGIN["Plugins"]:::plugin
        OUTPUT["Outputs"]:::output
        EXTERNAL["External"]:::external
    end

    %% Top: INI and CLI
    ini[/INI File/]:::cli
    subgraph "CLI Interfaces" 
        nkululeko["nkululeko.py"]:::cli
        ensemble["ensemble.py"]:::cli
        multidb["multidb.py"]:::cli
        demo["demo.py"]:::cli
        explore["explore.py"]:::cli
        augment["augment.py"]:::cli
        predict["predict.py"]:::cli
        segment["segment.py"]:::cli
        resample_cmd["resample.py"]:::cli
        optim_cmd["optim.py"]:::cli
        flags_cmd["flags.py"]:::cli
    end

    %% Configuration Loader
    subgraph "Configuration Loader" 
        flags["flags.py"]:::config
        glob_conf["glob_conf.py"]:::config
        nkuluflag["nkuluflag.py"]:::config
    end

    %% Core Engine
    subgraph "RunManager / Orchestrator"
        runmanager["runmanager.py"]:::engine
        experiment["experiment.py"]:::engine
        modelrunner["modelrunner.py"]:::engine
        optimizationrunner["optimizationrunner.py"]:::engine
    end

    %% Pipeline Stages
    subgraph "Pipeline"
        subgraph "Data Processing"
            dataset["dataset.py"]:::plugin
            dataset_csv["dataset_csv.py"]:::plugin
            process_db["process_database.py"]:::plugin
        end
        subgraph "Feature Extraction"
            feats["feats_*.py"]:::plugin
            transformer["transformer_feature_extractor.py"]:::plugin
        end
        subgraph "Augmentation / Resampling"
            augmenter["augmenter.py"]:::plugin
            randomsplicer["randomsplicer.py"]:::plugin
            randomsplicing["randomsplicing.py"]:::plugin
            resampler["resampler.py"]:::plugin
            resample_mod["resample.py"]:::plugin
        end
        subgraph "Model Training / Optimization"
            models["model_*.py"]:::plugin
            optim_mod["optim.py"]:::plugin
            opt_runner["optimizationrunner.py"]:::engine
        end
        subgraph "Reporting / Visualization"
            report["report.py"]:::output
            latex["latex_writer.py"]:::output
            plots["plots.py"]:::output
        end
    end

    %% External Dependencies & Outputs
    subgraph "External Resources"
        raw_data(("Raw Audio Datasets")):::external
        sklib(["sklearn, torch, Praat, openSMILE"]):::external
        fs_out(("File System Outputs")):::external
    end

    %% Connections
    ini -->|reads| nkululeko
    ini -->|reads| ensemble
    ini -->|reads| multidb
    ini -->|reads| demo
    ini -->|reads| explore
    ini -->|reads| augment
    ini -->|reads| predict
    ini -->|reads| segment
    ini -->|reads| resample_cmd
    ini -->|reads| optim_cmd
    ini -->|reads| flags_cmd

    nkululeko --> flags
    ensemble --> flags
    flags_cmd --> flags

    flags --> glob_conf
    flags --> nkuluflag

    nkululeko --> runmanager
    ensemble --> runmanager
    multidb --> runmanager
    demo --> runmanager
    explore --> runmanager
    augment --> runmanager
    predict --> runmanager
    segment --> runmanager
    resample_cmd --> runmanager
    optim_cmd --> runmanager

    runmanager --> experiment
    experiment --> modelrunner
    modelrunner --> optimizationrunner

    optimizationrunner -->|grid search| opt_runner

    runmanager --> dataset
    dataset --> dataset_csv
    dataset_csv --> process_db
    process_db --> raw_data

    runmanager --> feats
    feats --> transformer

    runmanager --> augmenter
    augmenter --> randomsplicer
    augmenter --> randomsplicing
    runmanager --> resampler
    resampler --> resample_mod

    runmanager --> models
    runmanager --> optimizationrunner

    runmanager --> report
    report --> latex
    report --> plots
    report --> fs_out

    %% Click Events
    click nkululeko "https://github.com/felixbur/nkululeko/blob/main/nkululeko/nkululeko.py"
    click ensemble "https://github.com/felixbur/nkululeko/blob/main/nkululeko/ensemble.py"
    click multidb "https://github.com/felixbur/nkululeko/blob/main/nkululeko/multidb.py"
    click demo "https://github.com/felixbur/nkululeko/blob/main/nkululeko/demo.py"
    click explore "https://github.com/felixbur/nkululeko/blob/main/nkululeko/explore.py"
    click augment "https://github.com/felixbur/nkululeko/blob/main/nkululeko/augment.py"
    click predict "https://github.com/felixbur/nkululeko/blob/main/nkululeko/predict.py"
    click segment "https://github.com/felixbur/nkululeko/blob/main/nkululeko/segment.py"
    click resample_cmd "https://github.com/felixbur/nkululeko/blob/main/nkululeko/resample.py"
    click optim_cmd "https://github.com/felixbur/nkululeko/blob/main/nkululeko/optim.py"
    click flags_cmd "https://github.com/felixbur/nkululeko/blob/main/nkululeko/flags.py"

    click flags "https://github.com/felixbur/nkululeko/blob/main/nkululeko/flags.py"
    click glob_conf "https://github.com/felixbur/nkululeko/blob/main/nkululeko/glob_conf.py"
    click nkuluflag "https://github.com/felixbur/nkululeko/blob/main/nkululeko/nkuluflag.py"

    click runmanager "https://github.com/felixbur/nkululeko/blob/main/nkululeko/runmanager.py"
    click experiment "https://github.com/felixbur/nkululeko/blob/main/nkululeko/experiment.py"
    click modelrunner "https://github.com/felixbur/nkululeko/blob/main/nkululeko/modelrunner.py"
    click optimizationrunner "https://github.com/felixbur/nkululeko/blob/main/nkululeko/optimizationrunner.py"

    click dataset "https://github.com/felixbur/nkululeko/blob/main/nkululeko/data/dataset.py"
    click dataset_csv "https://github.com/felixbur/nkululeko/blob/main/nkululeko/data/dataset_csv.py"
    click process_db "https://github.com/felixbur/nkululeko/blob/main/data/*/process_database.py"

    click feats "https://github.com/felixbur/nkululeko/blob/main/nkululeko/feat_extract/feats_praat.py"
    click transformer "https://github.com/felixbur/nkululeko/blob/main/nkululeko/feat_extract/transformer_feature_extractor.py"

    click augmenter "https://github.com/felixbur/nkululeko/blob/main/nkululeko/augmenting/augmenter.py"
    click randomsplicer "https://github.com/felixbur/nkululeko/blob/main/nkululeko/augmenting/randomsplicer.py"
    click randomsplicing "https://github.com/felixbur/nkululeko/blob/main/nkululeko/augmenting/randomsplicing.py"
    click resampler "https://github.com/felixbur/nkululeko/blob/main/nkululeko/augmenting/resampler.py"
    click resample_mod "https://github.com/felixbur/nkululeko/blob/main/nkululeko/resample.py"

    click models "https://github.com/felixbur/nkululeko/blob/main/nkululeko/models/model_svm.py"
    click optim_mod "https://github.com/felixbur/nkululeko/blob/main/nkululeko/optim.py"
    click opt_runner "https://github.com/felixbur/nkululeko/blob/main/nkululeko/optimizationrunner.py"

    click report "https://github.com/felixbur/nkululeko/blob/main/nkululeko/reporting/report.py"
    click latex "https://github.com/felixbur/nkululeko/blob/main/nkululeko/reporting/latex_writer.py"
    click plots "https://github.com/felixbur/nkululeko/blob/main/nkululeko/plots.py"

    %% Styles
    classDef cli fill:#a2f5a2,stroke:#333,stroke-width:1px
    classDef config fill:#fff79a,stroke:#333,stroke-width:1px
    classDef engine fill:#9fd8f1,stroke:#333,stroke-width:1px
    classDef plugin fill:#f4b6c2,stroke:#333,stroke-width:1px
    classDef output fill:#a9d3e2,stroke:#333,stroke-width:1px
    classDef external fill:#e0e0e0,stroke:#333,stroke-width:1px
```
