# medusa2o/grammar-scoring-engine-on-shl-s-data architecture

> Architecture diagram of the GitHub repository medusa2o/grammar-scoring-engine-on-shl-s-data, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-19). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/medusa2o/grammar-scoring-engine-on-shl-s-data
- Repository: https://github.com/medusa2o/grammar-scoring-engine-on-shl-s-data

## Overview

##### Project type and purpose
- This is a data-science/ML pipeline (not a web app) that ingests raw audio, extracts acoustic features, trains a stack of regression models, and produces a final scoring engine. Its core purpose is to predict human-labeled grammar scores from audio recordings.

##### Main components
a. Data Ingestion & Storage
- Raw audio files (e.g., WAV/MP3) stored on disk or S3
- Augmented audio store (output of pitch/time augmentations)
b. Preprocessing & Feature Extraction
- Resampling module (to 16 kHz)
- Data augmentation module (pitch shift, time stretch)
- Librosa-based extractor producing 98-dim feature vectors (MFCC, Chroma, Spectral Contrast, Tonnetz, Tempo, RMS, ZCR)
- Intermediate feature store (e.g., CSV, HDF5)
c. Model Training
- Base learners (GBR, RF, ET, SVR, KNN, Ada, HistGB, XGB)
– Each trained in parallel on the same feature set via 5-fold CV
- Out-of-fold prediction collector (stacking dataset)
- Meta-learner (SVR) trained on OOF predictions
- Trained model artifacts (pickles or joblib files)
d. Evaluation & Visualization
- Metrics calculator (RMSE, Pearson r)
- Plot generator (matplotlib/seaborn output.png, output(1).png, etc.)
- Reporting (notebook cells or HTML export)

##### Relationships & data flow
Raw Audio → Preprocessing → Augmented Audio → Feature Extraction → Features
Features → Base Model Training (parallel) → OOF Predictions → Meta-Model Training → Final Model
Final Model + Test Features → Predictions → Evaluation & Visualization
- CV folds feeding OOF predictions into the stacking layer.

##### Architectural patterns & design principles
- Modular pipeline pattern: separate stages with clear inputs/outputs
- Ensemble stacking pattern: base layer → meta layer
- Reproducibility: versioned data artifacts and model artifacts
- Scalability: parallel training of base learners

##### Relevant technologies & libraries
- Python (Jupyter Notebook)
- Librosa (audio preprocessing & feature extraction)
- scikit-learn (GBR, RF, ET, SVR, KNN, AdaBoost, HistGB, pipeline utilities)
- XGBoost (XGB)
- matplotlib/seaborn (visualization)
- joblib or pickle (model persistence)

##### Essential details to capture
- Data augmentation branching (show original audio → multiple augmented versions)
- Feature dimensionality (98 dims)
- Parallel vs. sequential stages
- Cross-validation feeding into stacking

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Data Ingestion & Storage
    subgraph "Data Ingestion & Storage"
        direction TB
        RawAudio[(Raw Audio Store)]:::datastore
        AugAudio[(Augmented Audio Store)]:::datastore
    end

    %% Preprocessing & Feature Extraction
    subgraph "Preprocessing & Feature Extraction"
        direction TB
        Resample["Resampling to 16kHz\n(rate=16kHz)"]:::transformation
        Augment["Data Augmentation\n(pitch±2, time×0.9/1.1)"]:::transformation
        Librosa["Librosa Feature Extractor\n(98-dim MFCC/Chroma/...)" ]:::transformation
        FeatureStore[(Intermediate Feature Store\nCSV/HDF5)]:::datastore
    end

    %% Model Training
    subgraph "Model Training (Ensemble Stacking)"
        direction TB
        subgraph "Base Learners (parallel, 5-fold CV)"
            direction LR
            GBR["GBR"]:::model
            RF["RF"]:::model
            ET["ET"]:::model
            SVR1["SVR"]:::model
            KNN["KNN"]:::model
            Ada["AdaBoost"]:::model
            HistGB["HistGB"]:::model
            XGB["XGB"]:::model
        end
        OOF["OOF Prediction Collector"]:::model
        MetaLearner["Meta-learner (SVR)"]:::model
        ModelArtifacts[(Trained Model Artifacts)]:::datastore
        %% CV loop
        GBR -.->|folds| OOF
        RF -.->|folds| OOF
        ET -.->|folds| OOF
        SVR1 -.->|folds| OOF
        KNN -.->|folds| OOF
        Ada -.->|folds| OOF
        HistGB -.->|folds| OOF
        XGB -.->|folds| OOF
        OOF -->|training data| MetaLearner
        MetaLearner -->|model pickle| ModelArtifacts
    end

    %% Evaluation & Visualization
    subgraph "Evaluation & Visualization"
        direction TB
        Metrics["Metrics Calculator\n(RMSE, Pearson r)"]:::evaluation
        Plots["Plot Generator\n(matplotlib/seaborn)"]:::evaluation
        Reporting["Reporting\n(Notebook/HTML)"]:::evaluation
    end

    %% Data Flow Arrows
    RawAudio --> Resample
    Resample --> Augment
    Augment --> AugAudio
    AugAudio --> Librosa
    Librosa --> FeatureStore
    FeatureStore --> GBR
    FeatureStore --> RF
    FeatureStore --> ET
    FeatureStore --> SVR1
    FeatureStore --> KNN
    FeatureStore --> Ada
    FeatureStore --> HistGB
    FeatureStore --> XGB
    ModelArtifacts --> Metrics
    FeatureStore --> Metrics
    Metrics --> Plots
    Plots --> Reporting

    %% Click Events
    click RawAudio "https://github.com/medusa2o/grammar-scoring-engine-on-shl-s-data/blob/main/Untitled7 (1).ipynb"
    click Resample "https://github.com/medusa2o/grammar-scoring-engine-on-shl-s-data/blob/main/Untitled7 (1).ipynb"
    click FeatureStore "https://github.com/medusa2o/grammar-scoring-engine-on-shl-s-data/blob/main/Untitled7 (1).ipynb"
    click GBR "https://github.com/medusa2o/grammar-scoring-engine-on-shl-s-data/blob/main/Untitled7 (1).ipynb"
    click OOF "https://github.com/medusa2o/grammar-scoring-engine-on-shl-s-data/blob/main/Untitled7 (1).ipynb"
    click MetaLearner "https://github.com/medusa2o/grammar-scoring-engine-on-shl-s-data/blob/main/Untitled7 (1).ipynb"
    click ModelArtifacts "https://github.com/medusa2o/grammar-scoring-engine-on-shl-s-data/blob/main/Untitled7 (1).ipynb"
    click Reporting "https://github.com/medusa2o/grammar-scoring-engine-on-shl-s-data/blob/main/Untitled7 (1).ipynb"
    click README "https://github.com/medusa2o/grammar-scoring-engine-on-shl-s-data/blob/main/README.md"

    %% Legend
    subgraph "Legend"
        direction LR
        L1["Datastore (blue cylinder)"]:::datastore
        L2["Transformation (green)"]:::transformation
        L3["Model Training (orange)"]:::model
        L4["Evaluation (purple)"]:::evaluation
    end

    %% Styles
    classDef datastore fill:#cce5ff,stroke:#2b76d2,stroke-width:2,shape:cylinder
    classDef transformation fill:#d4edda,stroke:#28a745,stroke-width:2
    classDef model fill:#ffe5cc,stroke:#ff8c00,stroke-width:2
    classDef evaluation fill:#e8daef,stroke:#6f42c1,stroke-width:2
```
