How rishig10/voice-based-pattern-detection works
Project Type and Purpose
• This is an open-source data‐processing and ML pipeline (not a web or mobile app).
• Its goal is to ingest raw voice recordings, extract speech/text and acoustic features, train a Random Forest model to distinguish “control” vs. “dementia,” then serialize the model for later inference.
File Structure
• Top level:
• audio_data/ — input dataset (two subdirs: control/, dementia/)
• notebook.ipynb & notebook_converted.py — the main training workflow
• test_model.py — standalone inference script
• saved_model_top/ — holds serialized artifacts: best_rf_model_top.pkl, scaler.pkl, feature_names.pkl
• README.md, .gitignore, etc.
• Patterns: a batch‐oriented ETL + model‐train pipeline, then an offline inference tool.
README for Insights
• Uses OpenAI Whisper for speech‐to‐text (needs ffmpeg).
• Extracts pause/hesitation metrics and acoustic features via librosa/pydub.
• Trains and evaluates a Random Forest (scikit‐learn), persists model+scaler+feature list.
Components and Interactions
A. Main Components
1. Data Store: File system folder “audio_data” with class subfolders
2. Preprocessing Pipeline (Notebook/Script):
a. Audio Loader → ffmpeg (via pydub)
b. Speech-to-Text Service (OpenAI Whisper model)
c. Acoustic Feature Extractor (librosa/pydub)
d. Text-Processing Module (pause/hesitation metrics)
e. Dataframe Builder (pandas) and Scaler (scikit-learn)
3. Model Trainer: Random Forest trainer + cross-validation → selects best model
4. Artifact Store: saved_model_top/ directory (model.pkl, scaler.pkl, feature_names.pkl)
5. Inference Module: test_model.py reads new audio file → runs same preprocessing → loads artifacts → outputs prediction & confidence
B. Relationships and Data Flow
C. Architectural Patterns
• Batch ETL + Model Training Stage separated from Runtime Inference Stage
• Pipeline broken into discrete steps (modular feature‐extractor, transcription, scaling)
• Reuse of code/modules for both training and inference
D. Key Technologies / Libraries
• OpenAI Whisper (speech-to-text)
• ffmpeg (audio decoding) via pydub
• librosa (acoustic features)
• pandas / numpy (data management)
• scikit-learn (scaler + Random Forest)
• pickle (model serialization)
Detail
• Sub‐component in the preprocessing stage (Transcriber, Acoustic Feature Extractor, Text Metrics).