How lff8888/audiosort_tflm works
Project Type and Purpose
• This is a collection of TinyML demos for microcontrollers:
• “ArduinoSineFunction”: a TensorFlow Lite for Microcontrollers (TFLM) sine-wave generator on Arduino.
• “AudioSortCode”: end-to-end audio classification on Arduino/ESP32 (data acquisition → feature extraction → model training → model deployment → on-device inference).
High-Level Components
A. Offline (Development) Pipeline
1. Raw Audio Data & Samples
– Stored in “AudioSortCode/SamplingAndMelSpectrumAndTrain/sample” (e.g., .mp3, .txt).
2. Preprocessing & Feature Extraction
– Python scripts:
• datasetSplit.py (split datasets)
• melSpectrumReader.py (compute mel spectrogram)
• aiModuleTrain.py (train TensorFlow model)
3. Model Training
– TensorFlow training code in aiModuleTrain.py
– Produces a TensorFlow .tflite model file
4. Model Conversion & Header Generation
– Conversion script (inside aiModuleTrain.py or external TFLite converter)
– Outputs audio_classification_model.tflite
– Converts .tflite → C array → audio_classification_model.h
B. On-Device (Runtime) Pipeline
1. Microcontroller Firmware
– Arduino sketches:
• SamplingAndMelSpectrumAndTrain.ino (optional on-device training/demo)
• SamplingAndRecognize.ino (audio capture & inference)
• TFLM_TEST.ino (simple inference demo)
– Includes generated model header (audio_classification_model.h)
– Includes TFLM library and Arduino APIs
2. Hardware Platforms
– Arduino UNO / Nano (8-bit)
– ESP32 / ESP32-S3 (more compute, camera integration)
3. Peripherals
– Microphone module (I²S or analog)
– Optional camera module (ESP32-S3 face detection demo)
4. Output
– Serial console or LEDs indicating classified audio class
C. Arduino Sine Function Demo
1. TinyML “model.h” provides sine parameters
2. ArduinoSineFunction.ino reads model and outputs sine wave via DAC/PWM
3. Demonstrates TFLM integration for non-ML DSP
Relationships & Data Flow
1) Offline lane (data → preprocessing → training → model conversion)
2) On-device lane (firmware + model → microcontroller → classification/sine output)
• Raw data → Python preprocessing → Trained model
• Trained model → TFLite conversion → C header
• C header + Sketch → Compiled firmware → Hardware → Inference output
Architectural Patterns & Principles
• Pipeline pattern separating offline and runtime concerns
• Modularization: Python scripts are decoupled from Arduino sketches
• Reusability: single .h model file reused across demos
• Minimal dependencies on microcontroller: only TFLM and Arduino core
Relevant Technologies & Libraries
• Python 3.x, TensorFlow, TensorFlow Lite Converter
• TensorFlow Lite for Microcontrollers (TFLM)
• Arduino IDE / PlatformIO, C++
• ESP-IDF or Arduino‐ESP32 for ESP32 variants