How iambasab/audiovisualonneucube works
Project type and purpose
• This is a research‐oriented data‐processing pipeline for audio–visual associative memory experiments using Spiking Neural Networks (NeuCube).
• It ingests raw audio/video, extracts and normalizes features, and prepares CSVs for downstream SNN training.
File structure
• Top‐level scripts and notebooks:
• Normalization.py – audio normalization logic
• code_used_to_extract_frames.py – video frame extraction
• RC_Extract_Audio_intervals.ipynb – splits audio into candidate intervals
• RC_Audio_Feature_Extraction.ipynb – extracts audio descriptors (MFCCs, spectral features, etc.)
• RC_work_with_train.ipynb / RC_work_without_plane.ipynb – pipelines that join features, label them (with/without train or plane) and export CSVs
• Data artifacts: raw media (Sample.mp4, sample_audio…mp3), intermediate CSVs (audio_with_train.csv, audio_without_train.csv), normalized CSVs
• README points out the preprocessing is for NeuCube SNN training
README for additional insights
• The pipeline is strictly for data preparation—no model training code here.
• All data files required to run preprocessing are included.
a. Main components
1. Raw Data Store
– Video files (Sample.mp4)
– Audio files (sample_audio…mp3)
2. Video Frame Extraction Module
– code_used_to_extract_frames.py
3. Audio Interval Extraction Module
– RC_Extract_Audio_intervals.ipynb
4. Audio Feature Extraction Module
– RC_Audio_Feature_Extraction.ipynb
5. Normalization Module
– Normalization.py
6. Labeling & CSV Assembly Modules
– RC_work_with_train.ipynb (labels “with train” samples)
– RC_work_without_plane.ipynb (labels “without train” samples)
7. Output Data Store
– audio_with_train.csv / audio_without_train.csv
– normalized variants (.csv)
8. Downstream SNN Training (NeuCube)
– External system
b. Relationships and data flow
Raw Video → Video Frame Extraction → Frame images (optional)
Raw Audio → Audio Interval Extraction → audio segments
audio segments → Audio Feature Extraction → raw feature sets
raw features → Normalization → normalized feature sets
normalized features + labels → CSV Assembly modules → final CSVs
final CSVs → NeuCube Training
c. Architectural patterns/design principles
– Linear data‐flow pipeline (ETL pattern)
– Separation of concerns: each notebook/script has a single responsibility
– Data immutability: every step writes new files, avoids in-place mutations
d. Key technologies/frameworks
– Python & Jupyter Notebooks
– Librosa (presumed for audio features)
– OpenCV or ffmpeg bindings (presumed for video frame extraction)
– Pandas (CSV handling)
– NeuCube (external spiking neural network platform)
Detailed checklist
• If helpful, number the steps to show execution order (1 through 6).