# iambasab/audiovisualonneucube architecture

> Architecture diagram of the GitHub repository iambasab/audiovisualonneucube, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/iambasab/audiovisualonneucube
- Repository: https://github.com/iambasab/audiovisualonneucube

## Overview

##### Project type and purpose
- This is a research‐oriented data‐processing pipeline for audio–visual associative memory experiments using Spiking Neural Networks (NeuCube).
- It ingests raw audio/video, extracts and normalizes features, and prepares CSVs for downstream SNN training.

##### File structure
- Top‐level scripts and notebooks:
- Normalization.py – audio normalization logic
- code_used_to_extract_frames.py – video frame extraction
- RC_Extract_Audio_intervals.ipynb – splits audio into candidate intervals
- RC_Audio_Feature_Extraction.ipynb – extracts audio descriptors (MFCCs, spectral features, etc.)
- RC_work_with_train.ipynb / RC_work_without_plane.ipynb – pipelines that join features, label them (with/without train or plane) and export CSVs
- Data artifacts: raw media (Sample.mp4, sample_audio…mp3), intermediate CSVs (audio_with_train.csv, audio_without_train.csv), normalized CSVs
- README points out the preprocessing is for NeuCube SNN training

##### README for additional insights
- The pipeline is strictly for data preparation—no model training code here.
- All data files required to run preprocessing are included.

a. Main components
1. Raw Data Store
– Video files (Sample.mp4)
– Audio files (sample_audio…mp3)
2. Video Frame Extraction Module
– code_used_to_extract_frames.py
3. Audio Interval Extraction Module
– RC_Extract_Audio_intervals.ipynb
4. Audio Feature Extraction Module
– RC_Audio_Feature_Extraction.ipynb
5. Normalization Module
– Normalization.py
6. Labeling & CSV Assembly Modules
– RC_work_with_train.ipynb (labels “with train” samples)
– RC_work_without_plane.ipynb (labels “without train” samples)
7. Output Data Store
– audio_with_train.csv / audio_without_train.csv
– normalized variants (.csv)
8. Downstream SNN Training (NeuCube)
– External system

b. Relationships and data flow
Raw Video → Video Frame Extraction → Frame images (optional)
Raw Audio → Audio Interval Extraction → audio segments
audio segments → Audio Feature Extraction → raw feature sets
raw features → Normalization → normalized feature sets
normalized features + labels → CSV Assembly modules → final CSVs
final CSVs → NeuCube Training

c. Architectural patterns/design principles
– Linear data‐flow pipeline (ETL pattern)
– Separation of concerns: each notebook/script has a single responsibility
– Data immutability: every step writes new files, avoids in-place mutations

d. Key technologies/frameworks
– Python & Jupyter Notebooks
– Librosa (presumed for audio features)
– OpenCV or ffmpeg bindings (presumed for video frame extraction)
– Pandas (CSV handling)
– NeuCube (external spiking neural network platform)

##### Detailed checklist
- If helpful, number the steps to show execution order (1 through 6).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Raw Inputs
    subgraph "Raw Data Store"
        direction TB
        RawVideo["Raw Video\n(Sample.mp4)"]:::raw
        RawAudio["Raw Audio\n(Sample_audio_used.mp3\nsample_audio_with_plane.mp3\nsample_audio_without_plane.mp3)"]:::raw
    end

    %% Processing Modules
    subgraph "Processing Modules"
        direction TB
        VideoExtract["Video Frame Extraction\n(code_used_to_extract_frames.py)"]:::proc
        AudioInterval["Audio Interval Extraction\n(RC_Extract_Audio_intervals.ipynb)"]:::proc
        AudioFeature["Audio Feature Extraction\n(RC_Audio_Feature_Extraction.ipynb)"]:::proc
        Normalize["Normalization\n(Normalization.py)"]:::proc
    end

    %% Labeling & CSV Assembly
    subgraph "Audio Processing & Labeling"
        direction TB
        WithTrain["Label & Assemble CSV (with train)\n(RC_work_with_train.ipynb)"]:::proc
        WithoutPlane["Label & Assemble CSV (without plane)\n(RC_work_without_plane.ipynb)"]:::proc
    end

    %% Intermediate & Output Data Stores
    subgraph "Data Artifacts"
        direction TB
        Frames["Extracted Frames\n(folder)"]:::data
        Segments["Audio Segments\n(.mp3 segments)"]:::data
        RawFeatures["Raw Feature Sets\n(.csv)"]:::data
        NormFeatures["Normalized Feature Sets\n(.csv)"]:::data
        RawCSVs["Raw CSVs\n(audio_with_train.csv\naudio_without_train.csv)"]:::data
        NormCSVs["Normalized CSVs\n(audio_with_train_normalized.csv\naudio_without_train_normalized.csv)"]:::data
    end

    %% External System
    NeuCube["NeuCube SNN Training"]:::external

    %% Flow
    RawVideo -->|".mp4"| VideoExtract -->|"frames"| Frames
    RawAudio -->|".mp3"| AudioInterval -->|"segments"| Segments
    Segments -->|".mp3 segments"| AudioFeature -->|"feature matrix"| RawFeatures
    RawFeatures -->|"raw features"| Normalize -->|"normalized features"| NormFeatures
    NormFeatures --> WithTrain
    NormFeatures --> WithoutPlane
    WithTrain -->|"with train CSVs"| RawCSVs
    WithoutPlane -->|"without plane CSVs"| RawCSVs
    RawCSVs --> NormCSVs
    NormCSVs -->|"final CSVs"| NeuCube

    %% Click Events
    click VideoExtract "https://github.com/iambasab/audiovisualonneucube/blob/main/code_used_to_extract_frames.py"
    click AudioInterval "https://github.com/iambasab/audiovisualonneucube/blob/main/RC_Extract_Audio_intervals.ipynb"
    click AudioFeature "https://github.com/iambasab/audiovisualonneucube/blob/main/RC_Audio_Feature_Extraction.ipynb"
    click Normalize "https://github.com/iambasab/audiovisualonneucube/blob/main/Normalization.py"
    click WithTrain "https://github.com/iambasab/audiovisualonneucube/blob/main/RC_work_with_train.ipynb"
    click WithoutPlane "https://github.com/iambasab/audiovisualonneucube/blob/main/RC_work_without_plane.ipynb"

    %% Styles
    classDef raw fill:#ADD8E6,stroke:#333,stroke-width:1px
    classDef proc fill:#90EE90,stroke:#333,stroke-width:1px
    classDef data fill:#FFA500,stroke:#333,stroke-width:1px
    classDef external fill:#DDA0DD,stroke:#333,stroke-width:1px
```
