How kaviyathamizhan/audio_context_layer works
ACL turns audio into event timelines, then answers questions deterministically from those timelines. The main user workflow is demonstrated in acl/demo.py; dataset generation, detector training, benchmark evaluation, and edge-feasibility analysis are supporting product capabilities. The README documents the model variants and oracle-timeline evaluation; unsampled implementation details are not inferred beyond those documented capabilities.
Main entry points: demo.py Inference demo · qa_engine.py Deterministic QA engine · evaluate.py Benchmark evaluation · edge_report.py Edge feasibility analysis
Components 20
Benchmark Data
- Scene and QA generator
acl/data_gen.py - Scenes, annotations, QA
- Dataset loader
acl/data.py
Sound Perception
- SED training
acl/train_sed.py - CRNN / AST models
acl/models.py - Trained run artifacts
runs/ast_bigru_medium - Inference demo
acl/demo.py - Timeline post-decoder
acl/timeline.py
Contextual QA
- Audio event timeline
- Question parser
acl/templates.py - Deterministic QA engine
acl/qa_engine.py - Scene and cause mappings
acl/classes.py - Grounded answer
Evaluation & Edge
- Benchmark evaluation
acl/evaluate.py - SED and QA metrics
acl/metrics.py - Edge feasibility analysis
acl/edge_report.py
Other components
- Audio clip
- ESC-50 recordings
- Questioner
- Pretrained AST backbone
Connections 30
- ESC-50 recordings → Scene and QA generator: supplies recordings
- Scene and QA generator → Scenes, annotations, QA: generates scenes and QA
- Scenes, annotations, QA → Dataset loader: loads splits
- SED training → Dataset loader: loads training data
- SED training → CRNN / AST models: trains model
- CRNN / AST models → Pretrained AST backbone: extracts AST features
- SED training → Trained run artifacts: saves model outputs
- Audio clip → Inference demo: provides clip
- Inference demo → Trained run artifacts: loads model and decode settings
- Inference demo → CRNN / AST models: runs detection
- Inference demo → Timeline post-decoder: decodes probabilities
- Timeline post-decoder → Audio event timeline: produces events
- Questioner → Inference demo: provides question
- Inference demo → Question parser: parses question
- Inference demo → Deterministic QA engine: requests answer
- Deterministic QA engine → Question parser: parses question
- Deterministic QA engine → Audio event timeline: reasons over events
- Deterministic QA engine → Scene and cause mappings: looks up mappings
- Deterministic QA engine → Grounded answer: returns answer
- Inference demo → Grounded answer: displays result
- Benchmark evaluation → Dataset loader: loads evaluation data
- Benchmark evaluation → Trained run artifacts: reads predictions and config
- Benchmark evaluation → Timeline post-decoder: decodes predictions
- Benchmark evaluation → Deterministic QA engine: evaluates timeline answers
- Benchmark evaluation → SED and QA metrics: computes benchmarks
- Edge feasibility analysis → Trained run artifacts: reads trained run
- Edge feasibility analysis → CRNN / AST models: measures model
- Edge feasibility analysis → Dataset loader: loads test audio
- Edge feasibility analysis → Timeline post-decoder: decodes test predictions
- Edge feasibility analysis → SED and QA metrics: measures event F1