How bavithaxa/audio-context-layer works
This repository explores an interpretable audio question-answering pipeline: synthetic scenes provide audio and ground-truth event manifests; audio perception produces timestamped context; a language model answers questions from that context. The graph also preserves the documented QA-generation and model-evaluation branches. Source excerpts are partial, so links between unsampled modules are included only where README documentation or sampled code supports them.
Main entry points: context Timestamped context · results Evaluation results · answer_question.py Answer question · evaluate_qa.py QA evaluation
Components 20
Scene dataset
- Compose scenes
scripts/compose_scenes.py - Scene audio
data - Event manifests
data/manifests - QA pairs
data/qa - Generate questions
scripts/generate_qa.py
Audio perception
- AST event detector
scripts/detect_events.py - YAMNet context builder
scripts/build_context.py - Timestamped context
data/context - Window index
data/processed/window_index.json
Question answering
- Answer question
scripts/answer_question.py
Evaluation
- QA evaluation
scripts/evaluate_qa.py - Score answers
scripts/score_qa.py - Error attribution
scripts/error_attribution.py - Perception evaluation
scripts/evaluate_ast.py - PANNs evaluation
scripts/evaluate_panns.py - YAMNet evaluation
scripts/evaluate_yamnet.py - QA predictions
results/qa_predictions.json - Evaluation results
results
Other components
- Questioner
- Groq language model
Connections 27
- Compose scenes → Scene audio: writes scenes
- Compose scenes → Event manifests: writes events
- Event manifests → Generate questions: supplies events
- Generate questions → QA pairs: writes questions
- Scene audio → AST event detector: processes audio
- AST event detector → Timestamped context: writes detections
- Scene audio → YAMNet context builder: processes audio
- YAMNet context builder → Timestamped context: writes context
- Questioner → Answer question: asks question
- Timestamped context → Answer question: provides evidence
- Answer question → Groq language model: requests answer
- Answer question → Questioner: returns answer
- QA pairs → QA evaluation: supplies questions
- Timestamped context → QA evaluation: supplies evidence
- QA evaluation → Groq language model: requests predictions
- QA evaluation → QA predictions: writes predictions
- QA predictions → Score answers: supplies answers
- Score answers → Evaluation results: writes scores
- Evaluation results → Error attribution: supplies scores
- Event manifests → Error attribution: supplies ground truth
- Timestamped context → Error attribution: supplies detections
- Window index → Perception evaluation: supplies windows
- Window index → PANNs evaluation: supplies windows
- Window index → YAMNet evaluation: supplies windows
- Perception evaluation → Evaluation results: writes results
- PANNs evaluation → Evaluation results: writes results
- YAMNet evaluation → Evaluation results: writes results