# kaviyathamizhan/audio_context_layer architecture

> Architecture diagram of the GitHub repository kaviyathamizhan/audio_context_layer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kaviyathamizhan/audio_context_layer
- Repository: https://github.com/kaviyathamizhan/audio_context_layer

## Overview

ACL turns audio into event timelines, then answers questions deterministically from those timelines. The main user workflow is demonstrated in `acl/demo.py`; dataset generation, detector training, benchmark evaluation, and edge-feasibility analysis are supporting product capabilities. The README documents the model variants and oracle-timeline evaluation; unsampled implementation details are not inferred beyond those documented capabilities.

Main entry points: [`acl/demo.py`](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/demo.py) (Inference demo), [`acl/qa_engine.py`](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/qa_engine.py) (Deterministic QA engine), [`acl/evaluate.py`](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/evaluate.py) (Benchmark evaluation), [`acl/edge_report.py`](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/edge_report.py) (Edge feasibility analysis)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_dataset["Benchmark Data"]
  node_generator["Scene and QA generator<br/>[data_gen.py]"]
  node_dataset_store[("Scenes, annotations, QA")]
  node_data_loader["Dataset loader<br/>[data.py]"]
end

subgraph group_perception["Sound Perception"]
  node_trainer["SED training<br/>[train_sed.py]"]
  node_detector_models["CRNN / AST models<br/>[models.py]"]
  node_run_artifacts["Trained run artifacts"]
  node_demo["Inference demo<br/>[demo.py]"]
  node_timeline_decoder["Timeline post-decoder<br/>[timeline.py]"]
end

subgraph group_reasoning["Contextual QA"]
  node_timeline["Audio event timeline"]
  node_question_parser["Question parser<br/>[templates.py]"]
  node_qa_engine["Deterministic QA engine<br/>[qa_engine.py]"]
  node_ontology["Scene and cause mappings<br/>[classes.py]"]
  node_answer["Grounded answer"]
end

subgraph group_analysis["Evaluation &amp; Edge"]
  node_evaluator["Benchmark evaluation<br/>[evaluate.py]"]
  node_metrics["SED and QA metrics<br/>[metrics.py]"]
  node_edge_report["Edge feasibility analysis<br/>[edge_report.py]"]
end

node_audio_user(("Audio clip"))
node_esc50[("ESC-50 recordings")]
node_question_user(("Questioner"))
node_ast_service["Pretrained AST backbone"]

node_esc50 -->|"supplies recordings"| node_generator
node_generator -->|"generates scenes and QA"| node_dataset_store
node_dataset_store -->|"loads splits"| node_data_loader
node_trainer -->|"loads training data"| node_data_loader
node_trainer -->|"trains model"| node_detector_models
node_detector_models -.->|"extracts AST features"| node_ast_service
node_trainer -->|"saves model outputs"| node_run_artifacts
node_audio_user -->|"provides clip"| node_demo
node_demo -->|"loads model and decode settings"| node_run_artifacts
node_demo -->|"runs detection"| node_detector_models
node_demo -->|"decodes probabilities"| node_timeline_decoder
node_timeline_decoder -->|"produces events"| node_timeline
node_question_user -->|"provides question"| node_demo
node_demo -->|"parses question"| node_question_parser
node_demo -->|"requests answer"| node_qa_engine
node_qa_engine -->|"parses question"| node_question_parser
node_qa_engine -->|"reasons over events"| node_timeline
node_qa_engine -->|"looks up mappings"| node_ontology
node_qa_engine -->|"returns answer"| node_answer
node_demo -->|"displays result"| node_answer
node_evaluator -->|"loads evaluation data"| node_data_loader
node_evaluator -->|"reads predictions and config"| node_run_artifacts
node_evaluator -->|"decodes predictions"| node_timeline_decoder
node_evaluator -->|"evaluates timeline answers"| node_qa_engine
node_evaluator -->|"computes benchmarks"| node_metrics
node_edge_report -->|"reads trained run"| node_run_artifacts
node_edge_report -->|"measures model"| node_detector_models
node_edge_report -->|"loads test audio"| node_data_loader
node_edge_report -->|"decodes test predictions"| node_timeline_decoder
node_edge_report -->|"measures event F1"| node_metrics

click node_generator "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/data_gen.py"
click node_data_loader "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/data.py"
click node_trainer "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/train_sed.py"
click node_detector_models "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/models.py"
click node_run_artifacts "https://github.com/kaviyathamizhan/audio_context_layer/tree/main/runs/ast_bigru_medium"
click node_demo "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/demo.py"
click node_timeline_decoder "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/timeline.py"
click node_question_parser "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/templates.py"
click node_qa_engine "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/qa_engine.py"
click node_ontology "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/classes.py"
click node_evaluator "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/evaluate.py"
click node_metrics "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/metrics.py"
click node_edge_report "https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/edge_report.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_generator,node_dataset_store,node_data_loader toneBlue
class node_trainer,node_detector_models,node_run_artifacts,node_demo,node_timeline_decoder,node_esc50 toneAmber
class node_timeline,node_question_parser,node_qa_engine,node_ontology,node_answer toneMint
class node_evaluator,node_metrics,node_edge_report toneRose
class node_audio_user,node_question_user,node_ast_service toneIndigo
```

## Components

### Benchmark Data

- [Scene and QA generator](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/data_gen.py): `acl/data_gen.py`
- **Scenes, annotations, QA**
- [Dataset loader](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/data.py): `acl/data.py`

### Sound Perception

- [SED training](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/train_sed.py): `acl/train_sed.py`
- [CRNN / AST models](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/models.py): `acl/models.py`
- [Trained run artifacts](https://github.com/kaviyathamizhan/audio_context_layer/tree/main/runs/ast_bigru_medium): `runs/ast_bigru_medium`
- [Inference demo](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/demo.py): `acl/demo.py`
- [Timeline post-decoder](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/timeline.py): `acl/timeline.py`

### Contextual QA

- **Audio event timeline**
- [Question parser](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/templates.py): `acl/templates.py`
- [Deterministic QA engine](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/qa_engine.py): `acl/qa_engine.py`
- [Scene and cause mappings](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/classes.py): `acl/classes.py`
- **Grounded answer**

### Evaluation & Edge

- [Benchmark evaluation](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/evaluate.py): `acl/evaluate.py`
- [SED and QA metrics](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/metrics.py): `acl/metrics.py`
- [Edge feasibility analysis](https://github.com/kaviyathamizhan/audio_context_layer/blob/main/acl/edge_report.py): `acl/edge_report.py`

### Other components

- **Audio clip**
- **ESC-50 recordings**
- **Questioner**
- **Pretrained AST backbone**

## Connections

- ESC-50 recordings → Scene and QA generator: supplies recordings
- Scene and QA generator → Scenes, annotations, QA: generates scenes and QA
- Scenes, annotations, QA → Dataset loader: loads splits
- SED training → Dataset loader: loads training data
- SED training → CRNN / AST models: trains model
- CRNN / AST models → Pretrained AST backbone: extracts AST features
- SED training → Trained run artifacts: saves model outputs
- Audio clip → Inference demo: provides clip
- Inference demo → Trained run artifacts: loads model and decode settings
- Inference demo → CRNN / AST models: runs detection
- Inference demo → Timeline post-decoder: decodes probabilities
- Timeline post-decoder → Audio event timeline: produces events
- Questioner → Inference demo: provides question
- Inference demo → Question parser: parses question
- Inference demo → Deterministic QA engine: requests answer
- Deterministic QA engine → Question parser: parses question
- Deterministic QA engine → Audio event timeline: reasons over events
- Deterministic QA engine → Scene and cause mappings: looks up mappings
- Deterministic QA engine → Grounded answer: returns answer
- Inference demo → Grounded answer: displays result
- Benchmark evaluation → Dataset loader: loads evaluation data
- Benchmark evaluation → Trained run artifacts: reads predictions and config
- Benchmark evaluation → Timeline post-decoder: decodes predictions
- Benchmark evaluation → Deterministic QA engine: evaluates timeline answers
- Benchmark evaluation → SED and QA metrics: computes benchmarks
- Edge feasibility analysis → Trained run artifacts: reads trained run
- Edge feasibility analysis → CRNN / AST models: measures model
- Edge feasibility analysis → Dataset loader: loads test audio
- Edge feasibility analysis → Timeline post-decoder: decodes test predictions
- Edge feasibility analysis → SED and QA metrics: measures event F1