# google/live-transcribe-speech-engine architecture

> Architecture diagram of the GitHub repository google/live-transcribe-speech-engine, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/google/live-transcribe-speech-engine
- Repository: https://github.com/google/live-transcribe-speech-engine

## Overview

Based on your project’s file tree and the README, this project is an Android client library that powers the Live Transcribe Speech Engine—a system designed to simulate an infinite streaming connection to Google’s Cloud Speech API. Its primary function is to continuously stream audio (with support for buffering, session restarts, and codec handling) so that real-time captioning can be achieved for accessibility applications.

##### Main Components
- Android Application Layer:
- UI & Activity: Represented by the Java classes in app/src/main/java/com/google/audio (for example, MainActivity.java), which provide the user interface and initial control logic.
- Session & Speech Management: Includes classes like RepeatingRecognitionSession, SpeechSession, SpeechSessionFactory, and others that manage the lifecycle of audio capture and streaming.
- Supporting Utilities: Components for network checking (NetworkConnectionChecker), audio buffering (CircularByteBuffer), and text formatting/labeling (SpeakerIDLabeler et al.) that support a robust streaming experience.
- Audio/Codec Processing Layer:
- Native Code: The C++ files under app/src/main/cpp (and its subdirectory jni) define critical parts of the system like the Ogg Opus encoder (ogg_opus_encoder.cc/.h), which handles low-latency audio encoding.
- JNI Bridge: The integration between Java and native C++ code so that the Android application can access the native audio encoding functionalities.
- Protocol & Configuration Components:
- gRPC/Cloud API Integration: Highlight the interaction with the Google Cloud Speech API, including the configuration of CloudSpeechSession and the use of various proto definitions (located in app/src/main/proto) which define the data structures for streaming sessions, transcription results, and audio codec parameters.
- Build & Configuration Files: The Android-specific configuration files (AndroidManifest.xml, gradle build scripts, CMakeLists.txt files) that ensure proper building and packaging of both Java and native code.
- Deployment & Distribution:
- APKs provided in the apk directory show that the project is distributed as pre-built binaries for quick evaluation.

##### Relationships & Interactions
- Data Flow: Illustrate how audio data moves from the device’s microphone through the Android capture components, gets buffered in the CircularByteBuffer, passed to the native codec layer via JNI, and ultimately transmitted as part of a streaming session to the Cloud Speech API.
- Session Management: Indicate the role of the RepeatingRecognitionSession in managing the lifespan of streaming sessions, ensuring that timeouts are avoided by closing and re-initiating sessions while buffering any interim audio.
- Error Handling & Resilience: Show how network checks (through NetworkConnectionChecker) and error publishers (like TranscriptionErrorPublisher) are connected to the session management to provide robustness.
- External Dependencies: Clearly denote that the application interacts with external systems such as the Google Cloud Speech API, using configurations specified by proto files and controlled via CloudSpeechSession classes.

##### Architectural Patterns & Design Principles
- Layered Architecture: Separate the Android UI/logic layer from the native audio processing, and further isolate network interactions and configuration management.
- Modular Design: Each module (UI, session management, audio buffering, encoding, and networking) is a distinct component that contributes to the overall system.
- Extensibility: Highlight points where the system is designed to be extensible (for instance, plug-in support for different speech detection or speaker ID algorithms).
- Integration via JNI: Clearly mark the boundary between managed Java code and native C++ code. This is integral to how the library handles high-performance, low-latency audio encoding.

##### Diagramming Details
- Use Clear Labels: Each component should have a distinct, descriptive title (e.g., “MainActivity – UI Layer”, “RepeatingRecognitionSession – Session Manager”, “Native Ogg Opus Encoder (C++/JNI)”).
- Annotations: Optionally annotate critical interactions—such as buffering for session continuity, or error resilience mechanisms—briefly describing their function.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Starting Point
    M["Microphone Input"]:::utility

    %% Android Client Layer
    subgraph "Android Client Layer"
        UI1["MainActivity - UI Layer"]:::ui
        UI2["Activity Layout Resource"]:::ui
        subgraph "Session & Speech Management"
            S1["RepeatingRecognitionSession - Session Manager"]:::session
            S2["SpeechSession - Audio Session"]:::session
            S3["SpeechSessionFactory - Session Factory"]:::session
            S4["TranscriptionErrorPublisher"]:::session
            S5["TranscriptionResultFormatter"]:::session
            S6["TranscriptionResultReceiver"]:::session
            S7["TranscriptionResultUpdatePublisher"]:::session
        end
        subgraph "Supporting Utilities"
            U1["CircularByteBuffer - Audio Buffer"]:::utility
            U2["NetworkConnectionChecker"]:::utility
            U3["SpeakerIDLabeler"]:::utility
        end
    end

    %% Native Audio/Codec Processing & JNI Bridge
    subgraph "Native Audio/Codec Processing & JNI Bridge"
        N1["JNI Bridge / Native Encoder (OGG Opus)"]:::native
    end

    %% Protocol & Cloud API Integration
    subgraph "Protocol & Cloud API Integration"
        C1["CloudSpeechSession - Cloud Session Manager"]:::protocol
        C2["CloudSpeechSessionFactory - Cloud Session Factory"]:::protocol
        C3["CloudSpeechStreamObserver"]:::protocol
        C4["TimestampCalculator"]:::protocol
        CP["Protocol Definitions"]:::protocol
    end

    %% Build & Configuration
    subgraph "Build & Configuration"
        B1["Android Manifest & Gradle Files"]:::build
        B2["Native Build Scripts"]:::build
    end

    %% Deployment & Distribution
    D1["APK Distribution"]:::deployment

    %% Data & Control Flow
    UI1 -->|"initiates"| S1
    UI2 -->|"supports"| UI1

    M -->|"buffers"| U1
    S1 -->|"sends_audio"| N1
    U1 -->|"passes_buffered_audio"| N1
    S1 -->|"checks_network"| U2
    S5 -->|"adds_speakerID"| U3
    S4 -->|"notifies_error"| S1

    N1 -->|"encodes_audio"| C1
    C1 -->|"observes_stream"| C3
    C3 -->|"acknowledges"| C1
    C1 -->|"configures_with"| CP
    C1 -->|"applies_timestamp"| C4

    %% Build and Deployment Connections
    B1 -->|"configures_app"| UI1
    B2 -->|"builds_native"| N1
    D1 -->|"distributes"| B1

    %% Classes Definition for Colors
    classDef ui fill:#f9c,stroke:#333,stroke-width:2px;
    classDef session fill:#bbf,stroke:#333,stroke-width:2px;
    classDef utility fill:#cfc,stroke:#333,stroke-width:2px;
    classDef native fill:#fc9,stroke:#333,stroke-width:2px;
    classDef protocol fill:#cff,stroke:#333,stroke-width:2px;
    classDef build fill:#ffc,stroke:#333,stroke-width:2px;
    classDef deployment fill:#fcc,stroke:#333,stroke-width:2px;

    %% Click Events for Android UI & Activity
    click UI1 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/MainActivity.java"
    click UI2 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/res/layout/activity_main.xml"

    %% Click Events for Session & Speech Management
    click S1 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/RepeatingRecognitionSession.java"
    click S2 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/SpeechSession.java"
    click S3 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/SpeechSessionFactory.java"
    click S4 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/TranscriptionErrorPublisher.java"
    click S5 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/TranscriptionResultFormatter.java"
    click S6 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/TranscriptionResultReceiver.java"
    click S7 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/TranscriptionResultUpdatePublisher.java"

    %% Click Events for Supporting Utilities
    click U1 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/CircularByteBuffer.java"
    click U2 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/NetworkConnectionChecker.java"
    click U3 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/SpeakerIDLabeler.java"

    %% Click Events for Native Audio/Codec Processing & JNI Bridge
    click N1 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/cpp/jni/ogg_opus_encoder.cc"
    click N1 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/cpp/jni/ogg_opus_encoder.h"
    click N1 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/cpp/ogg_opus_encoder.cc"
    click N1 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/cpp/ogg_opus_encoder.h"

    %% Click Events for Protocol & Cloud API Integration
    click C1 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/cloud/CloudSpeechSession.java"
    click C2 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/cloud/CloudSpeechSessionFactory.java"
    click C3 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/cloud/CloudSpeechStreamObserver.java"
    click C4 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/java/com/google/audio/asr/cloud/TimestampCalculator.java"
    click CP "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/proto/com/google/audio/asr/cloud_speech_session_params.proto"
    click CP "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/proto/com/google/audio/asr/cloud_speech_stream_observer_params.proto"
    click CP "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/proto/com/google/audio/asr/speech_recognition_model_options.proto"
    click CP "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/proto/com/google/audio/asr/transcription_result.proto"
    click CP "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/proto/com/google/audio/asr/transcription_result_formatter_options.proto"
    click CP "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/proto/com/google/audio/codec_and_bitrate.proto"
    click CP "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/proto/com/google/audio/network_state.proto"
    click CP "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/proto/com/google/audio/speaker_id_info.proto"

    %% Click Events for Build & Configuration
    click B1 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/AndroidManifest.xml"
    click B1 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/build.gradle"
    click B1 "https://github.com/google/live-transcribe-speech-engine/blob/master/build.gradle"
    click B1 "https://github.com/google/live-transcribe-speech-engine/blob/master/settings.gradle"
    click B2 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/CMakeLists.txt"
    click B2 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/cpp/CMakeLists.txt"
    click B2 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/src/main/cpp/jni/CMakeLists.txt"
    click B2 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/third_party/CMakeLists_libogg.txt"
    click B2 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/third_party/CMakeLists_libopus.txt"
    click B2 "https://github.com/google/live-transcribe-speech-engine/blob/master/app/third_party/CMakeLists_opus-tools.txt"

    %% Click Event for Deployment & Distribution
    click D1 "https://github.com/google/live-transcribe-speech-engine/tree/master/apk"
```
