# caladog/heygem architecture

> Architecture diagram of the GitHub repository caladog/heygem, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/caladog/heygem
- Repository: https://github.com/caladog/heygem

## Overview

Heygem is a two-part system: a desktop Electron client and a set of local Docker-based AI services. The repository mostly contains the desktop app plus deployment manifests for the companion services, not a standalone cloud backend.

High-level architecture

- Desktop application:
  - Built with Electron.
  - Renderer UI is a Vue application.
  - Main process owns local orchestration, filesystem access, persistence, and integration with media/AI services.
  - Preload script bridges renderer requests into the main process.

- Local AI services:
  - Started separately with Docker Compose from the `deploy/` directory.
  - README indicates three local services are expected to run on `127.0.0.1`, including:
    - ASR / speech preprocessing and TTS service on port `18180`
    - Video synthesis service on port `8383`
    - A third companion service in the Docker stack for the full workflow
  - These services are the heavy compute boundary; the Electron app coordinates them but does not implement the ML models itself.

Main subsystems in the repo

1. Electron main process layer
- Entry point: `src/main/index.js`
- Responsibilities:
  - Application bootstrap
  - Wiring handlers, services, configuration, and logging
  - Coordinating media jobs and talking to local HTTP services
  - Managing local files and domain state

- Important subfolders:
  - `src/main/handlers/`
    - IPC or app-facing request handlers
    - `app.js`, `file.js`, `index.js` suggest app control and file operations are exposed here
  - `src/main/service/`
    - Domain/business logic
    - `model.js`, `video.js`, `voice.js` map directly to the three user workflows described in the README
    - `context.js` suggests shared runtime/session state
  - `src/main/dao/`
    - Data access layer for domain records
    - Separate domain objects for `context`, `f2f-model`, `video`, and `voice`
  - `src/main/db/`
    - Local database bootstrap and SQL definitions
    - This is the app’s persistence boundary for metadata and workflow state
  - `src/main/api/`
    - HTTP client wrappers for external/local service calls
    - `request.js` is likely the shared transport layer
    - `f2f.js` and `tts.js` reflect specific service integrations
  - `src/main/util/ffmpeg.js`
    - Wrapper around bundled FFmpeg tooling for media manipulation
  - `src/main/interval/interval.js`
    - Background polling/scheduling, likely for job progress or state refresh
  - `src/main/config/config.js`
    - Central configuration
  - `src/main/logger.js`
    - Shared logging

2. Preload boundary
- `src/preload/index.js`
- This is the security boundary between renderer and main:
  - Exposes a limited API surface to the UI
  - Prevents direct Node.js / filesystem access from the renderer
  - Typically used for IPC calls into app handlers and services

3. Renderer UI layer
- Root UI entry:
  - `src/renderer/index.html`
  - `src/renderer/src/main.js`
  - `src/renderer/src/App.vue`

- UI framework and state:
  - Vue application with routing and state management
  - `src/renderer/src/router/`
  - `src/renderer/src/stores/`
    - `app.js`, `home.js`, `user.js`
  - `src/renderer/src/i18n/`
    - English and Chinese support
    - Matches the README’s multilingual client note

- Major feature views:
  - `src/renderer/src/views/home/`
    - Dashboard-style landing area
    - Contains model lists, works lists, video dialogs, and banner content
  - `src/renderer/src/views/account/`
    - User/account related page
  - `src/renderer/src/views/video-edit/`
    - Multi-step video editing/synthesis workflow
    - Split into `select`, `edit`, `preview`, and `header`
  - `src/renderer/src/components/model-create/`
    - Dedicated model creation workflow
    - Includes upload, guidance, and modal-based steps

- UI support:
  - Shared components like headers, dialogs, menus, and agreement screens
  - Assets and theme CSS under `src/renderer/src/assets/`
  - `src/renderer/src/api/` and `src/renderer/src/client/`
    - Frontend-side API wrappers that talk to the main-process bridge

4. Media and model runtime assets
- `resources/ffmpeg/`
  - Bundled FFmpeg binaries for Linux and Windows
  - Includes `ffmpeg`, `ffprobe`, `qt-faststart`, headers/libs, presets, and model files
  - This is a major operational dependency for audio/video preprocessing and packaging

- The `resources/ffmpeg/linux-amd64/model/` tree contains many VMAF-related and quality-evaluation model files
  - Indicates the app ships with video-quality / analysis model assets needed at runtime

5. Packaging and deployment
- `electron-builder.yml`, `electron.vite.config.mjs`, `build/`
  - Desktop packaging and OS-specific installer configuration
  - macOS entitlements are present, so the app targets multiple desktop platforms
- `deploy/`
  - Docker Compose manifests:
    - `docker-compose.yml`
    - `docker-compose-lite.yml`
    - `docker-compose-linux.yml`
  - These define the local AI service stack that the desktop client depends on
- `dev-app-update.yml`
  - Update/dev packaging support

Core data flows

1. Model training / voice cloning flow
- User starts in the renderer UI, typically from the model creation area.
- UI sends a request through the preload bridge into the main process.
- Main process service logic prepares files and metadata, then calls the local ASR/TTS Docker service.
- The README describes a two-step workflow:
  - preprocess audio and transcribe it via `preprocess_and_tran`
  - synthesize speech via `invoke`
- Returned artifacts include audio URLs/paths and reference text.
- DAO/database layers persist the model/voice metadata so the UI can later list and reuse it.

2. Video synthesis flow
- User selects source video, generated or uploaded audio, and task parameters in the video editor UI.
- Main service calls the local video synthesis endpoint on `8383`:
  - submit job
  - poll progress with a query endpoint
- FFmpeg tooling is available in the app runtime for local media handling, such as format conversion or preprocessing.
- The resulting output is saved locally and surfaced back to the UI for preview/download.

3. Content browsing and project management
- Home/dashboard screens list available models, works, and related artifacts.
- The renderer reads and updates app state through stores.
- Main-process services read/write to the local database and filesystem, keeping UI state and persisted metadata in sync.

Important boundaries

- UI boundary:
  - The renderer is presentation only.
  - It should be treated as a thin client that drives workflows and shows results.

- IPC boundary:
  - The preload layer is the controlled bridge between browser-like UI code and privileged desktop capabilities.

- Orchestration boundary:
  - The Electron main process is the central coordinator for local jobs, persistence, and service calls.

- Compute boundary:
  - The Dockerized services own the expensive ML inference and synthesis workloads.
  - This repo does not embed the model execution logic in the Electron UI; it integrates with those services over localhost HTTP.

- Storage boundary:
  - User data, generated media, and model artifacts are stored locally on disk.
  - The README emphasizes local directories such as `D:\heygem_data\...` for service-side data.

What to show in an architecture diagram

- A desktop Electron/Vue client as the front end
- A preload/IPC boundary into the Electron main process
- Main-process services for model, voice, and video workflows
- A local SQL persistence layer for app state and metadata
- Local filesystem/media processing with bundled FFmpeg
- External-but-local Docker services on `127.0.0.1` for ASR, TTS, and video synthesis
- Docker Compose deployment manifests as the operational setup for those services

In short, Heygem is an offline-first desktop orchestration app: the Electron client handles UX and workflow coordination, while Dockerized local AI services do the heavy inference and synthesis work.

Main entry points: [`src/main/handlers/index.js`](https://github.com/caladog/heygem/blob/main/src/main/handlers/index.js) (IPC Handlers), [`src/renderer/src/router/index.js`](https://github.com/caladog/heygem/blob/main/src/renderer/src/router/index.js) (State & Routes), [`src/main/service/index.js`](https://github.com/caladog/heygem/blob/main/src/main/service/index.js) (Domain Services), [`src/main/api/request.js`](https://github.com/caladog/heygem/blob/main/src/main/api/request.js) (Local APIs)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_desktop["Desktop App"]
  node_renderer_app["Vue UI<br/>renderer<br/>[App.vue]"]
  node_renderer_state["State & Routes<br/>ui state<br/>[index.js]"]
  node_renderer_feature_home["Home<br/>feature view<br/>[index.vue]"]
  node_renderer_feature_model["Model Create<br/>feature workflow"]
  node_renderer_feature_video["Video Edit<br/>feature workflow<br/>[VideoEditView.vue]"]
  node_renderer_feature_account["Account<br/>feature view<br/>[index.vue]"]
  node_preload_bridge{{"Preload Bridge<br/>ipc bridge<br/>[index.js]"}}
  node_main_boot["Main Bootstrap<br/>electron main<br/>[index.js]"]
  node_main_handlers["IPC Handlers<br/>request handlers<br/>[index.js]"]
  node_main_services["Domain Services<br/>workflow orchestration<br/>[index.js]"]
  node_main_api["Local APIs<br/>http clients<br/>[request.js]"]
  node_main_persist[("DB Layer<br/>local persistence<br/>[index.js]")]
  node_main_dao["DAOs<br/>data access<br/>[context.js]"]
  node_main_ffmpeg["FFmpeg Util<br/>media runtime<br/>[ffmpeg.js]"]
  node_main_interval(("Polling Loop<br/>scheduler<br/>[interval.js]"))
end

subgraph group_local["Local AI Stack"]
  node_local_asr_tts[("ASR/TTS Service<br/>docker service<br/>[docker-compose.yml]")]
  node_local_video[("Video Synthesis<br/>docker service<br/>[docker-compose.yml]")]
  node_local_companion[("Companion Service<br/>docker service<br/>[docker-compose.yml]")]
  node_runtime_media["FFmpeg Assets<br/>bundled binaries"]
end

subgraph group_ops["Packaging & Deploy"]
  node_deploy_manifests["Compose Stack<br/>deployment manifests<br/>[docker-compose.yml]"]
  node_packaging["Desktop Packaging<br/>build config"]
end

node_renderer_app -->|"uses"| node_renderer_state
node_renderer_state -->|"navigates"| node_renderer_feature_home
node_renderer_state -->|"navigates"| node_renderer_feature_model
node_renderer_state -->|"navigates"| node_renderer_feature_video
node_renderer_state -->|"navigates"| node_renderer_feature_account
node_renderer_app -->|"IPC"| node_preload_bridge
node_preload_bridge -->|"bridges"| node_main_handlers
node_main_boot -->|"registers"| node_main_handlers
node_main_boot -->|"orchestrates"| node_main_services
node_main_boot -->|"opens"| node_main_persist
node_main_services -->|"calls"| node_main_api
node_main_services -->|"persists"| node_main_dao
node_main_dao -->|"queries"| node_main_persist
node_main_services -->|"processes"| node_main_ffmpeg
node_main_services -->|"polls"| node_main_interval
node_main_api -->|"HTTP"| node_local_asr_tts
node_main_api -->|"HTTP"| node_local_video
node_main_api -->|"HTTP"| node_local_companion
node_renderer_feature_home -->|"requests"| node_main_handlers
node_renderer_feature_model -->|"requests"| node_main_handlers
node_renderer_feature_video -->|"requests"| node_main_handlers
node_renderer_feature_account -->|"requests"| node_main_handlers
node_deploy_manifests -->|"defines"| node_local_asr_tts
node_deploy_manifests -->|"defines"| node_local_video
node_deploy_manifests -->|"defines"| node_local_companion
node_runtime_media -->|"bundles"| node_main_ffmpeg
node_packaging -->|"packages"| node_main_boot
node_packaging -->|"packages"| node_renderer_app

click node_renderer_app "https://github.com/caladog/heygem/blob/main/src/renderer/src/App.vue"
click node_renderer_state "https://github.com/caladog/heygem/blob/main/src/renderer/src/router/index.js"
click node_renderer_feature_home "https://github.com/caladog/heygem/blob/main/src/renderer/src/views/home/index.vue"
click node_renderer_feature_model "https://github.com/caladog/heygem/blob/main/src/renderer/src/components/model-create/ModelCreateView.vue"
click node_renderer_feature_video "https://github.com/caladog/heygem/blob/main/src/renderer/src/views/video-edit/VideoEditView.vue"
click node_renderer_feature_account "https://github.com/caladog/heygem/blob/main/src/renderer/src/views/account/index.vue"
click node_preload_bridge "https://github.com/caladog/heygem/blob/main/src/preload/index.js"
click node_main_boot "https://github.com/caladog/heygem/blob/main/src/main/index.js"
click node_main_handlers "https://github.com/caladog/heygem/blob/main/src/main/handlers/index.js"
click node_main_services "https://github.com/caladog/heygem/blob/main/src/main/service/index.js"
click node_main_api "https://github.com/caladog/heygem/blob/main/src/main/api/request.js"
click node_main_persist "https://github.com/caladog/heygem/blob/main/src/main/db/index.js"
click node_main_dao "https://github.com/caladog/heygem/blob/main/src/main/dao/context.js"
click node_main_ffmpeg "https://github.com/caladog/heygem/blob/main/src/main/util/ffmpeg.js"
click node_main_interval "https://github.com/caladog/heygem/blob/main/src/main/interval/interval.js"
click node_local_asr_tts "https://github.com/caladog/heygem/blob/main/deploy/docker-compose.yml"
click node_local_video "https://github.com/caladog/heygem/blob/main/deploy/docker-compose.yml"
click node_local_companion "https://github.com/caladog/heygem/blob/main/deploy/docker-compose.yml"
click node_runtime_media "https://github.com/caladog/heygem/tree/main/resources/ffmpeg/linux-amd64/ffmpeg"
click node_deploy_manifests "https://github.com/caladog/heygem/blob/main/deploy/docker-compose.yml"
click node_packaging "https://github.com/caladog/heygem/blob/main/electron-builder.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_renderer_app,node_renderer_state,node_renderer_feature_home,node_renderer_feature_model,node_renderer_feature_video,node_renderer_feature_account,node_preload_bridge,node_main_boot,node_main_handlers,node_main_services,node_main_api,node_main_persist,node_main_dao,node_main_ffmpeg,node_main_interval toneBlue
class node_local_asr_tts,node_local_video,node_local_companion,node_runtime_media toneAmber
class node_deploy_manifests,node_packaging toneMint
```

## Components

### Desktop App

Electron client with renderer, preload bridge, and main-process orchestration.

- [Vue UI](https://github.com/caladog/heygem/blob/main/src/renderer/src/App.vue): renderer · Presentation layer for home, account, model creation, and video editing. · `src/renderer/src/App.vue`
- [State & Routes](https://github.com/caladog/heygem/blob/main/src/renderer/src/router/index.js): ui state · Client routing, stores, and language support. · `src/renderer/src/router/index.js`
- [Home](https://github.com/caladog/heygem/blob/main/src/renderer/src/views/home/index.vue): feature view · Dashboard for models, works, and workflow entry points. · `src/renderer/src/views/home/index.vue`
- [Model Create](https://github.com/caladog/heygem/blob/main/src/renderer/src/components/model-create/ModelCreateView.vue): feature workflow · UI flow for creating and uploading a model/voice profile. · `src/renderer/src/components/model-create/ModelCreateView.vue`
- [Video Edit](https://github.com/caladog/heygem/blob/main/src/renderer/src/views/video-edit/VideoEditView.vue): feature workflow · Step-based video selection, editing, and preview flow. · `src/renderer/src/views/video-edit/VideoEditView.vue`
- [Account](https://github.com/caladog/heygem/blob/main/src/renderer/src/views/account/index.vue): feature view · User/account page and related client state. · `src/renderer/src/views/account/index.vue`
- [Preload Bridge](https://github.com/caladog/heygem/blob/main/src/preload/index.js): ipc bridge · Controlled API surface between renderer and privileged main process. · `src/preload/index.js`
- [Main Bootstrap](https://github.com/caladog/heygem/blob/main/src/main/index.js): electron main · Application entry, wiring, and top-level orchestration. · `src/main/index.js`
- [IPC Handlers](https://github.com/caladog/heygem/blob/main/src/main/handlers/index.js): request handlers · App- and file-facing handlers exposed to the renderer. · `src/main/handlers/index.js`
- [Domain Services](https://github.com/caladog/heygem/blob/main/src/main/service/index.js): workflow orchestration · Model, voice, video, and shared context workflows. · `src/main/service/index.js`
- [Local APIs](https://github.com/caladog/heygem/blob/main/src/main/api/request.js): http clients · HTTP wrappers for localhost AI service calls. · `src/main/api/request.js`
- [DB Layer](https://github.com/caladog/heygem/blob/main/src/main/db/index.js): local persistence · Local database bootstrap and SQL definitions for app state. · `src/main/db/index.js`
- [DAOs](https://github.com/caladog/heygem/blob/main/src/main/dao/context.js): data access · Domain records for context, model, video, and voice metadata. · `src/main/dao/context.js`
- [FFmpeg Util](https://github.com/caladog/heygem/blob/main/src/main/util/ffmpeg.js): media runtime · Bundled FFmpeg wrapper for media preprocessing and packaging. · `src/main/util/ffmpeg.js`
- [Polling Loop](https://github.com/caladog/heygem/blob/main/src/main/interval/interval.js): scheduler · Background interval for progress refresh and job coordination. · `src/main/interval/interval.js`

### Local AI Stack

Docker-based companion services and runtime media assets on localhost.

- [ASR/TTS Service](https://github.com/caladog/heygem/blob/main/deploy/docker-compose.yml): docker service · Local speech preprocessing, transcription, and TTS boundary on 127.0.0.1:18180. · `deploy/docker-compose.yml`
- [Video Synthesis](https://github.com/caladog/heygem/blob/main/deploy/docker-compose.yml): docker service · Local video generation service on 127.0.0.1:8383. · `deploy/docker-compose.yml`
- [Companion Service](https://github.com/caladog/heygem/blob/main/deploy/docker-compose.yml): docker service · Third local service in the Docker stack supporting the full workflow. · `deploy/docker-compose.yml`
- [FFmpeg Assets](https://github.com/caladog/heygem/tree/main/resources/ffmpeg/linux-amd64/ffmpeg): bundled binaries · Packaged FFmpeg executables, presets, and quality-model assets. · `resources/ffmpeg/linux-amd64/ffmpeg`

### Packaging & Deploy

Build, installer, and service deployment manifests.

- [Compose Stack](https://github.com/caladog/heygem/blob/main/deploy/docker-compose.yml): deployment manifests · Docker Compose definitions for the local AI services. · `deploy/docker-compose.yml`
- [Desktop Packaging](https://github.com/caladog/heygem/blob/main/electron-builder.yml): build config · Electron builder and vite packaging for desktop installers. · `electron-builder.yml`

## Connections

- Vue UI → State & Routes: uses · UI routes and shared store state drive the app shell.
- State & Routes → Home: navigates · State and routing surface the home dashboard.
- State & Routes → Model Create: navigates · State and routing surface model creation flows.
- State & Routes → Video Edit: navigates · State and routing surface video editing flows.
- State & Routes → Account: navigates · State and routing surface account views.
- Vue UI → Preload Bridge: IPC · Renderer sends privileged requests through the preload boundary.
- Preload Bridge → IPC Handlers: bridges · Preload exposes a limited API into main-process handlers.
- Main Bootstrap → IPC Handlers: registers · Main process wires the handler surface at startup.
- Main Bootstrap → Domain Services: orchestrates · Bootstrap initializes domain services and app coordination.
- Main Bootstrap → DB Layer: opens · Main process initializes local persistence.
- Domain Services → Local APIs: calls · Workflow services invoke local HTTP integrations.
- Domain Services → DAOs: persists · Services store and load workflow metadata through DAOs.
- DAOs → DB Layer: queries · DAO operations map to the local SQL database.
- Domain Services → FFmpeg Util: processes · Media workflows use bundled FFmpeg tooling.
- Domain Services → Polling Loop: polls · Background polling tracks async job progress.
- Local APIs → ASR/TTS Service: HTTP · Speech preprocessing and TTS are served locally.
- Local APIs → Video Synthesis: HTTP · Video synthesis jobs are submitted to the local service.
- Local APIs → Companion Service: HTTP · Workflow orchestration also depends on the companion service.
- Home → IPC Handlers: requests · Home screens load models and works via handler calls.
- Model Create → IPC Handlers: requests · Model creation submits preprocessing and synthesis actions.
- Video Edit → IPC Handlers: requests · Video editing submits media jobs and progress queries.
- Account → IPC Handlers: requests · Account screens read and update app-facing state.
- Compose Stack → ASR/TTS Service: defines · Compose files declare the ASR/TTS container.
- Compose Stack → Video Synthesis: defines · Compose files declare the video synthesis container.
- Compose Stack → Companion Service: defines · Compose files declare the remaining companion service.
- FFmpeg Assets → FFmpeg Util: bundles · Packaged binaries and models support local media processing.
- Desktop Packaging → Main Bootstrap: packages · Desktop build config packages the Electron app.
- Desktop Packaging → Vue UI: packages · Build config includes the Vue renderer bundle.
