How caladog/heygem works
Heygem is a two-part system: a desktop Electron client and a set of local Docker-based AI services. The repository mostly contains the desktop app plus deployment manifests for the companion services, not a standalone cloud backend.
High-level architecture
• Desktop application:
- Built with Electron.
- Renderer UI is a Vue application.
- Main process owns local orchestration, filesystem access, persistence, and integration with media/AI services.
- Preload script bridges renderer requests into the main process.
• Local AI services:
- Started separately with Docker Compose from the deploy/ directory.
- README indicates three local services are expected to run on 127.0.0.1, including:
- ASR / speech preprocessing and TTS service on port 18180
- Video synthesis service on port 8383
- A third companion service in the Docker stack for the full workflow
- These services are the heavy compute boundary; the Electron app coordinates them but does not implement the ML models itself.
Main subsystems in the repo
1. Electron main process layer
• Entry point: src/main/index.js
• Responsibilities:
- Application bootstrap
- Wiring handlers, services, configuration, and logging
- Coordinating media jobs and talking to local HTTP services
- Managing local files and domain state
• Important subfolders:
- src/main/handlers/
- IPC or app-facing request handlers
- app.js, file.js, index.js suggest app control and file operations are exposed here
- src/main/service/
- Domain/business logic
- model.js, video.js, voice.js map directly to the three user workflows described in the README
- context.js suggests shared runtime/session state
- src/main/dao/
- Data access layer for domain records
- Separate domain objects for context, f2f-model, video, and voice
- src/main/db/
- Local database bootstrap and SQL definitions
- This is the app’s persistence boundary for metadata and workflow state
- src/main/api/
- HTTP client wrappers for external/local service calls
- request.js is likely the shared transport layer
- f2f.js and tts.js reflect specific service integrations
- src/main/util/ffmpeg.js
- Wrapper around bundled FFmpeg tooling for media manipulation
- src/main/interval/interval.js
- Background polling/scheduling, likely for job progress or state refresh
- src/main/config/config.js
- Central configuration
- src/main/logger.js
- Shared logging
2. Preload boundary
• src/preload/index.js
• This is the security boundary between renderer and main:
- Exposes a limited API surface to the UI
- Prevents direct Node.js / filesystem access from the renderer
- Typically used for IPC calls into app handlers and services
3. Renderer UI layer
• Root UI entry:
- src/renderer/index.html
- src/renderer/src/main.js
- src/renderer/src/App.vue
• UI framework and state:
- Vue application with routing and state management
- src/renderer/src/router/
- src/renderer/src/stores/
- app.js, home.js, user.js
- src/renderer/src/i18n/
- English and Chinese support
- Matches the README’s multilingual client note
• Major feature views:
- src/renderer/src/views/home/
- Dashboard-style landing area
- Contains model lists, works lists, video dialogs, and banner content
- src/renderer/src/views/account/
- User/account related page
- src/renderer/src/views/video-edit/
- Multi-step video editing/synthesis workflow
- Split into select, edit, preview, and header
- src/renderer/src/components/model-create/
- Dedicated model creation workflow
- Includes upload, guidance, and modal-based steps
• UI support:
- Shared components like headers, dialogs, menus, and agreement screens
- Assets and theme CSS under src/renderer/src/assets/
- src/renderer/src/api/ and src/renderer/src/client/
- Frontend-side API wrappers that talk to the main-process bridge
4. Media and model runtime assets
• resources/ffmpeg/
- Bundled FFmpeg binaries for Linux and Windows
- Includes ffmpeg, ffprobe, qt-faststart, headers/libs, presets, and model files
- This is a major operational dependency for audio/video preprocessing and packaging
• The resources/ffmpeg/linux-amd64/model/ tree contains many VMAF-related and quality-evaluation model files
- Indicates the app ships with video-quality / analysis model assets needed at runtime
5. Packaging and deployment
• electron-builder.yml, electron.vite.config.mjs, build/
- Desktop packaging and OS-specific installer configuration
- macOS entitlements are present, so the app targets multiple desktop platforms
• deploy/
- Docker Compose manifests:
- docker-compose.yml
- docker-compose-lite.yml
- docker-compose-linux.yml
- These define the local AI service stack that the desktop client depends on
• dev-app-update.yml
- Update/dev packaging support
Core data flows
1. Model training / voice cloning flow
• User starts in the renderer UI, typically from the model creation area.
• UI sends a request through the preload bridge into the main process.
• Main process service logic prepares files and metadata, then calls the local ASR/TTS Docker service.
• The README describes a two-step workflow:
- preprocess audio and transcribe it via preprocess_and_tran
- synthesize speech via invoke
• Returned artifacts include audio URLs/paths and reference text.
• DAO/database layers persist the model/voice metadata so the UI can later list and reuse it.
2. Video synthesis flow
• User selects source video, generated or uploaded audio, and task parameters in the video editor UI.
• Main service calls the local video synthesis endpoint on 8383:
- submit job
- poll progress with a query endpoint
• FFmpeg tooling is available in the app runtime for local media handling, such as format conversion or preprocessing.
• The resulting output is saved locally and surfaced back to the UI for preview/download.
3. Content browsing and project management
• Home/dashboard screens list available models, works, and related artifacts.
• The renderer reads and updates app state through stores.
• Main-process services read/write to the local database and filesystem, keeping UI state and persisted metadata in sync.
Important boundaries
• UI boundary:
- The renderer is presentation only.
- It should be treated as a thin client that drives workflows and shows results.
• IPC boundary:
- The preload layer is the controlled bridge between browser-like UI code and privileged desktop capabilities.
• Orchestration boundary:
- The Electron main process is the central coordinator for local jobs, persistence, and service calls.
• Compute boundary:
- The Dockerized services own the expensive ML inference and synthesis workloads.
- This repo does not embed the model execution logic in the Electron UI; it integrates with those services over localhost HTTP.
• Storage boundary:
- User data, generated media, and model artifacts are stored locally on disk.
- The README emphasizes local directories such as D:\heygem_data\... for service-side data.
What to show in an architecture diagram
• A desktop Electron/Vue client as the front end
• A preload/IPC boundary into the Electron main process
• Main-process services for model, voice, and video workflows
• A local SQL persistence layer for app state and metadata
• Local filesystem/media processing with bundled FFmpeg
• External-but-local Docker services on 127.0.0.1 for ASR, TTS, and video synthesis
• Docker Compose deployment manifests as the operational setup for those services
In short, Heygem is an offline-first desktop orchestration app: the Electron client handles UX and workflow coordination, while Dockerized local AI services do the heavy inference and synthesis work.
Main entry points: handlers/index.js IPC Handlers · router/index.js State & Routes · service/index.js Domain Services · request.js Local APIs
Components 21
Desktop App
Electron client with renderer, preload bridge, and main-process orchestration.
- Vue UI · Presentation layer for home, account, model creation, and video editing.
src/renderer/src/App.vue - State & Routes · Client routing, stores, and language support.
src/renderer/src/router/index.js - Home · Dashboard for models, works, and workflow entry points.
src/renderer/src/views/home/index.vue - Model Create · UI flow for creating and uploading a model/voice profile.
src/renderer/src/components/model-create/ModelCreateView.vue - Video Edit · Step-based video selection, editing, and preview flow.
src/renderer/src/views/video-edit/VideoEditView.vue - Account · User/account page and related client state.
src/renderer/src/views/account/index.vue - Preload Bridge · Controlled API surface between renderer and privileged main process.
src/preload/index.js - Main Bootstrap · Application entry, wiring, and top-level orchestration.
src/main/index.js - IPC Handlers · App- and file-facing handlers exposed to the renderer.
src/main/handlers/index.js - Domain Services · Model, voice, video, and shared context workflows.
src/main/service/index.js - Local APIs · HTTP wrappers for localhost AI service calls.
src/main/api/request.js - DB Layer · Local database bootstrap and SQL definitions for app state.
src/main/db/index.js - DAOs · Domain records for context, model, video, and voice metadata.
src/main/dao/context.js - FFmpeg Util · Bundled FFmpeg wrapper for media preprocessing and packaging.
src/main/util/ffmpeg.js - Polling Loop · Background interval for progress refresh and job coordination.
src/main/interval/interval.js
Local AI Stack
Docker-based companion services and runtime media assets on localhost.
- ASR/TTS Service · Local speech preprocessing, transcription, and TTS boundary on 127.0.0.1:18180.
deploy/docker-compose.yml - Video Synthesis · Local video generation service on 127.0.0.1:8383.
deploy/docker-compose.yml - Companion Service · Third local service in the Docker stack supporting the full workflow.
deploy/docker-compose.yml - FFmpeg Assets · Packaged FFmpeg executables, presets, and quality-model assets.
resources/ffmpeg/linux-amd64/ffmpeg
Packaging & Deploy
Build, installer, and service deployment manifests.
- Compose Stack · Docker Compose definitions for the local AI services.
deploy/docker-compose.yml - Desktop Packaging · Electron builder and vite packaging for desktop installers.
electron-builder.yml
Connections 28
- Vue UI → State & Routes: uses · UI routes and shared store state drive the app shell.
- State & Routes → Home: navigates · State and routing surface the home dashboard.
- State & Routes → Model Create: navigates · State and routing surface model creation flows.
- State & Routes → Video Edit: navigates · State and routing surface video editing flows.
- State & Routes → Account: navigates · State and routing surface account views.
- Vue UI → Preload Bridge: IPC · Renderer sends privileged requests through the preload boundary.
- Preload Bridge → IPC Handlers: bridges · Preload exposes a limited API into main-process handlers.
- Main Bootstrap → IPC Handlers: registers · Main process wires the handler surface at startup.
- Main Bootstrap → Domain Services: orchestrates · Bootstrap initializes domain services and app coordination.
- Main Bootstrap → DB Layer: opens · Main process initializes local persistence.
- Domain Services → Local APIs: calls · Workflow services invoke local HTTP integrations.
- Domain Services → DAOs: persists · Services store and load workflow metadata through DAOs.
- DAOs → DB Layer: queries · DAO operations map to the local SQL database.
- Domain Services → FFmpeg Util: processes · Media workflows use bundled FFmpeg tooling.
- Domain Services → Polling Loop: polls · Background polling tracks async job progress.
- Local APIs → ASR/TTS Service: HTTP · Speech preprocessing and TTS are served locally.
- Local APIs → Video Synthesis: HTTP · Video synthesis jobs are submitted to the local service.
- Local APIs → Companion Service: HTTP · Workflow orchestration also depends on the companion service.
- Home → IPC Handlers: requests · Home screens load models and works via handler calls.
- Model Create → IPC Handlers: requests · Model creation submits preprocessing and synthesis actions.
- Video Edit → IPC Handlers: requests · Video editing submits media jobs and progress queries.
- Account → IPC Handlers: requests · Account screens read and update app-facing state.
- Compose Stack → ASR/TTS Service: defines · Compose files declare the ASR/TTS container.
- Compose Stack → Video Synthesis: defines · Compose files declare the video synthesis container.
- Compose Stack → Companion Service: defines · Compose files declare the remaining companion service.
- FFmpeg Assets → FFmpeg Util: bundles · Packaged binaries and models support local media processing.
- Desktop Packaging → Main Bootstrap: packages · Desktop build config packages the Electron app.
- Desktop Packaging → Vue UI: packages · Build config includes the Vue renderer bundle.