How mrtzpngrtz/vectorvision works
Project type and purpose
• VectorVision is a full-stack desktop application built on Electron.
• Purpose: local, private, GPU-accelerated ML-powered image/video exploration, clustering, tagging, and similarity search.
File structure
• Top-level files:
• package.json / package-lock.json (dependencies & scripts)
• electron-main.js (Electron Main Process entry point)
• preload.js (secure IPC bridge)
• public/ (Renderer Process assets):
• index.html, CSS, and multiple JS modules (UI, canvas-renderer, SOM worker)
• server/ (ML processing services):
• index.js (server orchestration)
• analyzer.js (ONNX model inference pipeline)
• som.js (self-organizing map clustering)
• labels.js (tag-assignment logic)
• screenshots/ (documentation)
• Optional: models/ (downloaded ML models at runtime), dist/ (packaged app)
README for architectural insights
• Electron Main Process manages windows, IPC, and launches the server code.
• Preload script exposes a limited IPC API to the Renderer.
• Renderer Process (Chromium) handles 2D/3D UI, visualization, user interactions.
• Server modules handle ONNX Runtime inference (SigLIP, DINOv2, Aesthetic Predictor), SOM clustering, and image I/O via Sharp and FFmpeg.
Main components
a. Electron Main Process
• electron-main.js
• Coordinates window lifecycle, IPCMain handlers, spawns or requires server modules
b. Preload Bridge
• preload.js
• Exposes sanitized IPC channels (invoke/receive) to the Renderer
c. Renderer Process (UI)
• public/index.html + public/js/*.js
• Canvas renderer, SOM Web Worker, UI controls, sidebars, word-cloud filter
d. ML Server / Processing Layer
• server/index.js – orchestrator
• server/analyzer.js – ONNX Runtime (DirectML/CoreML) calls
• server/som.js – clustering algorithm
• server/labels.js – semantic tagging
e. Local File System & Cache
• User’s image/video libraries on disk
• models/ cache directory for ML model artifacts
f. External Dependencies
• FFmpeg – frame extraction for videos
• ONNX Runtime – GPU-accelerated inference
• Sharp – image decoding/resizing
Relationships and data flow
1. User interacts with UI in the Renderer
2. Renderer → (IPC Renderer.invoke) → Preload → (IPC Main.handle) → Electron Main
3. Main → calls server/index.js (in-process require or child process spawn)
4. server modules read images/videos, generate thumbnails, run ONNX models
5. server emits progress and results events → Electron Main → Preload → Renderer
6. Renderer updates 2D/3D visualization, tags, and clusters in real time
Key architectural patterns and design principles
• Event-driven, asynchronous IPC for process isolation and security
• Clear separation of concerns: UI (Renderer) vs. system/ML logic (Main + server)
• Worker threads (Web Worker) for CPU-bound clustering in the Renderer
• Local-only architecture: no external APIs, 100% data privacy
• GPU offload for model inference via ONNX Runtime (DirectML/CoreML)
Relevant technologies/frameworks
• Electron (Chromium + Node.js integration)
• ONNX Runtime with DirectML/CoreML backends
• SigLIP, DINOv2, Aesthetic Predictor (ONNX models)
• Sharp (Node.js image processing)
• FFmpeg (video frame extraction)
• ml-som (JavaScript SOM implementation)
– Renderer → Preload → Main (invoke)
– Main → Server modules (function call or spawn)
– Server → Main → Renderer (event stream)
– Server → File System (read/write images, models)
• Detail calls out: model download on first run, thumbnail generation, progress events, 2D/3D clustering pipeline.