valentinfrlch/ha-llmvision

Purpose and Host Runtime

LLM Vision is a Python custom integration for Home Assistant, installed through HACS. It analyzes camera images, videos, live feeds, and Frigate events with multimodal LLMs; it can retain recognized entities, update Home Assistant state, and publish event history. The integration boundary and Home Assistant lifecycle entry point are custom_components/llmvision/__init__.py and custom_components/llmvision/manifest.json.

Setup, Configuration, and Service Surface

Home Assistant config entries drive initial setup and provider creation through custom_components/llmvision/config_flow.py. The integration exposes its user-facing automation/service contract in custom_components/llmvision/services.yaml; constants and identifiers shared across modules reside in custom_components/llmvision/const.py. Blueprints provide a ready-made event-summary automation: blueprints/event_summary.yaml and blueprints/event_summary_beta.yaml.

Request Orchestration

custom_components/llmvision/api.py is the likely central application API: it accepts analysis requests, coordinates media preparation, provider invocation, and downstream result handling. This is the principal control boundary between Home Assistant automations/services and LLM-specific implementation. custom_components/llmvision/__init__.py registers the integration-level facilities that make this API available at runtime.

Media Acquisition and Normalization

custom_components/llmvision/media_handlers.py forms the media boundary for camera feeds, images, video files, snapshots, and Frigate-originated material. It isolates format/source handling before payloads reach model providers. Persistent snapshot/media storage is expected to use Home Assistant’s /media area, an operational requirement noted by the project setup guidance.

Multi-Provider LLM Adapter Layer

custom_components/llmvision/providers.py abstracts calls to cloud, local, and OpenAI-compatible multimodal backends. Supported external services include OpenRouter, OpenAI, Anthropic, Google Gemini, AWS Bedrock, Azure, Groq, Ollama, Open WebUI, and LocalAI. Configuration selects credentials/endpoints and model behavior, while the API layer should remain provider-neutral.

Memory and Derived Home Automation Data

custom_components/llmvision/memory.py manages retained knowledge about people, pets, and objects across analyses. Analysis responses can therefore serve both immediate prompts and longer-lived contextual recognition. The service definitions in custom_components/llmvision/services.yaml are the key contract for passing extracted results into Home Assistant automations and sensor/state updates.

Timeline, Calendar, and Consumer Interfaces

custom_components/llmvision/timeline.py persists or exposes analyzed camera-event history for dashboard and Assist-style queries. custom_components/llmvision/calendar.py provides a Home Assistant calendar-facing projection of those events. The external timeline API contract is captured in timeline_openapi_specs.yaml; localized timeline UI text lives under custom_components/llmvision/timeline_strings/.

Delivery and Validation

The integration is distributed as a HACS repository (hacs.json) and relies on Home Assistant’s Python runtime and configuration-entry framework. CI validates and tests the package through .github/workflows/tests.yaml and .github/workflows/validate.yaml; the test suite is structured around the major architectural modules in tests/.