# sharpai/deepcamera architecture

> Architecture diagram of the GitHub repository sharpai/deepcamera, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-04-09). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sharpai/deepcamera
- Repository: https://github.com/sharpai/deepcamera

## Overview

1. **Repo shape: platform + skill catalog + legacy apps**
   DeepCamera is not a single application; it is a platform made of many self-contained “skills” plus older standalone services under `src/`. The modern layer is the `skills/` tree and `skills.json` registry. The legacy layer includes camera monitoring, face recognition, detection pipelines, and several domain-specific apps.

2. **Skill boundary and contract**
   Each skill is packaged as an independent module with its own `SKILL.md`, runtime dependencies, model assets, config, and launch script. The key architectural rule is the shared JSONL stdin/stdout protocol: Aegis writes a frame event, the skill reads it, runs inference, and emits detections back on stdout. That makes detection/segmentation skills interchangeable from the host’s perspective.

3. **Shared hardware abstraction**
   `skills/lib/env_config.py` is the main cross-skill hardware selector. It detects the available accelerator and routes execution to the right backend: TensorRT for NVIDIA, CoreML for Apple Silicon, OpenVINO for Intel, ROCm/ONNX Runtime for AMD, CPU fallback otherwise. This is a major boundary because model conversion and runtime selection are centralized instead of duplicated per skill.

4. **Modern skill families**
   The `skills/` tree is organized by function: detection, segmentation, transformation/privacy, annotation, training, automation, integrations, camera providers, channels, streaming, and analysis. Most of these are thin wrappers around a model or external service, but they all follow the same “skill as a unit of deployment” pattern.

5. **Detection/segmentation/transform pipeline**
   The modern runtime is built around visual processing skills such as `yolo-detection-2026`, Coral TPU, OpenVINO, `sam2-segmentation`, and `depth-estimation`. The data flow is: camera frame → shared file/volume → skill inference → detections or transformed frames → overlay/alerts. In the readme, this is explicitly rate-limited and parallelized so it does not block the agent/UI path.

6. **Legacy camera application core**
   `src/camera/` is the older monolithic CCTV/NVR-style app. It is a Node.js service with EJS pages, client-side JS/CSS, SQL schemas, and a plugin system. The detector side lives in `src/camera/detector/`, which handles video processing, motion/deepeye logic, uploads, realtime messages, MQTT GIF hooks, and timeline/event generation.

7. **Plugin-based video analytics in the legacy app**
   The camera app extends via plugins under `src/camera/src/plugins/` for motion, OpenALPR, OpenCV, Microsoft vision, and child process delegation. This is a clear extension boundary: core camera ingestion and UI stay in the main app, while specific analytics are delegated to plugins.

8. **Face recognition and embedding stack**
   `src/embedding/` is a substantial Python subsystem for facial embeddings, classification, clustering, parameter serving, database migration, and upload/storage helpers. It includes model definitions, training utilities, a parameter server, REST client/server pieces, and face-processing code. This is the repo’s main face-recognition pipeline, separate from the newer generic skills catalog.

9. **Face detection as an async worker system**
   `src/face_detection/` is another Python/Celery-based detection stack. It contains detector/worker entrypoints, model binaries, preprocessing/filtering code, and a separate `assets/` build of the same functionality for packaged deployment. The presence of `celeryconfig`, `worker.py`, and `startq.py` shows it is designed as a queued background service rather than an in-process library.

10. **Specialized standalone applications**
   There are several application-specific pipelines under `src/` and `skills/` for person re-identification, fall detection, parking detection, laptop monitoring, and YOLOv7-based person detection/ReID. These are usually packaged as their own Docker images or compose stacks, with their own models and test harnesses, which makes them deployment units rather than shared libraries.

11. **Automation and external output channels**
   Detection results are not just displayed locally; they can be forwarded to Home Assistant, MQTT, webhooks, Telegram/Slack-like channels, Matrix, Signal, or LINE depending on the skill. The architecture therefore includes an event-export layer that turns camera inference into automation triggers and notifications.

12. **Infrastructure services bundled with the repo**
   The `docker/` directory provides reusable runtime services: broker, Redis, MinIO, Flower, TensorFlow, OD, and v4l2rtspserver. These are the supporting services for async execution, storage, and model serving/processing. Multiple Dockerfiles and compose variants exist for x86, arm64, and Jetson/L4T, so hardware-specific deployment is part of the architecture.

13. **Build and packaging boundaries**
   `src/build/` contains PyInstaller specs, shell wrappers, and architecture-specific build scripts for the services. This indicates the repo ships both containerized and packaged binary deployments, with separate build paths for detectors, monitors, parameter servers, embedding services, and ARM/AArch64 variants.

14. **Model lifecycle and validation tooling**
   The repo also includes training, annotation, benchmarking, and regression suites under `skills/analysis`, `skills/annotation`, `skills/training`, and `src/unit_test/`. These are not runtime services; they validate model quality, measure security/scene-understanding behavior, and support dataset/model iteration before deployment.

Main entry points: [`skills/detection/yolo-detection-2026/scripts/detect.py`](https://github.com/sharpai/deepcamera/blob/master/skills/detection/yolo-detection-2026/scripts/detect.py) (YOLO detect), [`skills.json`](https://github.com/sharpai/deepcamera/blob/master/skills.json) (Skill registry), [`docs/detection-protocol.md`](https://github.com/sharpai/deepcamera/blob/master/docs/detection-protocol.md) (Skill contract), [`skills/lib/env_config.py`](https://github.com/sharpai/deepcamera/blob/master/skills/lib/env_config.py) (Hardware router)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_modern["Modern skills"]
  node_skills_registry["Skill registry<br/>catalog<br/>[skills.json]"]
  node_skill_contract["Skill contract<br/>protocol"]
  node_env_config["Hardware router<br/>runtime selector<br/>[env_config.py]"]
  node_yolo_detect["YOLO detect<br/>detection skill<br/>[detect.py]"]
  node_openvino_detect["OpenVINO detect<br/>detection skill<br/>[detect.py]"]
  node_coral_detect["Coral detect<br/>detection skill<br/>[detect.py]"]
  node_segment_skill["SAM2 segment<br/>segmentation skill<br/>[segment.py]"]
  node_depth_skill["Depth transform<br/>transform skill<br/>[transform.py]"]
  node_ha_bridge["HA bridge<br/>integration skill<br/>[bridge.py]"]
  node_mqtt_skill["MQTT publish<br/>automation skill<br/>[mqtt_publish.py]"]
  node_webhook_skill["Webhook emit<br/>automation skill<br/>[webhook.py]"]
  node_camera_provider["Camera feeds<br/>provider skill<br/>[feed.py]"]
  node_stream_skill["go2rtc stream<br/>streaming skill<br/>[stream.py]"]
  node_benchmark_suite["Benchmarks<br/>evaluation<br/>[run-benchmark.cjs]"]
end

subgraph group_legacy["Legacy apps"]
  node_camera_app["Camera app<br/>legacy nodejs app<br/>[camera.js]"]
  node_camera_plugins["Camera plugins<br/>analytics extensions<br/>[shinobi-motion.js]"]
  node_camera_detector["Camera detector<br/>video pipeline<br/>[index.js]"]
  node_embedding_stack["Embedding stack<br/>face recognition"]
  node_face_detection_stack["Face detection<br/>queued worker<br/>[worker.py]"]
  node_specialized_apps["Specialized apps<br/>domain pipelines<br/>[detector.py]"]
end

subgraph group_infra["Infrastructure"]
  node_docker_runtime["Docker stack<br/>runtime services<br/>[docker-compose.yml]"]
  node_build_packaging["Build tools<br/>packaging<br/>[build_detector.sh]"]
end

node_skills_registry -->|"declares"| node_skill_contract
node_skills_registry -->|"includes"| node_yolo_detect
node_skills_registry -->|"includes"| node_openvino_detect
node_skills_registry -->|"includes"| node_coral_detect
node_skills_registry -->|"includes"| node_segment_skill
node_skills_registry -->|"includes"| node_depth_skill
node_skill_contract -->|"runtime API"| node_yolo_detect
node_skill_contract -->|"runtime API"| node_segment_skill
node_skill_contract -->|"runtime API"| node_depth_skill
node_env_config -->|"selects backend"| node_yolo_detect
node_env_config -->|"selects backend"| node_openvino_detect
node_env_config -->|"selects backend"| node_coral_detect
node_camera_provider -->|"feeds"| node_stream_skill
node_stream_skill -->|"frames"| node_yolo_detect
node_yolo_detect -->|"hands off"| node_segment_skill
node_yolo_detect -->|"hands off"| node_depth_skill
node_yolo_detect -->|"alerts"| node_ha_bridge
node_yolo_detect -->|"alerts"| node_mqtt_skill
node_yolo_detect -->|"alerts"| node_webhook_skill
node_camera_app -->|"extends via"| node_camera_plugins
node_camera_app -->|"drives"| node_camera_detector
node_camera_detector -->|"uses"| node_camera_plugins
node_embedding_stack -->|"depends on"| node_face_detection_stack
node_specialized_apps -->|"deploys on"| node_docker_runtime
node_face_detection_stack -->|"deploys on"| node_docker_runtime
node_embedding_stack -->|"packages with"| node_build_packaging
node_camera_app -->|"packages with"| node_build_packaging
node_benchmark_suite -->|"validates"| node_skill_contract

click node_skills_registry "https://github.com/sharpai/deepcamera/blob/master/skills.json"
click node_skill_contract "https://github.com/sharpai/deepcamera/blob/master/docs/detection-protocol.md"
click node_env_config "https://github.com/sharpai/deepcamera/blob/master/skills/lib/env_config.py"
click node_yolo_detect "https://github.com/sharpai/deepcamera/blob/master/skills/detection/yolo-detection-2026/scripts/detect.py"
click node_openvino_detect "https://github.com/sharpai/deepcamera/blob/master/skills/detection/yolo-detection-2026-openvino/scripts/detect.py"
click node_coral_detect "https://github.com/sharpai/deepcamera/blob/master/skills/detection/yolo-detection-2026-coral-tpu-macos/scripts/detect.py"
click node_segment_skill "https://github.com/sharpai/deepcamera/blob/master/skills/segmentation/sam2-segmentation/scripts/segment.py"
click node_depth_skill "https://github.com/sharpai/deepcamera/blob/master/skills/transformation/depth-estimation/scripts/transform.py"
click node_ha_bridge "https://github.com/sharpai/deepcamera/blob/master/skills/integrations/homeassistant-bridge/scripts/bridge.py"
click node_mqtt_skill "https://github.com/sharpai/deepcamera/blob/master/skills/automation/mqtt/scripts/mqtt_publish.py"
click node_webhook_skill "https://github.com/sharpai/deepcamera/blob/master/skills/automation/webhook/scripts/webhook.py"
click node_camera_provider "https://github.com/sharpai/deepcamera/blob/master/skills/camera-providers/reolink/scripts/feed.py"
click node_stream_skill "https://github.com/sharpai/deepcamera/blob/master/skills/streaming/go2rtc-cameras/scripts/stream.py"
click node_benchmark_suite "https://github.com/sharpai/deepcamera/blob/master/skills/analysis/home-security-benchmark/scripts/run-benchmark.cjs"
click node_camera_app "https://github.com/sharpai/deepcamera/blob/master/src/camera/src/camera.js"
click node_camera_plugins "https://github.com/sharpai/deepcamera/blob/master/src/camera/src/plugins/motion/shinobi-motion.js"
click node_camera_detector "https://github.com/sharpai/deepcamera/blob/master/src/camera/detector/index.js"
click node_embedding_stack "https://github.com/sharpai/deepcamera/blob/master/src/embedding/embedding_server.py"
click node_face_detection_stack "https://github.com/sharpai/deepcamera/blob/master/src/face_detection/worker.py"
click node_specialized_apps "https://github.com/sharpai/deepcamera/blob/master/src/fall_detection/src/detector.py"
click node_docker_runtime "https://github.com/sharpai/deepcamera/blob/master/docker/docker-compose.yml"
click node_build_packaging "https://github.com/sharpai/deepcamera/blob/master/src/build/build_detector.sh"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_skills_registry,node_skill_contract,node_env_config,node_yolo_detect,node_openvino_detect,node_coral_detect,node_segment_skill,node_depth_skill,node_ha_bridge,node_mqtt_skill,node_webhook_skill,node_camera_provider,node_stream_skill,node_benchmark_suite toneBlue
class node_camera_app,node_camera_plugins,node_camera_detector,node_embedding_stack,node_face_detection_stack,node_specialized_apps toneAmber
class node_docker_runtime,node_build_packaging toneMint
```

## Components

### Modern skills

Skill catalog and shared runtime used by the newer modular pipeline.

- [Skill registry](https://github.com/sharpai/deepcamera/blob/master/skills.json): catalog · Top-level skill manifest and deployment catalog for the modular layer. · `skills.json`
- [Skill contract](https://github.com/sharpai/deepcamera/blob/master/docs/detection-protocol.md): protocol · Shared stdin/stdout JSONL frame protocol that makes skills interchangeable. · `docs/detection-protocol.md`
- [Hardware router](https://github.com/sharpai/deepcamera/blob/master/skills/lib/env_config.py): runtime selector · Cross-skill accelerator detection and backend routing for CUDA, MPS/CoreML, OpenVINO, ROCm, TensorRT, and CPU. · `skills/lib/env_config.py`
- [YOLO detect](https://github.com/sharpai/deepcamera/blob/master/skills/detection/yolo-detection-2026/scripts/detect.py): detection skill · Main modern object-detection implementation for frame-based inference. · `skills/detection/yolo-detection-2026/scripts/detect.py`
- [OpenVINO detect](https://github.com/sharpai/deepcamera/blob/master/skills/detection/yolo-detection-2026-openvino/scripts/detect.py): detection skill · Intel-oriented detection backend packaged as a separate deployable skill. · `skills/detection/yolo-detection-2026-openvino/scripts/detect.py`
- [Coral detect](https://github.com/sharpai/deepcamera/blob/master/skills/detection/yolo-detection-2026-coral-tpu-macos/scripts/detect.py): detection skill · EdgeTPU/Coral-packaged detection variant for TPU-backed execution. · `skills/detection/yolo-detection-2026-coral-tpu-macos/scripts/detect.py`
- [SAM2 segment](https://github.com/sharpai/deepcamera/blob/master/skills/segmentation/sam2-segmentation/scripts/segment.py): segmentation skill · Visual segmentation stage used in the modern processing pipeline. · `skills/segmentation/sam2-segmentation/scripts/segment.py`
- [Depth transform](https://github.com/sharpai/deepcamera/blob/master/skills/transformation/depth-estimation/scripts/transform.py): transform skill · Frame transformation skill for depth estimation and related visual effects. · `skills/transformation/depth-estimation/scripts/transform.py`
- [HA bridge](https://github.com/sharpai/deepcamera/blob/master/skills/integrations/homeassistant-bridge/scripts/bridge.py): integration skill · Forwards detections into Home Assistant automation. · `skills/integrations/homeassistant-bridge/scripts/bridge.py`
- [MQTT publish](https://github.com/sharpai/deepcamera/blob/master/skills/automation/mqtt/scripts/mqtt_publish.py): automation skill · Publishes detection events onto MQTT for downstream consumers. · `skills/automation/mqtt/scripts/mqtt_publish.py`
- [Webhook emit](https://github.com/sharpai/deepcamera/blob/master/skills/automation/webhook/scripts/webhook.py): automation skill · Sends detection events to generic webhook endpoints. · `skills/automation/webhook/scripts/webhook.py`
- [Camera feeds](https://github.com/sharpai/deepcamera/blob/master/skills/camera-providers/reolink/scripts/feed.py): provider skill · Camera-provider adapters for vendor-specific feeds and stream acquisition. · `skills/camera-providers/reolink/scripts/feed.py`
- [go2rtc stream](https://github.com/sharpai/deepcamera/blob/master/skills/streaming/go2rtc-cameras/scripts/stream.py): streaming skill · Streaming adapter that bridges camera inputs into the skill pipeline. · `skills/streaming/go2rtc-cameras/scripts/stream.py`
- [Benchmarks](https://github.com/sharpai/deepcamera/blob/master/skills/analysis/home-security-benchmark/scripts/run-benchmark.cjs): evaluation · Benchmarking and regression tools for model quality and scenario validation. · `skills/analysis/home-security-benchmark/scripts/run-benchmark.cjs`

### Legacy apps

Older standalone services and application-specific pipelines under src/.

- [Camera app](https://github.com/sharpai/deepcamera/blob/master/src/camera/src/camera.js): legacy nodejs app · Older monolithic CCTV/NVR-style application with web UI, SQL schema, and detector service. · `src/camera/src/camera.js`
- [Camera plugins](https://github.com/sharpai/deepcamera/blob/master/src/camera/src/plugins/motion/shinobi-motion.js): analytics extensions · Plugin boundary for motion, OpenALPR, OpenCV, Microsoft vision, and child-process delegation. · `src/camera/src/plugins/motion/shinobi-motion.js`
- [Camera detector](https://github.com/sharpai/deepcamera/blob/master/src/camera/detector/index.js): video pipeline · Legacy realtime detection pipeline handling motion, deepeye logic, uploads, MQTT GIF hooks, and timeline events. · `src/camera/detector/index.js`
- [Embedding stack](https://github.com/sharpai/deepcamera/blob/master/src/embedding/embedding_server.py): face recognition · Python facial-embedding, classification, clustering, parameter-serving, and storage subsystem. · `src/embedding/embedding_server.py`
- [Face detection](https://github.com/sharpai/deepcamera/blob/master/src/face_detection/worker.py): queued worker · Celery-based face detection service with worker and queue entrypoints. · `src/face_detection/worker.py`
- [Specialized apps](https://github.com/sharpai/deepcamera/blob/master/src/fall_detection/src/detector.py): domain pipelines · Standalone deployment units for fall detection, parking, laptop monitoring, and YOLOv7 person/ReID workflows. · `src/fall_detection/src/detector.py`

### Infrastructure

Containerized runtime services and packaging/build support.

- [Docker stack](https://github.com/sharpai/deepcamera/blob/master/docker/docker-compose.yml): runtime services · Reusable broker, Redis, MinIO, Flower, TensorFlow, and camera-serving containers plus compose variants. · `docker/docker-compose.yml`
- [Build tools](https://github.com/sharpai/deepcamera/blob/master/src/build/build_detector.sh): packaging · PyInstaller specs, shell wrappers, and architecture-specific build scripts for packaged deployment. · `src/build/build_detector.sh`

## Connections

- Skill registry → Skill contract: declares · The registry and docs define the shared skill interface.
- Skill registry → YOLO detect: includes · Registry points to deployable detection skills.
- Skill registry → OpenVINO detect: includes · Registry points to deployable detection skills.
- Skill registry → Coral detect: includes · Registry points to deployable detection skills.
- Skill registry → SAM2 segment: includes · Registry points to deployable processing skills.
- Skill registry → Depth transform: includes · Registry points to deployable processing skills.
- Skill contract → YOLO detect: runtime API · Detection skills consume and emit frame events through the protocol.
- Skill contract → SAM2 segment: runtime API · Processing skills use the same stdin/stdout contract.
- Skill contract → Depth transform: runtime API · Processing skills use the same stdin/stdout contract.
- Hardware router → YOLO detect: selects backend · Centralized accelerator selection configures the detector runtime.
- Hardware router → OpenVINO detect: selects backend · Hardware routing chooses the Intel path when available.
- Hardware router → Coral detect: selects backend · Hardware routing chooses the TPU path when available.
- Camera feeds → go2rtc stream: feeds · Vendor camera adapters and stream adapters supply frames.
- go2rtc stream → YOLO detect: frames · Streams provide the frame source for inference skills.
- YOLO detect → SAM2 segment: hands off · Detection can flow into segmentation for follow-up processing.
- YOLO detect → Depth transform: hands off · Detection can flow into depth/transformation processing.
- YOLO detect → HA bridge: alerts · Detections are forwarded to home-automation outputs.
- YOLO detect → MQTT publish: alerts · Detections can be published to MQTT topics.
- YOLO detect → Webhook emit: alerts · Detections can be emitted to webhook integrations.
- Camera app → Camera plugins: extends via · Legacy app delegates analytics to plugin modules.
- Camera app → Camera detector: drives · Core app coordinates the legacy detector service.
- Camera detector → Camera plugins: uses · Legacy detector logic is augmented by plugin-based analytics.
- Embedding stack → Face detection: depends on · Face recognition pipeline consumes detected faces and related crops.
- Specialized apps → Docker stack: deploys on · Standalone pipelines are packaged into container stacks.
- Face detection → Docker stack: deploys on · Worker-based detection service is containerized.
- Embedding stack → Build tools: packages with · Embedding subsystem has dedicated build and packaging scripts.
- Camera app → Build tools: packages with · Legacy camera services are also shipped through build scripts/specs.
- Benchmarks → Skill contract: validates · Benchmark tools exercise the same event/detection contract.
