How sharpai/deepcamera works
1. Repo shape: platform + skill catalog + legacy apps
DeepCamera is not a single application; it is a platform made of many self-contained “skills” plus older standalone services under src/. The modern layer is the skills/ tree and skills.json registry. The legacy layer includes camera monitoring, face recognition, detection pipelines, and several domain-specific apps.
2. Skill boundary and contract
Each skill is packaged as an independent module with its own SKILL.md, runtime dependencies, model assets, config, and launch script. The key architectural rule is the shared JSONL stdin/stdout protocol: Aegis writes a frame event, the skill reads it, runs inference, and emits detections back on stdout. That makes detection/segmentation skills interchangeable from the host’s perspective.
3. Shared hardware abstraction
skills/lib/env_config.py is the main cross-skill hardware selector. It detects the available accelerator and routes execution to the right backend: TensorRT for NVIDIA, CoreML for Apple Silicon, OpenVINO for Intel, ROCm/ONNX Runtime for AMD, CPU fallback otherwise. This is a major boundary because model conversion and runtime selection are centralized instead of duplicated per skill.
4. Modern skill families
The skills/ tree is organized by function: detection, segmentation, transformation/privacy, annotation, training, automation, integrations, camera providers, channels, streaming, and analysis. Most of these are thin wrappers around a model or external service, but they all follow the same “skill as a unit of deployment” pattern.
5. Detection/segmentation/transform pipeline
The modern runtime is built around visual processing skills such as yolo-detection-2026, Coral TPU, OpenVINO, sam2-segmentation, and depth-estimation. The data flow is: camera frame → shared file/volume → skill inference → detections or transformed frames → overlay/alerts. In the readme, this is explicitly rate-limited and parallelized so it does not block the agent/UI path.
6. Legacy camera application core
src/camera/ is the older monolithic CCTV/NVR-style app. It is a Node.js service with EJS pages, client-side JS/CSS, SQL schemas, and a plugin system. The detector side lives in src/camera/detector/, which handles video processing, motion/deepeye logic, uploads, realtime messages, MQTT GIF hooks, and timeline/event generation.
7. Plugin-based video analytics in the legacy app
The camera app extends via plugins under src/camera/src/plugins/ for motion, OpenALPR, OpenCV, Microsoft vision, and child process delegation. This is a clear extension boundary: core camera ingestion and UI stay in the main app, while specific analytics are delegated to plugins.
8. Face recognition and embedding stack
src/embedding/ is a substantial Python subsystem for facial embeddings, classification, clustering, parameter serving, database migration, and upload/storage helpers. It includes model definitions, training utilities, a parameter server, REST client/server pieces, and face-processing code. This is the repo’s main face-recognition pipeline, separate from the newer generic skills catalog.
9. Face detection as an async worker system
src/face_detection/ is another Python/Celery-based detection stack. It contains detector/worker entrypoints, model binaries, preprocessing/filtering code, and a separate assets/ build of the same functionality for packaged deployment. The presence of celeryconfig, worker.py, and startq.py shows it is designed as a queued background service rather than an in-process library.
10. Specialized standalone applications
There are several application-specific pipelines under src/ and skills/ for person re-identification, fall detection, parking detection, laptop monitoring, and YOLOv7-based person detection/ReID. These are usually packaged as their own Docker images or compose stacks, with their own models and test harnesses, which makes them deployment units rather than shared libraries.
11. Automation and external output channels
Detection results are not just displayed locally; they can be forwarded to Home Assistant, MQTT, webhooks, Telegram/Slack-like channels, Matrix, Signal, or LINE depending on the skill. The architecture therefore includes an event-export layer that turns camera inference into automation triggers and notifications.
12. Infrastructure services bundled with the repo
The docker/ directory provides reusable runtime services: broker, Redis, MinIO, Flower, TensorFlow, OD, and v4l2rtspserver. These are the supporting services for async execution, storage, and model serving/processing. Multiple Dockerfiles and compose variants exist for x86, arm64, and Jetson/L4T, so hardware-specific deployment is part of the architecture.
13. Build and packaging boundaries
src/build/ contains PyInstaller specs, shell wrappers, and architecture-specific build scripts for the services. This indicates the repo ships both containerized and packaged binary deployments, with separate build paths for detectors, monitors, parameter servers, embedding services, and ARM/AArch64 variants.
14. Model lifecycle and validation tooling
The repo also includes training, annotation, benchmarking, and regression suites under skills/analysis, skills/annotation, skills/training, and src/unit_test/. These are not runtime services; they validate model quality, measure security/scene-understanding behavior, and support dataset/model iteration before deployment.
Main entry points: detect.py YOLO detect · skills.json Skill registry · detection-protocol.md Skill contract · env_config.py Hardware router
Components 22
Modern skills
Skill catalog and shared runtime used by the newer modular pipeline.
- Skill registry · Top-level skill manifest and deployment catalog for the modular layer.
skills.json - Skill contract · Shared stdin/stdout JSONL frame protocol that makes skills interchangeable.
docs/detection-protocol.md - Hardware router · Cross-skill accelerator detection and backend routing for CUDA, MPS/CoreML, OpenVINO, ROCm, TensorRT, and CPU.
skills/lib/env_config.py - YOLO detect · Main modern object-detection implementation for frame-based inference.
skills/detection/yolo-detection-2026/scripts/detect.py - OpenVINO detect · Intel-oriented detection backend packaged as a separate deployable skill.
skills/detection/yolo-detection-2026-openvino/scripts/detect.py - Coral detect · EdgeTPU/Coral-packaged detection variant for TPU-backed execution.
skills/detection/yolo-detection-2026-coral-tpu-macos/scripts/detect.py - SAM2 segment · Visual segmentation stage used in the modern processing pipeline.
skills/segmentation/sam2-segmentation/scripts/segment.py - Depth transform · Frame transformation skill for depth estimation and related visual effects.
skills/transformation/depth-estimation/scripts/transform.py - HA bridge · Forwards detections into Home Assistant automation.
skills/integrations/homeassistant-bridge/scripts/bridge.py - MQTT publish · Publishes detection events onto MQTT for downstream consumers.
skills/automation/mqtt/scripts/mqtt_publish.py - Webhook emit · Sends detection events to generic webhook endpoints.
skills/automation/webhook/scripts/webhook.py - Camera feeds · Camera-provider adapters for vendor-specific feeds and stream acquisition.
skills/camera-providers/reolink/scripts/feed.py - go2rtc stream · Streaming adapter that bridges camera inputs into the skill pipeline.
skills/streaming/go2rtc-cameras/scripts/stream.py - Benchmarks · Benchmarking and regression tools for model quality and scenario validation.
skills/analysis/home-security-benchmark/scripts/run-benchmark.cjs
Legacy apps
Older standalone services and application-specific pipelines under src/.
- Camera app · Older monolithic CCTV/NVR-style application with web UI, SQL schema, and detector service.
src/camera/src/camera.js - Camera plugins · Plugin boundary for motion, OpenALPR, OpenCV, Microsoft vision, and child-process delegation.
src/camera/src/plugins/motion/shinobi-motion.js - Camera detector · Legacy realtime detection pipeline handling motion, deepeye logic, uploads, MQTT GIF hooks, and timeline events.
src/camera/detector/index.js - Embedding stack · Python facial-embedding, classification, clustering, parameter-serving, and storage subsystem.
src/embedding/embedding_server.py - Face detection · Celery-based face detection service with worker and queue entrypoints.
src/face_detection/worker.py - Specialized apps · Standalone deployment units for fall detection, parking, laptop monitoring, and YOLOv7 person/ReID workflows.
src/fall_detection/src/detector.py
Infrastructure
Containerized runtime services and packaging/build support.
- Docker stack · Reusable broker, Redis, MinIO, Flower, TensorFlow, and camera-serving containers plus compose variants.
docker/docker-compose.yml - Build tools · PyInstaller specs, shell wrappers, and architecture-specific build scripts for packaged deployment.
src/build/build_detector.sh
Connections 28
- Skill registry → Skill contract: declares · The registry and docs define the shared skill interface.
- Skill registry → YOLO detect: includes · Registry points to deployable detection skills.
- Skill registry → OpenVINO detect: includes · Registry points to deployable detection skills.
- Skill registry → Coral detect: includes · Registry points to deployable detection skills.
- Skill registry → SAM2 segment: includes · Registry points to deployable processing skills.
- Skill registry → Depth transform: includes · Registry points to deployable processing skills.
- Skill contract → YOLO detect: runtime API · Detection skills consume and emit frame events through the protocol.
- Skill contract → SAM2 segment: runtime API · Processing skills use the same stdin/stdout contract.
- Skill contract → Depth transform: runtime API · Processing skills use the same stdin/stdout contract.
- Hardware router → YOLO detect: selects backend · Centralized accelerator selection configures the detector runtime.
- Hardware router → OpenVINO detect: selects backend · Hardware routing chooses the Intel path when available.
- Hardware router → Coral detect: selects backend · Hardware routing chooses the TPU path when available.
- Camera feeds → go2rtc stream: feeds · Vendor camera adapters and stream adapters supply frames.
- go2rtc stream → YOLO detect: frames · Streams provide the frame source for inference skills.
- YOLO detect → SAM2 segment: hands off · Detection can flow into segmentation for follow-up processing.
- YOLO detect → Depth transform: hands off · Detection can flow into depth/transformation processing.
- YOLO detect → HA bridge: alerts · Detections are forwarded to home-automation outputs.
- YOLO detect → MQTT publish: alerts · Detections can be published to MQTT topics.
- YOLO detect → Webhook emit: alerts · Detections can be emitted to webhook integrations.
- Camera app → Camera plugins: extends via · Legacy app delegates analytics to plugin modules.
- Camera app → Camera detector: drives · Core app coordinates the legacy detector service.
- Camera detector → Camera plugins: uses · Legacy detector logic is augmented by plugin-based analytics.
- Embedding stack → Face detection: depends on · Face recognition pipeline consumes detected faces and related crops.
- Specialized apps → Docker stack: deploys on · Standalone pipelines are packaged into container stacks.
- Face detection → Docker stack: deploys on · Worker-based detection service is containerized.
- Embedding stack → Build tools: packages with · Embedding subsystem has dedicated build and packaging scripts.
- Camera app → Build tools: packages with · Legacy camera services are also shipped through build scripts/specs.
- Benchmarks → Skill contract: validates · Benchmark tools exercise the same event/detection contract.