How edge-video-services/ekya works
Project Type and Purpose
• Ekya is a research‐grade, full-pipeline continuous‐learning system for video analytics on edge servers. It ingests live video streams (or prerecorded MP4s), performs object detection/inference, periodically retrains models in the background, and intelligently allocates GPU resources among inference and retraining jobs via a scheduler (e.g., “Thief Scheduler”).
• It uses Ray for distributed execution (actors/tasks), NVIDIA MPS for fine‐grained GPU sharing, and supports multiple datasets (Cityscapes, Waymo, UrbanTraffic, UrbanBuilding).
Top-Level Module Breakdown
• ekya/datasets & deployment: video samplelist generation and dataset preparation scripts
• ekya/classes & ekya/models: abstractions for cameras, ML models (ResNet, iCaRL, object detectors), and inference managers
• ekya/microprofilers: short “probe” runs to estimate retraining job resource→accuracy curves (“promise”)
• ekya/schedulers: pluggable scheduling algorithms (thief, fair, no_retrain, utilitysim) implementing a common BaseScheduler interface
• ekya/drivers & ekya/experiment_drivers: orchestrate end‐to‐end runs for paper figures, hyperparam sweeps, and dataset‐specific “driver_*.sh” scripts
• ekya/simulation & simulator/: lightweight simulators for validation and utilitysim schedule generation
• ekya/model_cache: baseline continuous‐model‐selection pipeline for comparison
• ekya/benchmarking & ekya/analysis & ekya/plotting_scripts & ekya/viz: offline analysis, benchmarking, and figure‐generation notebooks/scripts
• utils, tests, setup.py, environment.yml: support code, CI, and packaging
Key Architectural Components
A. Data Ingestion & Preprocessing
• Video streams (live MP4s or dataset folders) → ekya/datasets scripts → “samplelists” CSVs
• Dataset loader feeds frames into Ray actors for inference or retraining
B. Microprofiling
• ekya/microprofilers runs short training jobs with varying hyperparams to generate performance profiles
• Profiles stored in memory or on disk (profile_generator → runtime_data)
C. Scheduler & Resource Manager
• ekya/schedulers/BaseScheduler defines reallocation_callback() and get_inference_schedule()
• Implements Thief Scheduler: steal GPU cycles from low‐promise jobs → reallocate to high‐promise ones
• Scheduler consults microprofile data, current camera/job status, GPU capacity
D. Execution Engines
• Inference Engine: Ray remote actors running small‐chunk inference tasks continuously
• Retraining Engine: Ray actors running fine‐tuning jobs on per-camera models (with history weighting, ICaRL support)
• GPU managed via NVIDIA MPS daemon (fine‐grain resource slicing)
E. Model Cache (Baseline)
• ekya/model_cache: trains a set of “scenario” models offline and selects per‐window model at runtime
F. Simulation & Benchmarking
• simulator/ and ekya/simulation: discrete‐event simulators for scheduler validation
• ekya/benchmarking/mps: end-to-end profiling of candidate models on CIFAR10, Cityscapes for MPS overhead analysis
G. Logging & Visualization
• logs and results written under timestamped directories (results/ekya_expts/…)
• ekya/viz and ekya/plotting_scripts: Jupyter notebooks and scripts for generating figures
Interactions & Data Flows
• Inference Engine ──► Outputs detections & accuracy metrics
• Retraining Engine ──► Updated model checkpoints
• Inference Engine & Retraining Engine receive updated resource weights
Architectural Patterns & Principles
• Modular, plug-in schedulers via common interface (Open/Closed Principle)
• Actor model concurrency using Ray remote functions/actors for inference and training jobs
• Resource slicing via NVIDIA MPS and scheduler callbacks
• Separation of concerns: data ingestion, profiling, scheduling, execution, analysis