How mrnkim/surgical-tool-detection works
Surgical Tool Detection is a web platform for indexing surgical videos, detecting instruments, and exploring video intelligence results. The principal workflow is upload → video indexing → YOLO detection → persisted results → timeline/statistics and analysis. Search, SOAP notes, and clip creation are separate product capabilities. The frontend API and backend excerpts show key upload and detection handoffs; README statements support broader features. Some UI wiring and several API implementations were not sampled, so those relationships are omitted or cited as documented capabilities.
Main entry points: page.js Video workspace · upload/route.js Video upload API · [videoId]/route.js Detection result access · analysis/route.js Video chat API
Components 21
Web experience
- Video workspace
frontend/src/app/[factoryId]/page.js - Clip workspace
frontend/src/app/clips/[id]/page.js - Detection overlay
frontend/src/app/components/ToolDetectionOverlay.js - Tool timeline
frontend/src/app/components/ToolUsageTimeline.js - Tool statistics
frontend/src/app/components/ToolUsageStatistics.js - Search results
frontend/src/app/components/SearchResultModal.js - Clip creation
frontend/src/app/components/ClipBento.js
Video ingestion
- Video upload API
frontend/src/app/api/upload/route.js - Video indexing
Tool detection
- Detection gateway
frontend/src/app/api/detect-tools/[videoId]/route.js - YOLO detection API
backend/main.py - Detection result access
frontend/src/app/api/detect-tools/[videoId]/route.js - RTSP stream worker
rtsp-stream-worker/main.py
Video intelligence
- Chapter timeline API
frontend/src/app/api/timeline/route.js - Video chat API
frontend/src/app/api/analysis/route.js - Video search API
frontend/src/app/api/search/route.js
Analysis and results
- Video and result blobs
- SOAP notes API
frontend/src/app/api/analysis/[videoId]/soap/route.js - Analysis data API
frontend/src/app/api/analysis/[videoId]/route.js
Other components
- Video analyst
- TwelveLabs
Connections 24
- Video analyst → Video workspace: uses
- Video analyst → Video upload API: uploads video
frontend/src/app/api/upload/route.js - Video upload API → Video and result blobs: stores video
frontend/src/app/api/upload/route.js - Video upload API → TwelveLabs: indexes video
frontend/src/app/api/upload/route.js - Video upload API → Detection gateway: starts detection
frontend/src/app/api/upload/route.js - Detection gateway → YOLO detection API: requests detection
frontend/src/app/api/detect-tools/[videoId]/route.js - YOLO detection API → Detection gateway: saves results
backend/main.py - Detection gateway → Video and result blobs: persists results
frontend/src/app/api/detect-tools/[videoId]/route.js - Video workspace → Detection result access: views detections
- Detection result access → Detection overlay: supplies detections
- Detection result access → Tool timeline: feeds timeline
- Detection result access → Tool statistics: feeds statistics
- Video workspace → Chapter timeline API: requests chapters
- Chapter timeline API → TwelveLabs: generates chapters
frontend/src/app/api/timeline/route.js - Video workspace → Video chat API: asks video questions
- Video chat API → TwelveLabs: analyzes video
frontend/src/app/api/analysis/route.js - Video chat API → Video and result blobs: reads detections
frontend/src/app/api/analysis/route.js - Analysis data API → Video and result blobs: reads and writes
frontend/src/app/api/analysis/[videoId]/route.js - SOAP notes API → Analysis data API: edits analysis
- Video workspace → Video search API: searches videos
- Video search API → TwelveLabs: queries index
frontend/src/app/api/search/route.js - Search results → Clip creation: creates clips
- Video analyst → Clip workspace: reviews clips
- RTSP stream worker → YOLO detection API: runs CV analysis