How sherin-sef-ai/incident-lens-ai works
This repo is a frontend-only, client-side React 19 + TypeScript + Vite application for forensic video analysis / accident reconstruction. The core “engine” is not a local service; it’s Google Gemini (via @google/genai) accessed from the browser, with streaming responses + tool use (googleSearch) producing structured JSON that drives dashboards/charts and reporting.
• The browser app performs media preprocessing (frames/audio extraction), orchestration, visualization, and PDF export.
• The AI inference and search grounding occur via external Google services.
• There is no backend in this repo (but README explicitly recommends a backend proxy for production).
App shell / entry
• index.html (Vite HTML entry)
• index.tsx (React root mount)
• App.tsx (top-level composition, global state orchestration)
UI components
• components/VideoUploader.tsx
• evidence ingestion (drag/drop, file selection)
• likely emits “video selected + metadata” events into app state
• components/AnalysisDashboard.tsx
• main “case workspace”: timeline, stream view, knobs/sandbox, chat-like Q&A, etc.
• components/AnalysisCharts.tsx
• charts (Recharts radar/pie, etc.) and any computed views from structured JSON
Service layer (important boundary)
• services/geminiService.ts
• all calls to Gemini API (streaming generateContentStream)
• tool use / grounding via googleSearch
• prompt building + strict JSON response handling (typed structures)
Shared contract/config
• types.ts (typed JSON schema the model must output; drives UI)
• constants.ts (model name, prompt templates, chart constants, etc.)
• metadata.json (training scenarios or sample dataset metadata; represent as “local bundled scenarios/data”)
• package.json, vite.config.ts, tsconfig.json (build/runtime boundaries; show in deployment view)
Goal: who uses it, what external systems it depends on.
• User (Claims/Legal/Fleet)
• Incident Lens AI Web App (Browser SPA)
• Google Gemini API (Gemini 3 Pro)
• Google Search Tool / Grounding (as part of Gemini tool-use; still show explicitly)
• (Optional) Static Hosting/CDN (where the Vite build is deployed)
• User ↔ Web App: upload video, review dashboard, export PDF
• Web App → Gemini API: frames+audio+prompt (multimodal)
• Gemini API ↔ Google Search: tool calls for statutes/case law grounding
• Web App ← Gemini API: streaming tokens / structured JSON + citations
Goal: what runs where.
Containers (be explicit that this repo only contains one runtime container):
• Browser SPA (React/Vite)
Subsystems inside it:
• UI layer (components)
• Media preprocessing (Frame extraction, AudioContext waveform extraction)
• AI Orchestrator (Gemini service)
• Visualization (Recharts, Canvas annotations)
• Reporting (jsPDF client-side)
• External: Gemini Service
• External: Search Grounding
• “Client-side API key present (demo)” boundary
• “Production: backend proxy recommended” note
Goal: show internal separation inside the SPA.
• App.tsx (state + routing between uploader/dashboard)
• VideoUploader.tsx → emits EvidenceAsset / selected scenario
• “Media Processing Module” (even if not a separate file, represent as a logical component used by App/Dashboard)
• Frame sampler
• Audio extractor (AudioContext)
• geminiService.ts
• request builder (prompt + attachments)
• stream handler (incremental updates)
• JSON validator/parser (maps to types.ts)
• AnalysisDashboard.tsx
• live analysis stream viewer
• timeline controller
• variable sandbox inputs (friction, etc.)
• AnalysisCharts.tsx
• renders Recharts radar/pie etc. from typed JSON
• “PDF Exporter” (logical component; implemented somewhere in App/Dashboard)
• types.ts as “Shared Schema/Contracts”
• constants.ts as “Prompts/Model Config”
• metadata.json as “Bundled Scenario Metadata”
• Uploader → App state: videoFile, scenarioId, caseMetadata
• App/Dashboard → Media Processing: videoFile → {frames[], audioWaveform, timestamps}
• App/Dashboard → geminiService: {frames, audio, promptConfig} → streaming
• geminiService → App/Dashboard: {streamEvents, partialFindings, finalStructuredJSON}
• Dashboard → Charts: analysisResultJSON
• Dashboard → PDF Exporter: analysisResultJSON + citations + screenshots
Goal: capture the “reasoning stream” lifecycle precisely.
App triggers preprocessing
• extract frames (sampling rate)
• extract audio waveform (WAV/PCM representation)
Gemini streams back
• partial text/thinking trace (if shown)
• tool-use calls: googleSearch(query)
• returns grounded citations
• emits structured JSON blocks (final)
User exports PDF (jsPDF) containing
• findings
• charts
• citations / chain-of-custody notes (as supported)
Goal: show build artifacts and runtime environment.
Include:
• Developer machine: Vite dev server
• CI/build step: vite build → static assets
• Hosting: static site host/CDN
• Runtime: user browser
• External: Gemini endpoints
Call out env handling:
• .env API_KEY injected at build/runtime (client-side in this demo)
AI inference is remote (Gemini API); the app is primarily an orchestrator + viewer.
4) Tool grounding is remote (Gemini calling search) and should be drawn as a separate external dependency.
5) UI is driven by typed structured JSON (critical: types.ts is the contract between AI output and UI).
6) Streaming is not a simple request/response; it’s incremental updates that drive “Live Analysis Stream”.
Architectural patterns/design principles evident here
• SPA + Service Layer boundary: React components call a dedicated geminiService.ts (good separation).
• Schema-first UI: types.ts defines the output contract; UI components assume typed data.
• Streaming/event-driven UI updates: analysis is shown progressively; dashboard re-renders on stream events.
• Client-side pipeline orchestration: preprocessing → inference → render → export all in-browser.
“Don’t overthink it” — maximize component separation
• Evidence ingestion
• Media preprocessing
• AI orchestration/service
• Stream handling
• Schema validation
• Visualization (charts/timeline)
• Reporting/export
Optional: Production-target extension (annotate as “future architecture”)
• Backend API (proxy Gemini calls, protect keys)
• Object storage for video evidence (S3/GCS)
• Database for cases/metadata
• AuthN/AuthZ (enterprise users)
• Audit log / chain-of-custody immutability