How danielrapi/-mit_global_hackathon-_beaverfeaver works
Project Type and Purpose
• Type: Full-stack web application with embedded ML/AI pipeline.
File Structure
• Top-level directories:
• _frontend: Next.js/React TypeScript UI
• backend: Python FastAPI server + ML pipeline
• __testing__: unit/integration tests for CV and image utilities
• backend/patchcore, backend/utils, backend/interaction: ML model, helper utilities, and storage of inference outputs
• inference_outputs, uploads, results: on-disk storage of images, JSON summaries, pickles
• Key config files: Procfile (deployment), requirements.txt (Python deps), package.json/tsconfig.json/pnpm-lock (frontend deps).
A. User & Client
• Web browser (runs Next.js app)
B. Frontend Layer (_frontend)
• UI components: upload form, map (Leaflet), image modal, sliders
• API routes (/api/embeddings, any custom routes)
• Vector search & data utilities
C. API Gateway / Edge (Next.js serverless functions)
• /api/embeddings → forwards to backend
D. Backend API (FastAPI in backend/app.py)
• Endpoints: image upload, inference trigger, result retrieval
E. Processing Pipeline (backend/process_pipeline.py)
• Orchestrates steps: EXIF extraction, anomaly detection, LLM analysis, embedding generation
F. Anomaly Detection Engine (backend/patchcore)
• Patchcore model (Anomalib implementation)
• Weights stored under backend/patchcore/drone/v25/...
G. EXIF & Inference Utilities (backend/utils)
• exif_utils.py → GPS extraction or placeholder generation
• inference_utils.py → segmentation mask & heatmap creation
• llm_utils.py → calls to OpenAI GPT-4o-mini
H. Vector Embedding & Search
• Creates embeddings of LLM annotations
• Serves them via vector-search.ts (frontend) and corresponding backend functions (if any)
I. Storage
• File storage (local directories or S3): uploads/, inference_outputs/, results/
• Pickles, JSON summaries, CSV of predictions
J. External Service
• OpenAI API (for LLM analysis)
K. Database (optional/placeholder)
Relationships & Data Flow
1. User uploads image in browser → Next.js UI
2. Frontend calls Next.js API route → forwards POST to FastAPI upload endpoint
3. FastAPI writes raw image to uploads/
4. FastAPI invokes Processing Pipeline
a. EXIF extraction → get coordinates
b. Anomaly detection (Patchcore) → produce masks, heatmap
c. LLM analysis (via OpenAI) → textual interpretation
d. Embedding generation → vector representation
e. Write outputs to inference_outputs/ (images, JSON, pickles), results/
5. FastAPI returns summary JSON + URLs to frontend
6. Frontend fetches JSON, overlays heatmap and markers on Leaflet map
7. Frontend may query embeddings via vector-search for related anomalies
Architectural Patterns & Design Principles
• Layered architecture: UI → API → Orchestration → ML engines → Storage
• Separation of concerns:
• Frontend purely UI & API client
• Backend API only handles HTTP & orchestration
• Pipeline modules each focus on one domain (EXIF, ML, LLM, embedding)
• Modular ML: Patchcore and LLM as interchangeable services
• File-based storage for reproducibility; could evolve into object storage