How arceusomkar7/clustr works
Project type and purpose
• This is a full-stack web application (CLUSTR MVP) that lets users upload photos, automatically captions them, detects and clusters faces, and organizes albums by scene and people.
• Frontend: React + TypeScript + Tailwind (Vite).
• Backend: Python (FastAPI), local file storage, SQLite/JSON metadata.
• ML services: BLIP captioning, MTCNN + FaceNet for face embeddings, DBSCAN for clustering.
File structure
• Top level:
• backend/ – Python API server, ML & DB logic.
• frontend/ – React UI.
• backend/app/:
• routers/ – upload and base endpoints.
• services/ – upload_service, mongodb_service.
• ml/ – caption_service.
• db/ – mongodb.py wrapper (or SQLite stub).
• models/ – Pydantic models.
• utils/ – image preprocessing.
• frontend/src/:
• pages/ – Home, Gallery, Profile.
• components/ – UploadComponent, GalleryItem, ImageDetailModal, theme.
• services/ – api.ts, uploadService, galleryService.
• hooks/ – useGallery, useImageUpload.
• assets & ui primitives.
• Config & deployment:
• Dockerfile, .env-example in backend.
• vite.config.ts, tsconfig, package.json in frontend.
Key README insights
• Four phases:
1. Upload pipeline
2. Captioning (BLIP)
3. Face detection & clustering (MTCNN, FaceNet, DBSCAN)
4. Album generation by scene & people
• Tech summary: FastAPI/Flask, local storage, SQLite/JSON, HuggingFace BLIP, MTCNN + FaceNet, DBSCAN.
a. Main components
• User’s browser (React UI)
• Frontend app (static assets served by Vite / CDN)
• Backend API (FastAPI)
• File storage (local disk or cloud bucket)
• Metadata database (SQLite or MongoDB)
• ML modules (Captioning, Face Detection & Embedding, Clustering)
• (Optional) Message broker / task queue if asynchronous
b. Relationships & interactions
1. User → UploadComponent → POST /upload → Backend API
2. Backend receives file, saves to storage, writes metadata record
3. Backend synchronously (or via task queue) calls:
– caption_service.generate_caption(image) → returns text → update DB
– mtcnn.detect_faces(image) & facenet.embed(crops) → embeddings → DB
– clustering_service.cluster_all_embeddings() → person IDs → DB
4. Frontend polls or GETs /gallery endpoint → Backend reads DB, returns list of images + captions + person clusters
5. UI renders grid, modals, filters by scene keywords or person ID
c. Architectural patterns
• MVC-style separation: routers (controllers) → services (business logic) → models (data)
• Pipeline pattern for ML processing
• Client-Server REST API
• (Future) microservices for each ML component or async task queue
d. Relevant technologies
• React + Vite + Tailwind
• FastAPI
• Python ML libs: transformers (BLIP), facenet-pytorch (MTCNN, FaceNet), scikit-learn (DBSCAN)
• Pydantic models
• SQLite / MongoDB
• Docker for containerization
Detail
• Data schemas or key fields (image_id, caption_text, person_ids) if needed.