How artlensteam/backendhackudc25 works
Project Overview
• ArtLens is an interactive museum guide that enriches the visitor experience by recognizing artworks (using a CLIP model), generating rich, contextualized descriptions (using a modified LLama 3.2 model), searching similar artworks via embedding retrieval with Milvus, and converting text descriptions into audio using gTTS.
• The project was developed as part of HackUDC 2025 and is structured into separate repositories for the frontend and backend, with the current repository containing the backend service and dataset.
Key Components
a. Frontend (Mobile Application):
• Although hosted in a separate repository, represent it as the user’s mobile app interface.
• It handles camera capture for artwork recognition and displays personalized guides/audio to the user.
b. Backend API:
• Represented by the “app” directory. It includes:
• Main application entry point (app/main.py)
• REST routers (app/routers/audio.py and app/routers/search.py) for handling audio generation and search requests.
• Business logic and service layers (app/services) that encapsulate configuration for Milvus, search services, and style configuration.
• Logger and general app configuration (app/config.py, app/core/logger.py).
• Models representing data (app/models/models.py).
• Unit tests (app/tests/) to show there are testing measures in place.
c. AI and Processing Services:
• Artwork Recognition: Using the CLIP model (and possibly other AI components) to analyze imagery.
• Description Generation: Powered by the LLama 3.2 model with modifications.
• Text-to-Speech (TTS): Using gTTS to produce audio guides.
d. External Dependencies & Infrastructure:
• Milvus: A vector database for embedding retrieval; this is run as an independent service (started with docker-compose).
• Ollama: Mentioned in the Docker Compose setup for services.
• Dataset: The “dataset” folder holds art images along with JSON metadata, which is used by the system to provide context and reference data for artworks.
• Docker Services: The docker-compose.yaml file indicates external containerized services are used to support Milvus and potentially Ollama.
Relationships and Data Flow
• User Interaction: The mobile frontend sends requests (e.g., for artwork recognition or to retrieve an audio guide) to the backend API.
• Routing and Service Invocation:
• The REST API endpoints (audio, search routers) in the backend receive and process user requests.
• For artwork recognition, the backend integrates with the CLIP model (this component is implicit but central to the artwork identification workflow).
• Once recognized, the backend communicates with the LLama-based description generator to produce enriched text.
• The generated text is then passed to the TTS service (gTTS) for audio conversion.
• Embedding Retrieval:
• When personalized recommendations or similar artwork search is needed, the backend uses Milvus (running in its container via Docker Compose) for fast embedding lookups.
• Data Store:
• The dataset directory serves as the source of artwork images and their metadata, which might be ingested or referenced by the backend service.
Architectural Patterns and Design Principles
• Separation of Concerns: Clear division between front-end (presentation layer) and backend business logic/services.
• Modular Structure: The “app” directory is organized into core, models, routers, and services, which supports a microservices-inspired modular design (even if running within a single backend project).
• Containerization: External services like Milvus and Ollama are containerized, highlighting the emphasis on isolation and scalability.
• Externalization of AI/ML Components: The use of specialized AI models (CLIP, LLama, and gTTS) indicates a design focused on leveraging state-of-the-art models as separate components or services.
Final Recommendations
• That every component is drawn with clarity – avoid overlaps and ambiguities.
• External interactions with Docker Compose-managed services.
• Keep in mind that while the backend repository is the focus here, mention that the frontend is an integral component (with reference to its separate repository) to capture the full stack picture.