How timkjr/bookshelf-scanner-noncuda works
Project’s Main Components
• AI Service:
• Located in the "ai" directory. It contains image processing, segmentation, and LLM inference logic.
• Key files include src/bookscanner_ai/predict.py and utility modules (image_utils.py and path_utils.py). It also makes use of datasets and models.
• Architecturally, this service processes images by segmenting them using the YOLO model and then applying Moondream2 to extract text via LLM inference.
• Backend:
• Found in the "backend" directory, implemented with FastAPI.
• Defines the HTTP API (e.g., POST /api/predict) in its routers (predict_router.py), which receives the image, calls into the AI service, and streams results back to the client.
• Manages dependency injection, synchronous/asynchronous data streaming, and serves as the logical connector between the frontend and AI service.
• Frontend:
• Located in the "frontend" directory and built with Angular (with Bun as runtime/package manager).
• Provides user interface for image upload and displays the segmented image and recognized book details (titles, authors), updating dynamically as the backend streams the data.
Component Relationships and Data Flow
• The User uploads an image via the Frontend.
• The Frontend sends this image to the Backend API (using an endpoint like /api/predict).
• The Backend, upon receiving the image, invokes the AI component (bookscanner_ai) to start processing:
• First, image segmentation via YOLO detects each book.
• Then, individual book spines are cropped, possibly rotated, and forwarded to the Moondream2 model for title/author prediction.
• The Backend streams the results back to the Frontend:
• The first data chunk includes a segmented image (encoded in base64) for immediate display.
• Subsequent chunks stream recognized book details in an incremental fashion.
• Finally, the Frontend updates the UI in real-time as the data arrives.
Architectural Patterns and Design Principles
• Separation of Concerns:
• Keep the UI (Frontend) separate from business logic (Backend) and image processing/AI logic (AI Service).
• Modularity and Scalability:
• How each service can be deployed or scaled independently, especially given the resource-intensive tasks in AI processing.
• Data Streaming:
• Since the backend streams results from the AI service to the frontend, clearly mark this asynchronous streaming communication.