How aniike-t/content-based-image-retrieval works
Project Type and Purpose
• This is a full-stack application that implements a CBIR system using a Vision Transformer (ViT) for image feature extraction along with NLP (utilizing spaCy, NLTK, and WordNet) to process natural language queries.
• The backend (Python/Flask) handles image processing (including quadtree segmentation and vector space modeling) and interacts with an SQLite database, while the frontend (React.js) provides an interactive interface for image uploads, searches, and user feedback.
Main Components
• Frontend:
– React-based UI (see frontend/CBIR_Frontend) with components for image upload (UploadArea), search functionality (SearchBar), results display (ImageGrid), and feedback (binary and textual).
– The frontend communicates (via HTTP calls) with the backend server.
• Backend:
– HTTP API Server (main.py in backend/system) that orchestrates the retrieval and processing tasks.
– Retrieval Module: Includes algorithms for retrieving and ranking images (backend/system/Retrieval/RetrieveAlgo.py).
– NLP / Tokenisation Module: Contains scripts for generating SQL queries and processing sentences and word extraction (backend/system/tokenisation).
– Feature Extraction and Image Processing Modules: Handle tasks like downloading models (downloadModel), getting features using CNN (getFeaturesCNN), and image verification (checkIfImage).
– ThreadPool Module: Manages multi-threaded image processing tasks (backend/system/ThreadPool).
– SQL Methods: Database connection and common SQL operations (backend/system/SQLmethods) for interacting with SQLite (backend/database.db).
– Utility Modules: Helper scripts (randomStringGenerator, math.py, etc.) that assist other operations.
• Database:
– SQLite database (backend/database.db) stores image features, metadata, and user feedback.
• External Libraries & Frameworks:
– Vision Transformer (ViT) via Hugging Face Transformers.
– NLP libraries: spaCy, NLTK, and WordNet.
– Other dependencies as declared in requirements.txt and package.json.
Relationships and Data Flow
• Frontend-to-Backend:
– The React app sends HTTP requests (e.g., image upload, search query submission, or user feedback) to the backend API.
• Within the Backend:
– Upon receiving a request (for search or image upload), the backend delegates work to specialized modules:
▪ The Retrieval module processes image data.
▪ The Tokenisation/NLP module refines user queries.
▪ The ThreadPool manages parallel image processing.
– These components interact with the SQLite database via SQL methods to store or retrieve image features and metadata.
• Feedback Cycle:
– User interactions (likes, dislikes, textual feedback) result in updates to the database which help refine future retrieval results.
Important Architectural Patterns
• Full-stack modular separation: Clear boundaries between the frontend UI and backend processing layers.
• Micro-modular design in the backend: The separation into modules (retrieval, tokenisation, thread pool, etc.) promotes maintainability.
• Use of a thread pool pattern for concurrency in image processing.
• Integration of external services and libraries for feature extraction and NLP.