How nhanth301/smart-public-library-system works
Project Type and Purpose
• This is a full-stack, microservices-based system for a smart public library. Its goal is to allow users to interact with a book repository, receive chatbot-based book recommendations using a Retrieval-Augmented Generation (RAG) approach, and process real-time user interaction data for dynamic recommendation updates with reinforcement learning.
• The README highlights multiple pipeline components (data ingestion from Calibre, recommendation via FastAPI and Django, real-time processing with Kafka and Quixstream, and orchestration with Airflow), which are integrated via Docker and Docker Compose.
File Structure
• Top-Level Components:
• “airflow” directory: Contains Docker resources and entrypoint script; used to orchestrate ETL tasks and periodic model retraining.
• “calibre-config” directory: Manages book data and local databases (SQLite files and configuration secrets) for Calibre.
• “dags” directory: Hosts various scripts (ETL, model training, reinforcement learning routines) and source files that define workflow steps for Airflow.
• “django-library”: Contains the Django web application (backend and frontend) for the smart library; includes static files, templates, and app modules.
• “fastapi”: Contains a FastAPI service providing APIs that likely support the recommendation model and integrate the reinforcement learning algorithm.
• “linear_adapter”: Stores pre-trained model weights, dataset generation scripts, and Jupyter notebooks for ML experimentation and model evaluation. This indicates the use of a deep learning component (e.g., for embedding or policy networks).
• “quixstream”: Contains a containerized microservice for processing streaming data (a consumer likely subscribing to Apache Kafka topics).
• “server”: Contains a server configuration file (server.json) which might act as the central configuration for deployment.
• Docker-compose.yaml: Orchestrates deployment of multiple containers (Airflow, Django, FastAPI, Quixstream, etc.) ensuring that services communicate reliably.
README for Additional Insights
• The README details the system components – highlighting databases (SQLite for Calibre, PostgreSQL for long-term storage, and Redis for caching), API layers (FastAPI), data streaming (Kafka and Quixstream), the web interface (Django), and workflow orchestration (Airflow).
• It also describes the training routine and data handling aspects of the recommendation system (using REINFORCE reinforcement learning and contrastive learning via Triplet Loss in the "linear_adapter" module).
a. Main Components:
• Data Sources and Storage:
• Calibre Component: Book data management (with SQLite).
• PostgreSQL: Primary storage for long-term library data.
• Redis: Used as a temporary cache storage layer.
• ETL and Workflow Orchestration:
• Apache Airflow: Manages ETL tasks, model training, and periodic data updates.
• Dags: Scripts defining ETL and model training workflows.
• Backend & Frontend:
• Django Web Application: Handles user interface and library management.
• FastAPI: Serves RESTful APIs for machine learning and recommendation services.
• Real-Time Data Processing:
• Apache Kafka: Streams user interaction events.
• Quixstream: Processes streaming data from Kafka in a containerized microservice.
• Machine Learning and Recommendation:
• Linear Adapter & Reinforcement Learning Modules: Provide the trained models (policy networks and embedding components) and model retraining routines.
• Containerization & Deployment:
• Docker & Docker Compose: Containerizes all services and manages their intercommunication.
• Auxiliary files such as .env and server.json assist in configuration.
b. Relationships and Interactions:
• How Calibre’s book data moves from a local configuration (calibre-config) through ETL pipelines (dags via Airflow) to update central databases (PostgreSQL).
• Django web application’s interaction with PostgreSQL for serving the frontend, and with FastAPI’s recommendation endpoints.
• FastAPI as an API layer that integrates outputs from the ML models maintained by the linear_adapter components.
c. Architectural Patterns and Design Principles:
• Microservices Architecture: Each major component (Django, FastAPI, Airflow, Quixstream) runs in its own container.
• Containerization: All key services are containerized and deployed using Docker and Docker Compose.
• Layered Architecture: Separation of concerns between data ingestion (ETL), presentation (Django), business logic (FastAPI and ML modules), and data processing/streaming (Kafka and Quixstream).
• Integration of ML Model Training Loops: Demonstrates model retraining via reinforcement learning and contrastive pretraining.
d. Technologies, Frameworks, and Libraries:
• Python-based frameworks: Django, FastAPI.
• Docker and Docker Compose for container orchestration.
• Apache Airflow for ETL and workflow orchestration.
• Apache Kafka (and Quixstream for consuming its streams) for real-time data processing.
• Redis for caching.
• PostgreSQL and SQLite as database engines.
• Custom ML components (using PyTorch or similar frameworks) as seen in the linear_adapter’s .pth model files and reinforcement learning scripts.
Instructing the Principal Software Engineer
• Data Sources/Storage (Calibre/SQLite, PostgreSQL, Redis)
• ETL and Workflow Orchestration (Airflow, DAGs)
• Web Application (Django with front-end templates)
• API Layer and Recommendation Service (FastAPI)
• Real-Time Data Processing (Kafka, Quixstream)
• Machine Learning Engine (Linear Adapter with training and reinforcement learning modules)
• From raw book data to the ETL process,
• From user interactions (streaming data through Kafka) to the recommendation pipeline,
• And between the UI and the backend services (Django communicating with FastAPI/Databases)
• Make a note to include containerization details (Docker containers) and orchestration (Docker Compose) as an essential overlay.
• To include any inter-service dependencies or configuration files (e.g., .env, server.json) that influence service interaction.