# nhanth301/smart-public-library-system architecture

> Architecture diagram of the GitHub repository nhanth301/smart-public-library-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nhanth301/smart-public-library-system
- Repository: https://github.com/nhanth301/smart-public-library-system

## Overview

##### Project Type and Purpose
- This is a full-stack, microservices-based system for a smart public library. Its goal is to allow users to interact with a book repository, receive chatbot-based book recommendations using a Retrieval-Augmented Generation (RAG) approach, and process real-time user interaction data for dynamic recommendation updates with reinforcement learning.
- The README highlights multiple pipeline components (data ingestion from Calibre, recommendation via FastAPI and Django, real-time processing with Kafka and Quixstream, and orchestration with Airflow), which are integrated via Docker and Docker Compose.

##### File Structure
- Top-Level Components:
- “airflow” directory: Contains Docker resources and entrypoint script; used to orchestrate ETL tasks and periodic model retraining.
- “calibre-config” directory: Manages book data and local databases (SQLite files and configuration secrets) for Calibre.
- “dags” directory: Hosts various scripts (ETL, model training, reinforcement learning routines) and source files that define workflow steps for Airflow.
- “django-library”: Contains the Django web application (backend and frontend) for the smart library; includes static files, templates, and app modules.
- “fastapi”: Contains a FastAPI service providing APIs that likely support the recommendation model and integrate the reinforcement learning algorithm.
- “linear_adapter”: Stores pre-trained model weights, dataset generation scripts, and Jupyter notebooks for ML experimentation and model evaluation. This indicates the use of a deep learning component (e.g., for embedding or policy networks).
- “quixstream”: Contains a containerized microservice for processing streaming data (a consumer likely subscribing to Apache Kafka topics).
- “server”: Contains a server configuration file (server.json) which might act as the central configuration for deployment.
- Docker-compose.yaml: Orchestrates deployment of multiple containers (Airflow, Django, FastAPI, Quixstream, etc.) ensuring that services communicate reliably.

##### README for Additional Insights
- The README details the system components – highlighting databases (SQLite for Calibre, PostgreSQL for long-term storage, and Redis for caching), API layers (FastAPI), data streaming (Kafka and Quixstream), the web interface (Django), and workflow orchestration (Airflow).
- It also describes the training routine and data handling aspects of the recommendation system (using REINFORCE reinforcement learning and contrastive learning via Triplet Loss in the "linear_adapter" module).

a. Main Components:
- Data Sources and Storage:
- Calibre Component: Book data management (with SQLite).
- PostgreSQL: Primary storage for long-term library data.
- Redis: Used as a temporary cache storage layer.
- ETL and Workflow Orchestration:
- Apache Airflow: Manages ETL tasks, model training, and periodic data updates.
- Dags: Scripts defining ETL and model training workflows.
- Backend & Frontend:
- Django Web Application: Handles user interface and library management.
- FastAPI: Serves RESTful APIs for machine learning and recommendation services.
- Real-Time Data Processing:
- Apache Kafka: Streams user interaction events.
- Quixstream: Processes streaming data from Kafka in a containerized microservice.
- Machine Learning and Recommendation:
- Linear Adapter & Reinforcement Learning Modules: Provide the trained models (policy networks and embedding components) and model retraining routines.
- Containerization & Deployment:
- Docker & Docker Compose: Containerizes all services and manages their intercommunication.
- Auxiliary files such as .env and server.json assist in configuration.

b. Relationships and Interactions:
- How Calibre’s book data moves from a local configuration (calibre-config) through ETL pipelines (dags via Airflow) to update central databases (PostgreSQL).
- Django web application’s interaction with PostgreSQL for serving the frontend, and with FastAPI’s recommendation endpoints.
- FastAPI as an API layer that integrates outputs from the ML models maintained by the linear_adapter components.

c. Architectural Patterns and Design Principles:
- Microservices Architecture: Each major component (Django, FastAPI, Airflow, Quixstream) runs in its own container.
- Containerization: All key services are containerized and deployed using Docker and Docker Compose.
- Layered Architecture: Separation of concerns between data ingestion (ETL), presentation (Django), business logic (FastAPI and ML modules), and data processing/streaming (Kafka and Quixstream).
- Integration of ML Model Training Loops: Demonstrates model retraining via reinforcement learning and contrastive pretraining.

d. Technologies, Frameworks, and Libraries:
- Python-based frameworks: Django, FastAPI.
- Docker and Docker Compose for container orchestration.
- Apache Airflow for ETL and workflow orchestration.
- Apache Kafka (and Quixstream for consuming its streams) for real-time data processing.
- Redis for caching.
- PostgreSQL and SQLite as database engines.
- Custom ML components (using PyTorch or similar frameworks) as seen in the linear_adapter’s .pth model files and reinforcement learning scripts.

##### Instructing the Principal Software Engineer
- Data Sources/Storage (Calibre/SQLite, PostgreSQL, Redis)
- ETL and Workflow Orchestration (Airflow, DAGs)
- Web Application (Django with front-end templates)
- API Layer and Recommendation Service (FastAPI)
- Real-Time Data Processing (Kafka, Quixstream)
- Machine Learning Engine (Linear Adapter with training and reinforcement learning modules)
- From raw book data to the ETL process,
- From user interactions (streaming data through Kafka) to the recommendation pipeline,
- And between the UI and the backend services (Django communicating with FastAPI/Databases)
- Make a note to include containerization details (Docker containers) and orchestration (Docker Compose) as an essential overlay.
- To include any inter-service dependencies or configuration files (e.g., .env, server.json) that influence service interaction.

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% Data Sources and Storage
    A["Calibre Book Data (SQLite)"]:::data_storage
    B["PostgreSQL Database\n(Long-term Library)"]:::data_storage
    C["Redis Cache"]:::data_storage

    %% ETL and Workflow Orchestration
    subgraph "ETL and Workflow Orchestration"
        D["Apache Airflow (ETL & Workflow)"]:::etl
        E["DAGs (ETL/Training Scripts)"]:::etl
    end

    %% Web Application
    F["Django Web App (Backend & Frontend)"]:::backend

    %% API Layer and Recommendation Service
    G["FastAPI Service (Recommendation API)"]:::api

    %% Machine Learning and Recommendation
    H["Linear Adapter (ML & Recommendation)"]:::ml

    %% Real-Time Data Processing
    subgraph "Real-Time Data Processing"
        I["Apache Kafka (Data Streaming)"]:::realtime
        J["Quixstream (Real-time Processor)"]:::realtime
    end

    %% Containerization and Configuration
    K["Server Configuration"]:::container
    L["Docker Compose Orchestration"]:::container

    %% Data Flow Connections
    A -->|"book_data"| D
    A -->|"book_data"| E
    D -->|"update"| B
    E -->|"update"| B
    F -->|"reads/writes"| B
    F -->|"reads/writes"| C
    F -->|"calls_API"| G
    G -->|"fetches_model"| H
    H -->|"model_response"| G
    I -->|"user_events"| J
    J -->|"trigger_ETL"| D

    %% Container orchestration and config flows
    K -->|"configures"| L
    L -->|"orchestrates"| D
    L -->|"orchestrates"| F
    L -->|"orchestrates"| G
    L -->|"orchestrates"| J

    %% Click Events
    click A "https://github.com/nhanth301/smart-public-library-system/tree/main/calibre-config/"
    click D "https://github.com/nhanth301/smart-public-library-system/tree/main/airflow/"
    click E "https://github.com/nhanth301/smart-public-library-system/tree/main/dags/"
    click F "https://github.com/nhanth301/smart-public-library-system/tree/main/django-library/"
    click G "https://github.com/nhanth301/smart-public-library-system/tree/main/fastapi/"
    click H "https://github.com/nhanth301/smart-public-library-system/tree/main/linear_adapter/"
    click J "https://github.com/nhanth301/smart-public-library-system/tree/main/quixstream/"
    click K "https://github.com/nhanth301/smart-public-library-system/blob/main/server/server.json"
    click L "https://github.com/nhanth301/smart-public-library-system/blob/main/docker-compose.yaml"

    %% Styles
    classDef data_storage fill:#AED6F1,stroke:#0000FF,stroke-width:2px;
    classDef etl fill:#A9DFBF,stroke:#196F3D,stroke-width:2px;
    classDef backend fill:#F9E79F,stroke:#B9770E,stroke-width:2px;
    classDef api fill:#F5B7B1,stroke:#CB4335,stroke-width:2px;
    classDef ml fill:#D7BDE2,stroke:#6C3483,stroke-width:2px;
    classDef realtime fill:#FADBD8,stroke:#D35400,stroke-width:2px;
    classDef container fill:#D6EAF8,stroke:#2874A6,stroke-width:2px;
```
