# decodingml/llm-twin-course architecture

> Architecture diagram of the GitHub repository decodingml/llm-twin-course, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/decodingml/llm-twin-course
- Repository: https://github.com/decodingml/llm-twin-course

## Overview

The project is a production-ready system for building an “LLM Twin” that powers an AI replica using LLMs and advanced RAG (Retrieval-Augmented Generation) techniques. It follows a microservices architecture with multiple Python pipelines focused on data collection, feature engineering, training, and inference. The README clearly outlines that the system is split into four primary Python microservices (data collection, feature pipeline, training pipeline, and inference pipeline) with an extra bonus module for Superlinked RAG optimization. External dependencies and integrations (such as MongoDB, RabbitMQ, Bytewax, Qdrant, Redis, AWS SageMaker, Hugging Face model registry, and others) are also a core part of the design.

##### Main Components
- Data Collection Pipeline:
- Components from the “src/data_crawling” directory for crawling data (social media content, GitHub, Medium, etc.).
- “src/data_cdc” implements Change Data Capture (CDC) to push changes from a Mongo NoSQL DB to a RabbitMQ queue.
- Feature Engineering Pipeline:
- Code under “src/feature_pipeline” cleans, chunks, embeds data, and loads it into vector databases (Qdrant, and in the bonus, Redis via Superlinked).
- Bytewax streaming code is used to dynamically consume queued data.
- Training Pipeline:
- Found under “src/training_pipeline”, this pipeline takes processed feature data to create custom fine-tuning datasets.
- It fine-tunes the LLM (using LoRA or QLoRA) with integration to experiment trackers (e.g., Comet ML) and model registries (Hugging Face).
- Inference Pipeline:
- Under “src/inference_pipeline” the fine-tuned model is deployed (e.g., via AWS SageMaker endpoints) and enhanced using advanced RAG techniques.
- Interfaces for prompt monitoring and feedback (with tools such as Opik) are part of the inference loop.
- Bonus RAG Module:
- “src/bonus_superlinked_rag” provides optimized RAG implementations using the Superlinked vector compute engine for even more efficient retrieval and query expansion.

##### Component Relationships and Interactions
- Data flows sequentially: Data is first gathered and cleaned, then forwarded into the feature pipeline where it is transformed into embeddings.
- The CDC process monitors a live database (MongoDB) and sends updates via a message queue system (RabbitMQ) to downstream pipelines.
- The Feature Pipeline consumes streaming data (using Bytewax), processes it, and indexes vectors into vector databases (Qdrant and/or Redis).
- The Training Pipeline leverages the processed data to fine-tune the LLM, logging experiments and registering models.
- The Inference Pipeline loads the fine-tuned model and exposes a REST API (via AWS SageMaker endpoints) for production queries with support for prompt evaluation and monitoring.
- Cross-component communication may occur between microservices (for example, sharing configuration, logging, or even resource handling handled via internal libraries in “src/core”).

##### Important Architectural Patterns and Design Principles
- Microservices Architecture: The system is split into clearly defined services (one per pipeline) that can be independently developed, deployed, and scaled.
- Data Streaming and Event-Driven Processing: Real-time message streaming via Bytewax with a CDC pattern ensures timely data propagation.
- Containerization & Deployment: With multiple Dockerfiles and docker-compose configurations (.docker, docker-compose.yml), the project is designed for both local development and cloud deployment.
- Modularity and Reusability: Common utilities, configuration files, and core libraries (e.g., core/rag, core/db) support cross-microservice functionality.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Data Collection Layer
    subgraph "Data Collection Layer"
        DC["Data Crawling"]:::microservice
        CDC["Change Data Capture (CDC)"]:::microservice
    end

    MongoDB[("MongoDB")]:::external
    RabbitMQ[("RabbitMQ")]:::external

    %% Feature Engineering Layer
    subgraph "Feature Engineering Layer"
        FEATURE["Feature Pipeline"]:::microservice
        BYTEWAX["Bytewax Streaming"]:::microservice
    end

    %% Model Training & Inference Layer
    subgraph "Model Training & Inference Layer"
        TRAIN["Training Pipeline"]:::microservice
        INFER["Inference Pipeline (Gradio UI)"]:::microservice
    end

    BONUS["Bonus RAG Module"]:::microservice
    CORE["Core Libraries & Shared Utilities"]:::core

    QDRANT[("Qdrant")]:::external
    REDIS[("Redis")]:::external
    HUGGINGFACE["Hugging Face Model Registry"]:::external
    COMET["Comet ML"]:::external
    SAGEMAKER[("AWS SageMaker")]:::external
    OPIK["Opik"]:::external

    %% Data Collection Flow
    DC --> CDC
    MongoDB -.-> CDC
    CDC --> RabbitMQ

    %% Feature Engineering Flow
    RabbitMQ --> FEATURE
    FEATURE --> BYTEWAX
    FEATURE --> QDRANT
    FEATURE --> REDIS

    %% Training and Inference Flow
    FEATURE --> TRAIN
    FEATURE --> INFER
    TRAIN --> HUGGINGFACE
    TRAIN --> COMET
    INFER --> SAGEMAKER
    INFER --> OPIK

    %% Bonus RAG Integration
    FEATURE --> BONUS
    BONUS --> INFER

    %% Core Libraries Integration (shared utilities)
    CORE --- DC
    CORE --- CDC
    CORE --- FEATURE
    CORE --- TRAIN
    CORE --- INFER
    CORE --- BONUS

    %% Click Events for Mapped Components
    click DC "https://github.com/decodingml/llm-twin-course/tree/main/src/data_crawling"
    click CDC "https://github.com/decodingml/llm-twin-course/tree/main/src/data_cdc"
    click FEATURE "https://github.com/decodingml/llm-twin-course/tree/main/src/feature_pipeline"
    click TRAIN "https://github.com/decodingml/llm-twin-course/tree/main/src/training_pipeline"
    click INFER "https://github.com/decodingml/llm-twin-course/tree/main/src/inference_pipeline"
    click BONUS "https://github.com/decodingml/llm-twin-course/tree/main/src/bonus_superlinked_rag"
    click CORE "https://github.com/decodingml/llm-twin-course/tree/main/src/core"

    %% Style Definitions
    classDef microservice fill:#FFEEAD,stroke:#333,stroke-width:2px;
    classDef external fill:#AEDFF7,stroke:#333,stroke-width:2px,stroke-dasharray: 5,5;
    classDef core fill:#C1E1C1,stroke:#333,stroke-width:2px;
```
