How decodingml/llm-twin-course works
The project is a production-ready system for building an “LLM Twin” that powers an AI replica using LLMs and advanced RAG (Retrieval-Augmented Generation) techniques. It follows a microservices architecture with multiple Python pipelines focused on data collection, feature engineering, training, and inference. The README clearly outlines that the system is split into four primary Python microservices (data collection, feature pipeline, training pipeline, and inference pipeline) with an extra bonus module for Superlinked RAG optimization. External dependencies and integrations (such as MongoDB, RabbitMQ, Bytewax, Qdrant, Redis, AWS SageMaker, Hugging Face model registry, and others) are also a core part of the design.
Main Components
• Data Collection Pipeline:
• Components from the “src/data_crawling” directory for crawling data (social media content, GitHub, Medium, etc.).
• “src/data_cdc” implements Change Data Capture (CDC) to push changes from a Mongo NoSQL DB to a RabbitMQ queue.
• Feature Engineering Pipeline:
• Code under “src/feature_pipeline” cleans, chunks, embeds data, and loads it into vector databases (Qdrant, and in the bonus, Redis via Superlinked).
• Bytewax streaming code is used to dynamically consume queued data.
• Training Pipeline:
• Found under “src/training_pipeline”, this pipeline takes processed feature data to create custom fine-tuning datasets.
• It fine-tunes the LLM (using LoRA or QLoRA) with integration to experiment trackers (e.g., Comet ML) and model registries (Hugging Face).
• Inference Pipeline:
• Under “src/inference_pipeline” the fine-tuned model is deployed (e.g., via AWS SageMaker endpoints) and enhanced using advanced RAG techniques.
• Interfaces for prompt monitoring and feedback (with tools such as Opik) are part of the inference loop.
• Bonus RAG Module:
• “src/bonus_superlinked_rag” provides optimized RAG implementations using the Superlinked vector compute engine for even more efficient retrieval and query expansion.
Component Relationships and Interactions
• Data flows sequentially: Data is first gathered and cleaned, then forwarded into the feature pipeline where it is transformed into embeddings.
• The CDC process monitors a live database (MongoDB) and sends updates via a message queue system (RabbitMQ) to downstream pipelines.
• The Feature Pipeline consumes streaming data (using Bytewax), processes it, and indexes vectors into vector databases (Qdrant and/or Redis).
• The Training Pipeline leverages the processed data to fine-tune the LLM, logging experiments and registering models.
• The Inference Pipeline loads the fine-tuned model and exposes a REST API (via AWS SageMaker endpoints) for production queries with support for prompt evaluation and monitoring.
• Cross-component communication may occur between microservices (for example, sharing configuration, logging, or even resource handling handled via internal libraries in “src/core”).
Important Architectural Patterns and Design Principles
• Microservices Architecture: The system is split into clearly defined services (one per pipeline) that can be independently developed, deployed, and scaled.
• Data Streaming and Event-Driven Processing: Real-time message streaming via Bytewax with a CDC pattern ensures timely data propagation.
• Containerization & Deployment: With multiple Dockerfiles and docker-compose configurations (.docker, docker-compose.yml), the project is designed for both local development and cloud deployment.
• Modularity and Reusability: Common utilities, configuration files, and core libraries (e.g., core/rag, core/db) support cross-microservice functionality.