How packtpublishing/llm-engineers-handbook works
Project & Its Purpose
• This repository is the official code for the LLM Engineer’s Handbook. It is an end-to-end system for building LLM-based pipelines—with data collection, feature engineering, training, evaluation, and inference components.
• The project emphasizes best practices in productionization including integration with AWS (SageMaker), databases (MongoDB, Qdrant), model registries (Hugging Face), experiment tracking (Comet ML/Opik), and pipeline orchestration (ZenML).
File Structure and Infer Architectural Layers
• The top-level directories reveal several logical groupings. For instance:
• llm_engineering/: This is the heart of the project. It is split into subpackages following Domain-Driven Design principles:
▸ application/ – business logic including crawlers, dataset generation, preprocessing, RAG (Retrieval Augmented Generation), and networking routines.
▸ domain/ – core business entities, types, and exceptions.
▸ infrastructure/ – integrations with external services (AWS, MongoDB, Qdrant) and utility modules.
▸ model/ – contains the LLM training, fine-tuning, evaluation, and inference routines.
• pipelines/ and steps/ – These contain ZenML pipeline definitions and reusable components (steps) that orchestrate the complete ML lifecycle.
• tools/ – Contains utility scripts to run pipelines or inference services.
• configs/ – YAML configuration files that drive the behavior of various pipelines.
• tests/ – Contains unit and integration tests.
• Dockerfile, docker-compose.yml, and CI/CD workflows give insight into containerized development and continuous integration/deployment strategies.
Main Components
• Data Ingestion / ETL Layer:
• “Crawlers” (under llm_engineering/application/crawlers) that collect data from different sources (GitHub, LinkedIn, Medium, custom articles).
• Configuration files in configs/digital_data_etl_*.
• Tools and scripts in “steps/etl/”.
• Data Processing & Feature Engineering:
• Preprocessing and operations in llm_engineering/application/preprocessing.
• Pipeline Orchestration:
• ZenML-managed pipelines (in pipelines/) that coordinate data collection, processing, training, and evaluation.
• Each pipeline driven by separate YAML configs and executed via “poetry poe” commands.
• Model Management:
• Training and Fine-Tuning (llm_engineering/model/finetuning and training pipelines in steps/training).
• Evaluation (llm_engineering/model/evaluation and steps/evaluating).
• Inference operations (llm_engineering/model/inference) and supplementary scripts (tools/run.py, tools/ml_service.py).
• External Service Integrations and Infrastructure:
• AWS integration visible under llm_engineering/infrastructure/aws for deploying SageMaker endpoints, scaling, and roles.
• Database access components for MongoDB (in llm_engineering/infrastructure/db/mongo.py) and vector databases using Qdrant (llm_engineering/infrastructure/db/qdrant.py).
• Third-party integrations: Hugging Face registry, Comet ML/Opik.
• Developer & CI/CD Support:
• CI/CD workflows defined in the .github folder.
• Docker and docker-compose configurations for local development.
• Testing folders under tests/ for quality assurance.
Interactions and Dependencies
• Data flows from crawlers → preprocessing → feature engineering → dataset generation → training.
• During training and evaluation, the pipelines call components in the “model” package for LLM fine-tuning.
• After training, the inference pipeline (via RESTful API and AWS SageMaker endpoints) integrates with the “inference” module.
• Infrastructure modules (external integrations and databases) support both the model and application layers.
• Domain layer underpins the business logic across the system.
Incorporate Architectural Patterns and Technologies
• Usage of Domain-Driven Design (divide llm_engineering into domain, application, infrastructure, and model).
• Orchestration pattern using ZenML pipelines and steps as modular components.
• External services as separate entities:
• AWS (SageMaker – for training and inference, IAM roles)
• MongoDB and Qdrant (for storage and retrieval)
• Hugging Face (for model registry)
• Comet ML/Opik (for experiment tracking and prompt monitoring)
• CI/CD integration via GitHub Actions.