# packtpublishing/llm-engineers-handbook architecture

> Architecture diagram of the GitHub repository packtpublishing/llm-engineers-handbook, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/packtpublishing/llm-engineers-handbook
- Repository: https://github.com/packtpublishing/llm-engineers-handbook

## Overview

##### Project & Its Purpose
- This repository is the official code for the LLM Engineer’s Handbook. It is an end-to-end system for building LLM-based pipelines—with data collection, feature engineering, training, evaluation, and inference components.
- The project emphasizes best practices in productionization including integration with AWS (SageMaker), databases (MongoDB, Qdrant), model registries (Hugging Face), experiment tracking (Comet ML/Opik), and pipeline orchestration (ZenML).

##### File Structure and Infer Architectural Layers
- The top-level directories reveal several logical groupings. For instance:
- llm_engineering/: This is the heart of the project. It is split into subpackages following Domain-Driven Design principles:
▸ application/ – business logic including crawlers, dataset generation, preprocessing, RAG (Retrieval Augmented Generation), and networking routines.
▸ domain/ – core business entities, types, and exceptions.
▸ infrastructure/ – integrations with external services (AWS, MongoDB, Qdrant) and utility modules.
▸ model/ – contains the LLM training, fine-tuning, evaluation, and inference routines.
- pipelines/ and steps/ – These contain ZenML pipeline definitions and reusable components (steps) that orchestrate the complete ML lifecycle.
- tools/ – Contains utility scripts to run pipelines or inference services.
- configs/ – YAML configuration files that drive the behavior of various pipelines.
- tests/ – Contains unit and integration tests.
- Dockerfile, docker-compose.yml, and CI/CD workflows give insight into containerized development and continuous integration/deployment strategies.

##### Main Components
- Data Ingestion / ETL Layer:
- “Crawlers” (under llm_engineering/application/crawlers) that collect data from different sources (GitHub, LinkedIn, Medium, custom articles).
- Configuration files in configs/digital_data_etl_*.
- Tools and scripts in “steps/etl/”.

- Data Processing & Feature Engineering:
- Preprocessing and operations in llm_engineering/application/preprocessing.

- Pipeline Orchestration:
- ZenML-managed pipelines (in pipelines/) that coordinate data collection, processing, training, and evaluation.
- Each pipeline driven by separate YAML configs and executed via “poetry poe” commands.

- Model Management:
- Training and Fine-Tuning (llm_engineering/model/finetuning and training pipelines in steps/training).
- Evaluation (llm_engineering/model/evaluation and steps/evaluating).
- Inference operations (llm_engineering/model/inference) and supplementary scripts (tools/run.py, tools/ml_service.py).

- External Service Integrations and Infrastructure:
- AWS integration visible under llm_engineering/infrastructure/aws for deploying SageMaker endpoints, scaling, and roles.
- Database access components for MongoDB (in llm_engineering/infrastructure/db/mongo.py) and vector databases using Qdrant (llm_engineering/infrastructure/db/qdrant.py).
- Third-party integrations: Hugging Face registry, Comet ML/Opik.

- Developer & CI/CD Support:
- CI/CD workflows defined in the .github folder.
- Docker and docker-compose configurations for local development.
- Testing folders under tests/ for quality assurance.

##### Interactions and Dependencies
- Data flows from crawlers → preprocessing → feature engineering → dataset generation → training.
- During training and evaluation, the pipelines call components in the “model” package for LLM fine-tuning.
- After training, the inference pipeline (via RESTful API and AWS SageMaker endpoints) integrates with the “inference” module.
- Infrastructure modules (external integrations and databases) support both the model and application layers.
- Domain layer underpins the business logic across the system.

##### Incorporate Architectural Patterns and Technologies
- Usage of Domain-Driven Design (divide llm_engineering into domain, application, infrastructure, and model).
- Orchestration pattern using ZenML pipelines and steps as modular components.
- External services as separate entities:
- AWS (SageMaker – for training and inference, IAM roles)
- MongoDB and Qdrant (for storage and retrieval)
- Hugging Face (for model registry)
- Comet ML/Opik (for experiment tracking and prompt monitoring)
- CI/CD integration via GitHub Actions.

## Diagram (Mermaid)

```mermaid
graph TD
    %% Data Ingestion / ETL Layer
    subgraph "Data Ingestion / ETL Layer"
        A1["Crawlers"] 
        A2["ETL Config Maxime Labonne"]
        A3["ETL Config Paul Iusztin"]
        A4["ETL Steps"]
    end

    %% Data Processing & Feature Engineering
    subgraph "Data Processing & Feature Engineering"
        B1["Preprocessing"]
        B2["Feature Engineering Steps"]
    end

    %% Pipeline Orchestration
    subgraph "Pipeline Orchestration"
        C1["ZenML Pipelines"]
        C2["Pipeline Steps"]
    end

    %% Model Management
    subgraph "Model Management"
        D1["Model Finetuning & Training"]
        D1b["Training Step"]
        D2["Model Evaluation"]
        D2b["Evaluation Step"]
        D3["Inference Module"]
        D3b["Inference Tool: run.py"]
        D3c["Inference Service: ml_service.py"]
    end

    %% External Service Integrations & Infrastructure
    subgraph "External Service Integrations & Infrastructure"
        E1["AWS Integration"]
        E2["Hugging Face Deployment"]
        E3["MongoDB"]
        E4["Qdrant"]
    end

    %% Developer & CI/CD Support
    subgraph "Developer & CI/CD Support"
        F1["CI/CD Workflows"]
        F2["Dockerfile"]
        F3["Docker Compose"]
        F4["Testing"]
    end

    %% Relationships for Data Ingestion / ETL
    A2 -->|"configures"| A4
    A3 -->|"configures"| A4
    A1 -->|"collects_data"| B1
    A4 -->|"executes_ETL"| B1

    %% Data Processing flows
    B1 -->|"prepares_data"| B2

    %% Pipeline Orchestration flows
    B2 -->|"triggers"| C1
    C1 -->|"uses_steps"| C2

    %% Pipeline triggering Model Management
    C1 -->|"orchestrates"| D1
    C1 -->|"orchestrates"| D2
    C1 -->|"orchestrates"| D3
    C2 -->|"executes_for"| D1
    C2 -->|"executes_for"| D2
    C2 -->|"executes_for"| D3

    %% Details within Model Management
    D1 -->|"training_detail"| D1b
    D2 -->|"evaluation_detail"| D2b
    D3 -->|"inference_detail"| D3b
    D3 -->|"service_interface"| D3c

    %% Interaction with External Integrations
    D1 -->|"deploys_to"| E1
    D2 -->|"deploys_to"| E1
    D3 -->|"deploys_to"| E1
    D1 -->|"updates_registry"| E2
    D2 -->|"logs_to"| E3
    D2 -->|"queries"| E4

    %% Developer & CI/CD triggering pipelines
    F1 -->|"triggers_pipelines"| C1
    F4 -->|"validates"| C1

    %% Class Definitions
    class A1,A2,A3,A4,B1,B2,C1,C2,D1,D1b,D2,D2b,D3,D3b,D3c internal
    class E1,E2,E3,E4 external
    class F1,F2,F3,F4 dev

    %% Click Events
    click A1 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/llm_engineering/application/crawlers"
    click A2 "https://github.com/packtpublishing/llm-engineers-handbook/blob/main/configs/digital_data_etl_maxime_labonne.yaml"
    click A3 "https://github.com/packtpublishing/llm-engineers-handbook/blob/main/configs/digital_data_etl_paul_iusztin.yaml"
    click A4 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/steps/etl"
    click B1 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/llm_engineering/application/preprocessing"
    click B2 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/steps/feature_engineering"
    click C1 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/pipelines"
    click C2 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/steps"
    click D1 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/llm_engineering/model/finetuning"
    click D1b "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/steps/training"
    click D2 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/llm_engineering/model/evaluation"
    click D2b "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/steps/evaluating"
    click D3 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/llm_engineering/model/inference"
    click D3b "https://github.com/packtpublishing/llm-engineers-handbook/blob/main/tools/run.py"
    click D3c "https://github.com/packtpublishing/llm-engineers-handbook/blob/main/tools/ml_service.py"
    click E1 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/llm_engineering/infrastructure/aws"
    click E2 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/llm_engineering/infrastructure/aws/deploy/huggingface"
    click E3 "https://github.com/packtpublishing/llm-engineers-handbook/blob/main/llm_engineering/infrastructure/db/mongo.py"
    click E4 "https://github.com/packtpublishing/llm-engineers-handbook/blob/main/llm_engineering/infrastructure/db/qdrant.py"
    click F1 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/.github/workflows"
    click F2 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/Dockerfile"
    click F3 "https://github.com/packtpublishing/llm-engineers-handbook/blob/main/docker-compose.yml"
    click F4 "https://github.com/packtpublishing/llm-engineers-handbook/tree/main/tests"

    %% Styles
    classDef internal fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef external fill:#FDB813,stroke:#000,stroke-width:2px;
    classDef dev fill:#90EE90,stroke:#000,stroke-width:2px;
```
