# datadreamer-dev/datadreamer architecture

> Architecture diagram of the GitHub repository datadreamer-dev/datadreamer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/datadreamer-dev/datadreamer
- Repository: https://github.com/datadreamer-dev/datadreamer

## Overview

##### Project type and purpose
- DataDreamer is an open-source Python library and CLI tool for building multi-step prompting workflows, generating synthetic data, and training/aligning models using LLMs. It provides:
- A command-line interface and Python API
- A workflow engine based on “Steps” and “Tasks”
- Abstractions for LLM backends, embedders, retrievers
- Caching, parallel execution, and resumability
- Trainers for fine-tuning and alignment

##### High-level architecture and main components
A. User Interface Layer
- CLI (`src/__cli__.py`, `__main__.py`)
- Python API (`src/datadreamer.py`)
B. Workflow Orchestrator Layer
- Core workflow engine and steps (`src/steps/`, `src/_cachable/`, `src/_parallel_cachable.py`)
- Step types: prompt steps, data-source steps, background steps, export steps
- Task models (`src/task_models/`, `src/trainers/`)
C. Core Services Layer
- LLM abstraction (`src/llms/`): adapters for OpenAI, Anthropic, HF, vLLM, etc.
- Embedder abstraction (`src/embedders/`): various embedding backends
- Retriever abstraction (`src/retrievers/`): retrieval from synthetic or real data
- Caching & pickling (`src/pickling/`, `src/_cachable/`)
- Logging service (`src/logging/logger.py`)
- Persistence (`src/project/persistent_storage.py`)
- Environment & configuration (`src/project/environment.py`, `.env`)
D. External Systems Layer
- Cloud and API services: OpenAI, Anthropic, Cohere, Vertex AI, Hugging Face API, Bedrock
- Data sources: local files, HF datasets, CSV, JSON
- Storage backends: local disk, AWS S3 (via persistent_storage), cluster file systems
E. Orchestration & Infrastructure Layer
- Cluster scripts (`scripts/.cluster/`) for direct, SGE, Slurm runs
- Build, format, lint, CI/CD (`.github/workflows/`, `scripts/package.sh`)
- Documentation build (`docs/`)

##### Relationships and data flow
- The CLI/SDK invokes the Workflow Orchestrator
- The orchestrator sequences Steps:
1. DataSource step loads data
2. Prompt/Task step calls LLM or embedder via core services
3. Retriever steps fetch additional context
4. Outputs get cached/pickled if caching is enabled
5. Trainer steps use synthetic or real data for fine-tuning
- Logging events flow from all layers into the logging service

##### Architectural patterns and design principles
- Layered architecture separates user interface, orchestration, core services, and external systems
- Strategy/adapter pattern for LLMs, embedders, retrievers—easy to plug in new backends
- Caching decorator pattern (`@_cachable`) allows resuming workflows
- Parallel execution abstraction in `_parallel_cachable` for concurrent step runs
- Single responsibility: each module has a clear focus (steps vs. data vs. llm vs. training)

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% User Interface Layer
    subgraph "User Interface Layer"
        direction TB
        CLI["CLI"]:::ui
        API["Python API"]:::ui
    end

    %% Workflow Orchestrator Layer
    subgraph "Workflow Orchestrator Layer"
        direction TB
        CoreEngine["Core Engine (Caching & Parallel)"]:::orchestrator
        subgraph "Steps"
            direction TB
            PromptSteps["Prompt Steps"]:::orchestrator
            DataSourceSteps["Data-Source Steps"]:::orchestrator
            BackgroundSteps["Background Steps"]:::orchestrator
            ExportSteps["Export Steps"]:::orchestrator
        end
        TaskModels["Task Models"]:::orchestrator
        Trainers["Trainer Entry Points"]:::orchestrator
    end

    %% Core Services Layer
    subgraph "Core Services Layer"
        direction TB
        subgraph "Abstractions"
            direction TB
            LLMs["LLM Adapters"]:::core
            Embedders["Embedder Adapters"]:::core
            Retrievers["Retriever Adapters"]:::core
        end
        Pickling["Pickling Service"]:::core
        CachingDeco["Caching Decorator"]:::core
        LoggingSvc["Logging Service"]:::core
        Persistence["Persistent Storage"]:::core
        EnvConfig["Environment & Configuration"]:::core
    end

    %% External Systems Layer
    subgraph "External Systems Layer"
        direction TB
        subgraph "Cloud APIs"
            direction TB
            OpenAI["OpenAI"]:::external
            Anthropic["Anthropic"]:::external
            Cohere["Cohere"]:::external
            VertexAI["Vertex AI"]:::external
            HFAPI["Hugging Face API"]:::external
            Bedrock["AWS Bedrock"]:::external
        end
        subgraph "Data Sources"
            direction TB
            LocalFiles["Local Files"]:::external
            HFDatasets["HF Datasets"]:::external
            CSVJSON["CSV/JSON"]:::external
        end
        subgraph "Storage Backends"
            direction TB
            LocalDisk["Local Disk"]:::external
            S3["AWS S3"]:::external
            ClusterFS["Cluster FS"]:::external
        end
    end

    %% Orchestration & Infrastructure Layer
    subgraph "Orchestration & Infrastructure Layer"
        direction TB
        ClusterScripts["Cluster Scripts"]:::infra
        CICD["CI/CD Workflows"]:::infra
        Package["Packaging Script"]:::infra
        Docs["Documentation Build"]:::infra
    end

    %% Connections
    CLI -->|"invokes"| CoreEngine
    API -->|"invokes"| CoreEngine

    CoreEngine -->|"executes"| DataSourceSteps
    CoreEngine -->|"executes"| PromptSteps
    CoreEngine -->|"executes"| BackgroundSteps
    CoreEngine -->|"executes"| ExportSteps
    CoreEngine -->|"manages"| TaskModels
    CoreEngine -->|"manages"| Trainers

    PromptSteps -->|"calls"| LLMs
    PromptSteps -->|"uses"| Embedders
    DataSourceSteps -->|"fetches"| Retrievers

    LLMs -->|"API call"| OpenAI
    LLMs -->|"API call"| Anthropic
    LLMs -->|"API call"| Cohere
    LLMs -->|"API call"| VertexAI
    LLMs -->|"API call"| HFAPI
    LLMs -->|"API call"| Bedrock

    Embedders -->|"API call"| HFAPI
    Retrievers -->|"reads from"| LocalFiles
    Retrievers -->|"reads from"| HFDatasets
    Retrievers -->|"reads from"| CSVJSON

    Persistence -->|"stores on"| LocalDisk
    Persistence -->|"stores on"| S3
    Persistence -->|"stores on"| ClusterFS

    CoreEngine -->|"caches via"| CachingDeco
    CoreEngine -->|"pickles via"| Pickling

    %% Logging from all layers
    CLI -->|logs| LoggingSvc
    API -->|logs| LoggingSvc
    CoreEngine -->|logs| LoggingSvc
    LLMs -->|logs| LoggingSvc
    Embedders -->|logs| LoggingSvc
    Retrievers -->|logs| LoggingSvc

    %% Infrastructure
    CoreEngine -->|"orchestrated by"| ClusterScripts
    ClusterScripts -->|"configured by"| CICD
    CICD -->|"triggers"| Package
    CICD -->|"triggers"| Docs

    %% Click Events
    click CLI "https://github.com/datadreamer-dev/datadreamer/blob/main/src/__cli__.py"
    click CLI "https://github.com/datadreamer-dev/datadreamer/blob/main/src/__main__.py"
    click API "https://github.com/datadreamer-dev/datadreamer/blob/main/src/datadreamer.py"
    click CoreEngine "https://github.com/datadreamer-dev/datadreamer/tree/main/src/_cachable/"
    click CoreEngine "https://github.com/datadreamer-dev/datadreamer/blob/main/src/_cachable/_cachable.py"
    click CoreEngine "https://github.com/datadreamer-dev/datadreamer/blob/main/src/_cachable/_parallel_cachable.py"
    click PromptSteps "https://github.com/datadreamer-dev/datadreamer/tree/main/src/steps/prompt/"
    click DataSourceSteps "https://github.com/datadreamer-dev/datadreamer/tree/main/src/steps/data_sources/"
    click BackgroundSteps "https://github.com/datadreamer-dev/datadreamer/blob/main/src/steps/step_background.py"
    click ExportSteps "https://github.com/datadreamer-dev/datadreamer/blob/main/src/steps/step_export.py"
    click TaskModels "https://github.com/datadreamer-dev/datadreamer/tree/main/src/task_models/"
    click Trainers "https://github.com/datadreamer-dev/datadreamer/tree/main/src/trainers/"
    click LLMs "https://github.com/datadreamer-dev/datadreamer/tree/main/src/llms/"
    click Embedders "https://github.com/datadreamer-dev/datadreamer/tree/main/src/embedders/"
    click Retrievers "https://github.com/datadreamer-dev/datadreamer/tree/main/src/retrievers/"
    click Pickling "https://github.com/datadreamer-dev/datadreamer/tree/main/src/pickling/"
    click CachingDeco "https://github.com/datadreamer-dev/datadreamer/tree/main/src/_cachable/"
    click LoggingSvc "https://github.com/datadreamer-dev/datadreamer/blob/main/src/logging/logger.py"
    click Persistence "https://github.com/datadreamer-dev/datadreamer/blob/main/src/project/persistent_storage.py"
    click EnvConfig "https://github.com/datadreamer-dev/datadreamer/blob/main/src/project/environment.py"
    click EnvConfig "https://github.com/datadreamer-dev/datadreamer/blob/main/src/.env"
    click ClusterScripts "https://github.com/datadreamer-dev/datadreamer/tree/main/scripts/.cluster/"
    click CICD "https://github.com/datadreamer-dev/datadreamer/tree/main/.github/workflows/"
    click Package "https://github.com/datadreamer-dev/datadreamer/blob/main/scripts/package.sh"
    click Docs "https://github.com/datadreamer-dev/datadreamer/tree/main/docs/"

    %% Styles
    classDef ui fill:#e0f7fa,stroke:#006064,color:#006064;
    classDef orchestrator fill:#bbdefb,stroke:#0d47a1,color:#0d47a1;
    classDef core fill:#fff9c4,stroke:#f57f17,color:#f57f17;
    classDef external fill:#c8e6c9,stroke:#1b5e20,color:#1b5e20;
    classDef infra fill:#ffe0b2,stroke:#e65100,color:#e65100;
```
