How datadreamer-dev/datadreamer works
Project type and purpose
• DataDreamer is an open-source Python library and CLI tool for building multi-step prompting workflows, generating synthetic data, and training/aligning models using LLMs. It provides:
• A command-line interface and Python API
• A workflow engine based on “Steps” and “Tasks”
• Abstractions for LLM backends, embedders, retrievers
• Caching, parallel execution, and resumability
• Trainers for fine-tuning and alignment
High-level architecture and main components
A. User Interface Layer
• CLI (src/__cli__.py, __main__.py)
• Python API (src/datadreamer.py)
B. Workflow Orchestrator Layer
• Core workflow engine and steps (src/steps/, src/_cachable/, src/_parallel_cachable.py)
• Step types: prompt steps, data-source steps, background steps, export steps
• Task models (src/task_models/, src/trainers/)
C. Core Services Layer
• LLM abstraction (src/llms/): adapters for OpenAI, Anthropic, HF, vLLM, etc.
• Embedder abstraction (src/embedders/): various embedding backends
• Retriever abstraction (src/retrievers/): retrieval from synthetic or real data
• Caching & pickling (src/pickling/, src/_cachable/)
• Logging service (src/logging/logger.py)
• Persistence (src/project/persistent_storage.py)
• Environment & configuration (src/project/environment.py, .env)
D. External Systems Layer
• Cloud and API services: OpenAI, Anthropic, Cohere, Vertex AI, Hugging Face API, Bedrock
• Data sources: local files, HF datasets, CSV, JSON
• Storage backends: local disk, AWS S3 (via persistent_storage), cluster file systems
E. Orchestration & Infrastructure Layer
• Cluster scripts (scripts/.cluster/) for direct, SGE, Slurm runs
• Build, format, lint, CI/CD (.github/workflows/, scripts/package.sh)
• Documentation build (docs/)
Relationships and data flow
• The CLI/SDK invokes the Workflow Orchestrator
• The orchestrator sequences Steps:
1. DataSource step loads data
2. Prompt/Task step calls LLM or embedder via core services
3. Retriever steps fetch additional context
4. Outputs get cached/pickled if caching is enabled
5. Trainer steps use synthetic or real data for fine-tuning
• Logging events flow from all layers into the logging service
Architectural patterns and design principles
• Layered architecture separates user interface, orchestration, core services, and external systems
• Strategy/adapter pattern for LLMs, embedders, retrievers—easy to plug in new backends
• Caching decorator pattern (@_cachable) allows resuming workflows
• Parallel execution abstraction in _parallel_cachable for concurrent step runs
• Single responsibility: each module has a clear focus (steps vs. data vs. llm vs. training)