How peremartra/wizardsdata works
Overall Project Type and Purpose
• This project is a Python library (SDK) that automates the generation of conversation datasets by simulating dialogues between two roles (a client and an advisor) using templates.
• Its primary function is to dynamically build prompts using JSON profile data and Jinja2 templates and then call language model APIs to generate responses. The use cases include generating dialogues in financial, healthcare, or other regulated sectors.
Break Down the Main Components
a. Core Library Modules:
• CLI Module (wizardsdata/cli.py): Acts as the command-line interface and primary entry point for triggering conversation generation.
• Configuration Module (wizardsdata/config.py): Handles reading, setting, saving, and loading configuration parameters including API keys, file paths, and model settings.
• Generator Module (wizardsdata/generator.py): Contains the core logic to drive the conversation simulation process—including iterating over profiles, rendering templates, making API calls, and compiling results.
• Templates and Prompt Rendering (wizardsdata/templates.py): Manages loading Jinja2 templates and merging them with profile data to generate dynamic prompts.
• Utility Functions (wizardsdata/utils.py): Provides helper functionalities for file operations, logging, or any repeated tasks.
b. Supporting Directories:
• Usage Examples: Contains practical scripts (e.g., basic_usage.py and a dedicated financial dataset example) to show how to use the library.
• Built Distribution (build and dist folders): These show packaging and distribution details.
• Documentation (docs): Comprehensive documentation that covers API usage, examples, and advanced usage scenarios.
• Templates and Profiles: Real example templates and JSON profiles (under folders such as templates/financial01) show how to structure input data.
c. External Interactions:
• Language Model API: An external service (e.g., OpenAI) which is used during prompt generation to generate responses.
• User Data: Input profiles provided as JSON files and prompt templates that drive the conversation simulation.
Relationships and Data Flow
• The user configures the library via the CLI or programmatically (via a Python script) by setting up configuration parameters, including paths to profile JSON, prompt templates, and model selection.
• The configuration module reads and validates these parameters.
• The generator module then iterates through each profile from the JSON input.
• For every profile, the templates module uses Jinja2 to render client and advisor prompt templates by injecting profile data.
• The rendered prompts are sent to the external language model API. The responses are then collected by the generator and stored (e.g., in a JSON output file).
• Supporting utility functions facilitate proper logging, error handling, and file I/O throughout.
• The CLI ties everything together for execution in a command-line environment.
Architectural Patterns and Relevant Technologies
• Modular Design: Separation of concerns across CLI, configuration, prompt generation, and utility functionalities.
• Template Rendering: Use of Jinja2 for dynamic prompt creation.
• External API Integration: Clearly defined interaction with a language model API (configurable model settings).
• Extensibility: Profiles and prompt templates are designed to be flexibly defined, which allows the library to support various conversation simulation scenarios across industries.
• Packaging: The presence of build and dist directories, along with setup.py and pyproject.toml, indicates that the project is structured for simple distribution and installation.