# eladlev/autoprompt architecture

> Architecture diagram of the GitHub repository eladlev/autoprompt, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/eladlev/autoprompt
- Repository: https://github.com/eladlev/autoprompt

## Overview

##### Project Type and Purpose
- AutoPrompt is a prompt optimization framework that refines user-provided prompts for LLMs through an iterative calibration process. It caters to use cases like movie review classification, content generation, and moderation scenarios.
- The system is designed to work with various prompt types (classification, ranking, generation) and integrates with external services such as Argilla (for human-in-the-loop annotation), WandB (for monitoring), and LLM providers (e.g., OpenAI’s GPT-4).

##### File Structure to Identify Key Components
- Top-level configuration files: “config” folder (includes default YAML configurations, LLM environment settings, and task-specific configurations) indicates centralized configuration management.
- Pipeline Entry Points: Scripts like run_pipeline.py and run_generation_pipeline.py represent the main starting points for different workflows (classification versus generation pipelines).
- Core Modules:
- The “estimator” directory contains modules (estimator_llm.py, estimator_llm_batch.py) managing the estimation process and interfacing with LLM APIs.
- The “dataset” directory (e.g., base_dataset.py) is responsible for constructing or managing datasets used in prompt refinement.
- The “eval” directory with evaluator and evaluation utilities defines how prompt outputs are measured and improved.
- The “prompts” directory holds various prompt templates—separated by task types (classification, completion, generation, ranking) and modifiers—indicating a modular design for handling prompt variants.
- “utils” (e.g., llm_chain.py, config.py) provides helper functions, including chaining calls to LLMs and configuration parsing.

##### Components
a. Input & Configuration:
- User provides an initial prompt, task description, and optionally examples.
- Configuration Manager (from config files such as config/config_default.yml, config/llm_env.yml) to set parameters (e.g., API keys, annotation settings, budget constraints).
b. Pipeline Controller:
- Main pipeline runners (run_pipeline.py and run_generation_pipeline.py) act as orchestrators.
c. Data Handling:
- Dataset module (dataset/base_dataset.py) responsible for loading or synthesizing data samples.
d. Prompt Processing:
e. Estimation and Evaluation:
- Estimator modules (from the estimator directory) which include different strategies for prompt optimization (for example, estimator_llm.py for direct LLM inference).
- Evaluator module (from eval/evaluator.py and eval_utils.py) which is used to assess prompt quality and output performance.
f. LLM Interaction & External Services:
- LLM Chain (from utils/llm_chain.py) that handles iterative calls to the LLM (e.g., GPT-4, GPT-3.5).
- External integrations for monitoring (W&B), annotations (Argilla), and potentially other LLM providers.
g. Optimization & Feedback Loop:
- An iterative optimization loop where the system uses evaluation feedback to refine the prompt over multiple iterations (as detailed in the README’s “System Overview” and optimization tips).
h. Output:
- Final refined prompt and generated benchmark samples (saved to a “dump” directory by default) that can be used to re-run or analyze the workflow.

##### Relationships and Interactions
- The user/provided input flowing into the configuration manager.
- The configuration manager funneling information to the pipeline controller.
- The pipeline controller triggering the data module (dataset) and then passing inputs into the estimator chain.
- The estimator and prompt modules interacting with the LLM chain to generate candidate prompts.
- Outputs from the LLM chain are sent to the evaluator module, which calculates performance metrics.
- Evaluator feedback flows back into the estimator/pipeline controller forming an iterative optimization feedback loop.

##### Architectural Patterns and Relevant Technologies
- Modularity: The separation seen in the estimator, prompts, dataset, and evaluation modules indicates a modular and pluggable architecture.
- Iterative Feedback Loop: The prompt calibration process continuously refines the prompt based on evaluation metrics.
- Integration with External Services: Highlight integrations with Argilla, WandB, and LLM providers like OpenAI.
- Use of YAML configuration and scripting for environment management (Pipfile, environment_dev.yml) emphasizes adherence to modern DevOps practices.

##### Final Advice
- Incorporate components from both the file tree structure and insights from the README documentation.
- Revisit the existing “docs/architecture.md” for inspiration, but enrich it by highlighting modules that are specific to the AutoPrompt’s functioning—especially the iterative feedback loop that drives prompt optimization.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input & Configuration
    subgraph "Input & Configuration"
        UI["User Input"]:::input
        ConfigMgr["Configuration Manager"]:::config
    end

    %% Pipeline Controller
    subgraph "Pipeline Controller"
        PipeCtrl["Pipeline Controller (run_pipeline.py / run_generation_pipeline.py)"]:::controller
    end

    %% Data Handling
    subgraph "Data Handling"
        DataMod["Dataset Module"]:::data
    end

    %% Prompt Processing
    subgraph "Prompt Processing"
        PromptProc["Prompt Templates & Modifiers"]:::prompt
    end

    %% Estimation & Evaluation
    subgraph "Estimation & Evaluation"
        Estimator["Estimator Module"]:::estimation
        Evaluator["Evaluator Module"]:::evaluation
    end

    %% LLM Interaction
    subgraph "LLM Interaction"
        LLMChain["LLM Chain"]:::llm
    end

    %% External Services
    subgraph "External Services"
        WandB["WandB Monitoring"]:::external
        Argilla["Argilla Annotation"]:::external
        LLMProviders["LLM Providers"]:::external
    end

    %% Output
    Output["Final Output (Refined Prompt & Samples)"]:::output

    %% Connections
    UI -->|"input"| ConfigMgr
    ConfigMgr -->|"config_parameters"| PipeCtrl
    PipeCtrl -->|"start_pipeline"| DataMod
    DataMod -->|"load_data"| PromptProc
    PromptProc -->|"generate_candidates"| Estimator
    Estimator -->|"invoke_LLM"| LLMChain
    LLMChain -->|"return_candidates"| Evaluator
    Evaluator -->|"feedback"| PipeCtrl
    Evaluator -->|"refined_output"| Output
    PipeCtrl -->|"logs_tracking"| WandB
    Estimator ---|"argilla_integration"| Argilla
    LLMChain ---|"LLM_calls"| LLMProviders
    Evaluator -->|"iterative_feedback"| Estimator

    %% Click Events
    click ConfigMgr "https://github.com/eladlev/autoprompt/blob/main/config/config_default.yml"
    click PipeCtrl "https://github.com/eladlev/autoprompt/blob/main/run_pipeline.py"
    click DataMod "https://github.com/eladlev/autoprompt/blob/main/dataset/base_dataset.py"
    click PromptProc "https://github.com/eladlev/autoprompt/tree/main/prompts/"
    click Estimator "https://github.com/eladlev/autoprompt/blob/main/estimator/estimator_llm.py"
    click Evaluator "https://github.com/eladlev/autoprompt/blob/main/eval/evaluator.py"
    click LLMChain "https://github.com/eladlev/autoprompt/blob/main/utils/llm_chain.py"
    click WandB "https://github.com/eladlev/autoprompt/blob/main/docs/architecture.md"

    %% Styles
    classDef input fill:#c7f464,stroke:#333,stroke-width:2px;
    classDef config fill:#a2d5f2,stroke:#333,stroke-width:2px;
    classDef controller fill:#f7c59f,stroke:#333,stroke-width:2px;
    classDef data fill:#ffeb8a,stroke:#333,stroke-width:2px;
    classDef prompt fill:#dcd0ff,stroke:#333,stroke-width:2px;
    classDef estimation fill:#fcd1d1,stroke:#333,stroke-width:2px;
    classDef evaluation fill:#ffc9de,stroke:#333,stroke-width:2px;
    classDef llm fill:#d1fcd1,stroke:#333,stroke-width:2px;
    classDef external fill:#f9e79f,stroke:#333,stroke-width:2px;
    classDef output fill:#a9dfbf,stroke:#333,stroke-width:2px;
```
