# microsoft/lost_in_conversation architecture

> Architecture diagram of the GitHub repository microsoft/lost_in_conversation, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/lost_in_conversation
- Repository: https://github.com/microsoft/lost_in_conversation

## Overview

##### Project type and purpose
- This repository (“Lost in Conversation”) is a research‐oriented benchmarking tool that simulates and evaluates multi‐turn conversations with large language models (LLMs).
- It provides scripts to run experiments (run_experiments.py, simulator_*.py), per-task logic under tasks/, a prompt library under prompts/, input data under data/, wrappers for LLM APIs (model_openai.py, system_agent.py, user_agent.py), and a Streamlit viewer (app_conv_viewer.py).

##### High-Level Architectural Components
a. Runner / Orchestrator
- Entry points: run_experiments.py (CLI for batch simulations), simulator_full.py / simulator_sharded.py / simulator_snowball.py / simulator_recap.py (variants).
- Responsibilities: load configuration, dispatch simulation jobs, collect results, write logs/output files.
b. Simulation Core
- Simulation Manager (within simulator_*.py) coordinating multi-turn or sharded‐turn dialogue.
- Agents:
- system_agent.py – abstracts the LLM “system” role, holds system prompts, calls LLM API.
- user_agent.py – simulates the user, holds user messages and sharded instructions.
- Prompt repository: prompts/ hierarchy storing system‐ and user‐side text templates per task.
- Task modules: tasks/ subfolders for each domain; each implements:
- Task class with methods to generate initial prompts, interpret responses, evaluate correctness.
- eval_* and task_*.py files for evaluation logic.
c. LLM API Layer
- model_openai.py – wrapper for OpenAI/Azure OpenAI calls (handles rate limits, retries, streaming).
- Configured via environment variables: OPENAI_API_KEY, AZURE_OPENAI_KEY/ENDPOINT.
d. Data Storage & Processing
- data/sharded_instructions_600.json – input instructions to shard into multi-turn format.
- data/spider/ – SQL databases for the “database” task.
- Results and logs persisted by utils_log.py into timestamped directories or CSV/JSON.
e. Visualization Layer
- app_conv_viewer.py – Streamlit application to load simulation outputs, display turn‐by‐turn conversation along with model outputs and evaluations.
f. Utilities
- utils.py – common helper functions (file I/O, concurrency, sharding).
- utils_log.py – logging, metrics aggregation.
- requirements.txt – Python dependencies.

##### Relationships and Data Flow
1. Developer invokes run_experiments.py (or one of simulator_*.py)
2. Runner reads config/CLI args → loads data/sharded_instructions_600.json → splits into tasks
3. For each shard/task:
a. Task module builds the first user turn (using prompts/)
b. Simulation Manager alternates:
– user_agent sends “user” message → system_agent calls model_openai API → returns LLM “assistant” reply
– Task module evaluates or transforms reply, determines next “user” shard or termination
c. utils_log records turn-level inputs, outputs, evaluation metrics
4. Upon completion, Runner aggregates results, writes report files (JSON/CSV) in a designated output folder
5. Developer launches app_conv_viewer.py → loads simulation output folder → renders interactive conversation viewer

##### Architectural Patterns / Design Principles
- Modular “plugin” structure for tasks: adding tasks requires only a new tasks/<new_task> folder and register in tasks.py.
- Agent abstraction separates message orchestration (Simulation Manager) from LLM invocation (system_agent, user_agent, model_openai).
- Clear separation of concerns:
- Orchestration (runner, simulators)
- Task logic (tasks/)
- Prompt management (prompts/)
- External communication (model_openai.py)
- Visualization (app_conv_viewer.py)

##### Summary
- Decompose the project into: Orchestrator, Simulation Core, Task Plugins, Prompt Store, Agent Layer, External LLM API, Data Store, Logging/Results, Viewer.
- Plugin architecture and API interactions.

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% Orchestrator Layer
    subgraph "Orchestrator Layer"
        direction TB
        Runner["run_simulations.py"]:::orchestrator
        SimulatorFull["simulator_full.py"]:::orchestrator
        SimulatorSharded["simulator_sharded.py"]:::orchestrator
        SimulatorSnowball["simulator_snowball.py"]:::orchestrator
        SimulatorRecap["simulator_recap.py"]:::orchestrator
    end

    %% Simulation Core
    subgraph "Simulation Core"
        direction TB
        subgraph "Agents"
            direction TB
            SysAgent["system_agent.py"]:::agent
            UserAgent["user_agent.py"]:::agent
        end
        subgraph "Task Modules"
            direction TB
            TaskBase["tasks/task_base.py"]:::tasks
            TasksIndex["tasks/tasks.py"]:::tasks
            CodeTask["tasks/code/task_code.py"]:::tasks
            CodeEval["tasks/code/eval_code.py"]:::tasks
            ActionsTask["tasks/actions/task_actions.py"]:::tasks
            ActionsEval["tasks/actions/eval_bfcl.py"]:::tasks
            Data2TextTask["tasks/data2text/task_data2text.py"]:::tasks
            Data2TextUtil["tasks/data2text/utils_totto.py"]:::tasks
            DatabaseTask["tasks/database/task_database.py"]:::tasks
            DBExecEval["tasks/database/eval_spider_exec.py"]:::tasks
            DBParseEval["tasks/database/eval_spider_parse.py"]:::tasks
            MathTask["tasks/math/task_math.py"]:::tasks
            SummaryTask["tasks/summary/task_summary.py"]:::tasks
            SummaryEval["tasks/summary/eval_summhay.py"]:::tasks
            TranslationTask["tasks/translation/task_translation.py"]:::tasks
        end
        PromptRepo["prompts/"]:::prompt
    end

    %% LLM API Layer
    subgraph "LLM API Layer"
        direction TB
        ModelWrapper["model_openai.py"]:::api
        ExternalLLM["OpenAI/Azure API"]:::external
    end

    %% Data Storage & Processing
    DataStoreInstructions["sharded_instructions_600.json"]:::datastore
    DataStoreTranslation["sharded_translation.json"]:::datastore
    DataStoreSpider["data/spider/"]:::datastore

    %% Logging & Results
    UtilsLog["utils_log.py"]:::util

    %% General Utilities
    Utils["utils.py"]:::util

    %% Visualization
    Viewer["app_conv_viewer.py"]:::viewer

    %% Dependencies
    Req["requirements.txt"]:::util

    %% Connections
    Runner -->|"start simulation"| SimulatorFull
    Runner --> SimulatorSharded
    Runner --> SimulatorSnowball
    Runner --> SimulatorRecap

    SimulatorFull -->|"orchestrates"| SysAgent
    SimulatorFull --> UserAgent
    SimulatorFull --> PromptRepo
    SimulatorFull --> CodeTask
    SimulatorFull --> UtilsLog

    SimulatorSharded --> SysAgent
    SimulatorSnowball --> SysAgent
    SimulatorRecap --> SysAgent

    SysAgent --> ModelWrapper
    UserAgent --> PromptRepo
    ModelWrapper --> ExternalLLM

    CodeTask --> PromptRepo
    ActionsTask --> PromptRepo
    Data2TextTask --> PromptRepo
    DatabaseTask --> PromptRepo
    MathTask --> PromptRepo
    SummaryTask --> PromptRepo
    TranslationTask --> PromptRepo

    SimulatorFull --> DataStoreInstructions
    SimulatorFull --> DataStoreTranslation
    SimulatorFull --> DataStoreSpider

    SimulatorFull --> Utils
    SimulatorSharded --> Utils
    SimulatorSnowball --> Utils
    SimulatorRecap --> Utils

    UtilsLog -->|"writes output"| DataStoreInstructions
    Viewer -->|"reads output"| DataStoreInstructions

    %% Click Events
    click Runner "https://github.com/microsoft/lost_in_conversation/blob/main/run_simulations.py"
    click SimulatorFull "https://github.com/microsoft/lost_in_conversation/blob/main/simulator_full.py"
    click SimulatorSharded "https://github.com/microsoft/lost_in_conversation/blob/main/simulator_sharded.py"
    click SimulatorSnowball "https://github.com/microsoft/lost_in_conversation/blob/main/simulator_snowball.py"
    click SimulatorRecap "https://github.com/microsoft/lost_in_conversation/blob/main/simulator_recap.py"
    click SysAgent "https://github.com/microsoft/lost_in_conversation/blob/main/system_agent.py"
    click UserAgent "https://github.com/microsoft/lost_in_conversation/blob/main/user_agent.py"
    click PromptRepo "https://github.com/microsoft/lost_in_conversation/tree/main/prompts/"
    click TaskBase "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/task_base.py"
    click TasksIndex "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/tasks.py"
    click CodeTask "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/code/task_code.py"
    click CodeEval "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/code/eval_code.py"
    click ActionsTask "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/actions/task_actions.py"
    click ActionsEval "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/actions/eval_bfcl.py"
    click Data2TextTask "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/data2text/task_data2text.py"
    click Data2TextUtil "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/data2text/utils_totto.py"
    click DatabaseTask "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/database/task_database.py"
    click DBExecEval "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/database/eval_spider_exec.py"
    click DBParseEval "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/database/eval_spider_parse.py"
    click MathTask "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/math/task_math.py"
    click SummaryTask "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/summary/task_summary.py"
    click SummaryEval "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/summary/eval_summhay.py"
    click TranslationTask "https://github.com/microsoft/lost_in_conversation/blob/main/tasks/translation/task_translation.py"
    click ModelWrapper "https://github.com/microsoft/lost_in_conversation/blob/main/model_openai.py"
    click DataStoreInstructions "https://github.com/microsoft/lost_in_conversation/blob/main/data/sharded_instructions_600.json"
    click DataStoreTranslation "https://github.com/microsoft/lost_in_conversation/blob/main/data/sharded_translation.json"
    click DataStoreSpider "https://github.com/microsoft/lost_in_conversation/tree/main/data/spider/"
    click UtilsLog "https://github.com/microsoft/lost_in_conversation/blob/main/utils_log.py"
    click Utils "https://github.com/microsoft/lost_in_conversation/blob/main/utils.py"
    click Viewer "https://github.com/microsoft/lost_in_conversation/blob/main/app_conv_viewer.py"
    click Req "https://github.com/microsoft/lost_in_conversation/blob/main/requirements.txt"

    %% Styles
    classDef orchestrator fill:#d0e1f9,stroke:#2a6f97,color:#000
    classDef agent fill:#ffddb3,stroke:#cc8400,color:#000
    classDef prompt fill:#d0f0c0,stroke:#3b7d00,color:#000
    classDef tasks fill:#ccffcc,stroke:#008000,color:#000
    classDef api fill:#ffddb3,stroke:#cc8400,color:#000
    classDef datastore fill:#fff5ba,stroke:#cbc100,color:#000
    classDef util fill:#e0e0e0,stroke:#888888,color:#000
    classDef viewer fill:#e2d1f9,stroke:#663399,color:#000
    classDef external fill:#f0f0f0,stroke:#666666,color:#000
```
