How microsoft/lost_in_conversation works
Project type and purpose
• This repository (“Lost in Conversation”) is a research‐oriented benchmarking tool that simulates and evaluates multi‐turn conversations with large language models (LLMs).
• It provides scripts to run experiments (run_experiments.py, simulator_*.py), per-task logic under tasks/, a prompt library under prompts/, input data under data/, wrappers for LLM APIs (model_openai.py, system_agent.py, user_agent.py), and a Streamlit viewer (app_conv_viewer.py).
High-Level Architectural Components
a. Runner / Orchestrator
• Entry points: run_experiments.py (CLI for batch simulations), simulator_full.py / simulator_sharded.py / simulator_snowball.py / simulator_recap.py (variants).
• Responsibilities: load configuration, dispatch simulation jobs, collect results, write logs/output files.
b. Simulation Core
• Simulation Manager (within simulator_*.py) coordinating multi-turn or sharded‐turn dialogue.
• Agents:
• system_agent.py – abstracts the LLM “system” role, holds system prompts, calls LLM API.
• user_agent.py – simulates the user, holds user messages and sharded instructions.
• Prompt repository: prompts/ hierarchy storing system‐ and user‐side text templates per task.
• Task modules: tasks/ subfolders for each domain; each implements:
• Task class with methods to generate initial prompts, interpret responses, evaluate correctness.
• eval_* and task_*.py files for evaluation logic.
c. LLM API Layer
• model_openai.py – wrapper for OpenAI/Azure OpenAI calls (handles rate limits, retries, streaming).
• Configured via environment variables: OPENAI_API_KEY, AZURE_OPENAI_KEY/ENDPOINT.
d. Data Storage & Processing
• data/sharded_instructions_600.json – input instructions to shard into multi-turn format.
• data/spider/ – SQL databases for the “database” task.
• Results and logs persisted by utils_log.py into timestamped directories or CSV/JSON.
e. Visualization Layer
• app_conv_viewer.py – Streamlit application to load simulation outputs, display turn‐by‐turn conversation along with model outputs and evaluations.
f. Utilities
• utils.py – common helper functions (file I/O, concurrency, sharding).
• utils_log.py – logging, metrics aggregation.
• requirements.txt – Python dependencies.
Relationships and Data Flow
1. Developer invokes run_experiments.py (or one of simulator_*.py)
2. Runner reads config/CLI args → loads data/sharded_instructions_600.json → splits into tasks
3. For each shard/task:
a. Task module builds the first user turn (using prompts/)
b. Simulation Manager alternates:
– user_agent sends “user” message → system_agent calls model_openai API → returns LLM “assistant” reply
– Task module evaluates or transforms reply, determines next “user” shard or termination
c. utils_log records turn-level inputs, outputs, evaluation metrics
4. Upon completion, Runner aggregates results, writes report files (JSON/CSV) in a designated output folder
5. Developer launches app_conv_viewer.py → loads simulation output folder → renders interactive conversation viewer
Architectural Patterns / Design Principles
• Modular “plugin” structure for tasks: adding tasks requires only a new tasks/<new_task> folder and register in tasks.py.
• Agent abstraction separates message orchestration (Simulation Manager) from LLM invocation (system_agent, user_agent, model_openai).
• Clear separation of concerns:
• Orchestration (runner, simulators)
• Task logic (tasks/)
• Prompt management (prompts/)
• External communication (model_openai.py)
• Visualization (app_conv_viewer.py)
Summary
• Decompose the project into: Orchestrator, Simulation Core, Task Plugins, Prompt Store, Agent Layer, External LLM API, Data Store, Logging/Results, Viewer.
• Plugin architecture and API interactions.