How rasahq/tod-in-context-learning works
Project type and purpose
• This is a research codebase implementing and evaluating a task-oriented dialogue bot built on top of Rasa Pro. It provides three “versions” (v1, v2, v3) of the bot:
• version_1: a CALM (LLM-powered) implementation using in-context learning.
• version_2: an intent-based minimal baseline.
• version_3: an intent-based, larger-budget baseline.
• The project’s primary purpose is to train and end-to-end test these bots against a common suite of YAML-defined scenarios and compare performance for an academic paper.
File structure
• Top-level:
• e2e_tests/ : YAML conversations categorized by behavior (happy_path, corrections, cancellations, etc.).
• version_1/, version_2/, version_3/: three independent bot implementations, each with its own Rasa domain, data, actions, DB JSON files, and config.
• scripts/: result-processing scripts (combine_results.py, count_lines.py, describe_tests.py).
• test_results/: raw text outputs of each test suite run.
• Makefile, run_tests.sh: orchestration of install, train, and test commands.
• Each version directory:
• data/: NLU/training data, rules, stories (v2 & v3 use standard Rasa data layout; v1 uses custom .yml flows).
• actions/: custom action server modules that interact with DB JSON files, external services.
• domain/: slot/intents/entities definitions.
• endpoints.yml: Rasa endpoints (action server).
• External-service stubs/config:
• LLM API (OpenAI or alternative) in version_1/config.yml & action code.
• Duckling server URL in check_restaurant_availability.py.
Extract additional insights from the README
• A Rasa Pro license is required to run the code.
• Version_1 optionally calls out to OpenAI via env var OPENAI_API_KEY.
• Training and testing are invoked via make train-version version=<n> and make test-version version=<n>.
• Results are collected, then processed into line-counts.json and text files under test_results.
Final pointers
• Be explicit about version isolation: each version runs in its own container or process with its own config.yml, domain, data and endpoints.
• Flow of test data from YAML → bot → log → report.
• Don’t forget environment variables (RASA_PRO_LICENSE, OPENAI_API_KEY) as enablers for certain paths.
• Keep component labels high-level, but annotate subfolders (actions/, data/, domain/) where helpful.