How umass-ml4ed/dialogue-kt works
Project Type and Purpose
• This is a research-oriented full-stack Python toolkit for “dialogue knowledge tracing” (KT) in tutor-student dialogues.
• It automates annotation of raw dialogue data using the OpenAI API, implements a suite of KT models (LLM-based and classic sequence models), trains/evaluates them, and visualizes results.
Core Architectural Components
A. Data Layer
• Raw source data (data/src/…): unannotated CoMTA, MathDial, ATC standards.
• Annotated data (data/annotated/…): pre-tagged CSV/JSON files with knowledge components (KCs) and correctness labels.
• Artifacts directories: results/ (metrics, learning curves), saved_models/ (trained model checkpoints).
B. Annotation Pipeline
• Entry point: dialogue_kt/main.py CLI command annotate.
• Modules:
• dialogue_kt/annotate.py – high-level orchestration
• dialogue_kt/prompting.py – constructs LLM prompts for KC & correctness
• dialogue_kt/openai_api.py – wraps OpenAI API calls
• dialogue_kt/data_loading.py & dialogue_kt/kt_data_loading.py – load raw dialogues
• dialogue_kt/utils.py – helpers and I/O
C. Model Training & Evaluation
• Entry point: dialogue_kt/main.py CLI commands train, test.
• Model implementations (dialogue_kt/models/…):
• lm.py, dkt_multi_kc.py, dkt_sem.py, simplekt.py – various KT algorithms
• Training orchestration: dialogue_kt/training.py
• Evaluation & human-comparison: dialogue_kt/human_eval.py
• Results saved to results/ and saved_models/.
D. Visualization
External Dependencies & Environment
• Python 3.10, libraries in requirements.txt (likely PyTorch/TensorFlow, pandas, matplotlib).
• External service: OpenAI API (environment variable OPENAI_API_KEY).
• Determinism config (CUBLAS_WORKSPACE_CONFIG).
Relationships & Data Flow
1. Raw data → (data_loading) → Annotation pipeline → Annotated data
2. Annotated data → (training module) → Model checkpoints → Results (metrics)
3. Checkpoints + results → Visualization → Plots in results/
4. CLI (main.py) orchestrates each stage
Architectural Patterns & Principles
• Modular layering: Data → Annotation → Modeling → Evaluation → Visualization
• Clear CLI façade (main.py) separating concerns
• Plugin-style models directory for easy addition of KT algorithms
• Configurable hyperparameter sweep and cross-validation support