How tarun1125/csaiml-capstone-project-20 works
This is a benchmark and demonstration project for natural-language-to-PyMongo query generation, comparing zero-shot, retrieval-augmented, and LoRA-adapted Qwen arms. Its main workflow takes verified question/query pairs, creates a held-out set, generates queries, and measures correctness by executing them against MongoDB Atlas and comparing results with executed gold answers. RAG retrieval and fine-tuning are distinct preparation paths. The README documents the overall workflow; sampled source confirms parts of generation, demo inference, and execution/scoring, so uncertain implementation links are omitted.
Main entry points: rag_test.json Held-out test cases · execute_queries.py Execute and compare · build_prompts.py Retrieved prompts · normalize.py Query normalization
Components 24
Dataset and gold
- Verified query pairs
data/reference_queries.json - Gold executions
data/gold_results.json - Few-shot and test split
rag/build_split.py - Few-shot pool
rag/data/rag_fewshot_pool.json - Held-out test cases
rag/data/rag_test.json
RAG arm
- FAISS retrieval index
rag/build_retrieval_index.py - Retrieval metadata
rag/data/fewshot_metadata.json - Retrieved prompts
rag/build_prompts.py - RAG prompt set
rag/data/rag_prompts.json
Generation arms
- Baseline generation
generate_baseline_mlx.py - RAG generation
rag/generate_rag_mlx.py - LoRA training data
fine_tuning/data_23db - LoRA training
fine_tuning - 23-database adapter
fine_tuning/adapters_23db_1000iter - Fine-tuned generation
fine_tuning/generate_predictions_23db.py
Execution evaluation
- Query normalization
normalize.py - Execute and compare
evaluation/execute_queries.py - RAG scoring
rag/score_rag.py - Execution results
data
Interactive demo
- Streamlit demo
demo_ui/app.py - Live inference wrapper
demo_ui/live_inference.py - Curated examples
demo_ui/curated_examples.py
Other components
- Engineer / evaluator
- MongoDB Atlas
Connections 30
- Verified query pairs → Gold executions: executes gold queries
- Verified query pairs → Few-shot and test split: splits cases
- Few-shot and test split → Few-shot pool: produces pool
- Few-shot and test split → Held-out test cases: produces holdout
- Few-shot pool → FAISS retrieval index: embeds questions
- FAISS retrieval index → Retrieved prompts: retrieves neighbors
- Retrieval metadata → Retrieved prompts: resolves neighbors
- Held-out test cases → Retrieved prompts: builds prompts
- Retrieved prompts → RAG prompt set: writes prompts
- Held-out test cases → Baseline generation: provides cases
- RAG prompt set → RAG generation: provides prompts
- Few-shot pool → LoRA training data: forms training data
- LoRA training data → LoRA training: trains on cases
- LoRA training → 23-database adapter: produces adapter
- Held-out test cases → Fine-tuned generation: provides cases
- 23-database adapter → Fine-tuned generation: loads adapter
- Baseline generation → Query normalization: normalizes generated queries
- RAG generation → Query normalization: normalizes generated queries
- Fine-tuned generation → Query normalization: normalizes generated queries
- Query normalization → Execute and compare: provides normalized queries
- Execute and compare → MongoDB Atlas: executes queries
- Execute and compare → Gold executions: compares results
- Execute and compare → Execution results: writes scores
- RAG scoring → MongoDB Atlas: executes RAG and baseline
- RAG scoring → Gold executions: compares outputs
- RAG scoring → Execution results: writes scores
- Engineer / evaluator → Streamlit demo: chooses arm
- Streamlit demo → Curated examples: renders examples
- Streamlit demo → Live inference wrapper: dispatches inference
- Live inference wrapper → MongoDB Atlas: optionally executes query