How pesc101/dbqr works
Project type and purpose
• This is an NLP/ML pipeline for question answering over databases, orchestrated via Python scripts and configurable via YAML.
• It includes: data ingestion, prompt generation, LLM inference, code execution for reasoning, result tidying, evaluation, experiment logging, and an optional Streamlit UI.
High-level component breakdown
A. Configuration Manager
– Files: conf/*.yaml, config.py
– Purpose: Reads model parameters, tuning settings, WandB defaults.
B. Data Layer
– Files: src/data/overview.py, src/data/read_data.py
– Purpose: Loads question-answer pairs or schemas from disk.
C. Prompt Engine
– Files: prompts/**/*.j2, prompts/sys_prompts/*.txt
– Purpose: Uses Jinja2 templates to generate tailored prompts for LLMs.
D. Inference Module
– Files: src/code/code_executor.py, src/code/tools*.txt, src/code/tools.py
– Purpose: Submits prompts to the LLM API (OpenAI/HuggingFace), optionally runs generated Python for reasoning.
E. Post-Processing
– Files: src/tidy.py, src/code/execute_tidy.py
– Purpose: Cleans up and normalizes LLM outputs.
F. Evaluation
– File: src/evaluate.py
– Purpose: Computes accuracy, metrics for QA.
G. Orchestration Entry Point
– File: main.py
– Purpose: Pulls together all stages: load config, read data, generate prompts, call inference, tidy, evaluate, log.
H. Experiment Logging
– Files: conf/wandb/defaults.yaml, implicit WandB client in code
– Purpose: Tracks runs, metrics, parameters.
I. Interactive UI (Optional)
– File: streamlit/explore.py
– Purpose: A Streamlit-based dashboard to browse examples and results.
Relationships & Data Flows
• main.py → Configuration Manager: fetches model and logging settings.
• main.py → Data Layer: loads dataset.
• main.py → Prompt Engine: generates instructions for each example.
• main.py → Inference Module: sends prompt to external LLM service and/or executes generated code.
• Inference Module ↔ External LLM API: network call to OpenAI/HuggingFace.
• Inference Module → Post-Processing: normalizes raw responses.
• Post-Processing → Evaluation: computes QA metrics.
• Every stage → Experiment Logging (WandB): logs inputs, outputs, metrics.
• Streamlit UI reads from outputs of Post-Processing/Evaluation for interactive exploration.
Architectural patterns & design principles
• Pipeline pattern: linear flow through distinct stages (ingest → infer → tidy → evaluate).
• Separation of concerns: config, data I/O, prompt templating, inference, and evaluation are modular.
• Template-driven prompt engineering: Jinja2 for reusability.
• Pluggable model backends via YAML (GPT, LLaMA).
• Optional UI layer disconnected from batch pipeline.
Final checklist
– A note or sidebar listing major file locations under each component.