How vnktadithya/rca-lora-scaling-study works
This repository is a controlled Mistral-7B LoRA scaling study for structured root-cause analysis. Its main workflow collects and filters GitHub issue/PR examples, builds train/evaluation datasets, trains two adapters, generates predictions, and scores them across format, semantic-similarity, and LLM-judge tiers before comparing results. The README documents these capabilities; sampled source confirms key collection, training, serving, and judge calls. Evaluation scripts not sampled are represented by their exact tree paths, with only documented relationships shown.
Main entry points: generate_dataset.py Issue pipeline · train_ft20.py FT-20 trainer · train_ft500.py FT-500 trainer · logs Model predictions
Components 27
Data collection
- Issue pipeline
Data_Collection/generate_dataset.py - Traceback filter
Data_Collection/check_error_logs.py - Deduplication
Data_Collection/dedup.py - Processed ledger
Data_Collection/pipeline_state.py - Synthetic examples
Data_Collection/LLM_Generator.py - JSONL writer
Data_Collection/add_to_jsonl.py - Train/eval split
Data_Collection/train_test_split.py - Train/eval datasets
Data
Model training
- FT-20 trainer
Model_Training/train_ft20.py - FT-500 trainer
Model_Training/train_ft500.py - Prompt formatter
Model_Training/data_formatting.py - Input tokenizer
Model_Training/input_tokenizer.py - LoRA adapters
Checkpoints
Inference
- Batched generation
Model_Evaluation/output_generation.py - Model predictions
logs
Evaluation
- Tier 1 validity
Model_Evaluation/validity_evaluation.py - Tier 2 similarity
Model_Evaluation/similarity_evaluation.py - Tier 3 judge
Model_Evaluation/LLM_judge_evaluation.py - Judge API client
Model_Evaluation/LLM_judge.py - Metric comparison
Model_Evaluation/performance_comparision.py - Judge scores
logs - Benchmark plots
plots
Other components
- Study operator
- GitHub issues/PRs
- Gemini generator
- Mistral base model
- Gemini judge
Connections 36
- Study operator → Issue pipeline: starts collection
- Issue pipeline → GitHub issues/PRs: searches issues
- GitHub issues/PRs → Issue pipeline: returns issue data
- Issue pipeline → Traceback filter: checks tracebacks
- Issue pipeline → Deduplication: checks duplicates
- Issue pipeline → Processed ledger: reads/writes ledger
- Issue pipeline → Synthetic examples: requests examples
- Synthetic examples → Gemini generator: generates RCA
- Issue pipeline → Train/eval split: assigns split
- Issue pipeline → JSONL writer: appends accepted data
- JSONL writer → Train/eval datasets: writes JSONL
- Train/eval datasets → FT-20 trainer: supplies FT-20 data
- Train/eval datasets → FT-500 trainer: supplies full data
- FT-20 trainer → Prompt formatter: formats prompts
- FT-500 trainer → Prompt formatter: formats prompts
- FT-20 trainer → Input tokenizer: tokenizes examples
- FT-500 trainer → Input tokenizer: tokenizes examples
- FT-20 trainer → Mistral base model: fine-tunes
- FT-500 trainer → Mistral base model: fine-tunes
- FT-20 trainer → LoRA adapters: saves adapter
- FT-500 trainer → LoRA adapters: saves adapter
- Train/eval datasets → Batched generation: provides eval prompts
- Batched generation → Mistral base model: generates base output
- Batched generation → LoRA adapters: loads adapters
- Batched generation → Model predictions: writes predictions
- Model predictions → Tier 1 validity: scores format
- Model predictions → Tier 2 similarity: scores semantics
- Model predictions → Tier 3 judge: supplies predictions
- Tier 3 judge → Judge API client: evaluates examples
- Judge API client → Gemini judge: requests scoring
- Tier 3 judge → Judge scores: writes scores
- Model predictions → Metric comparison: provides metrics input
- Judge scores → Metric comparison: provides judge scores
- Metric comparison → Tier 1 validity: computes Tier 1
- Metric comparison → Tier 2 similarity: computes Tier 2
- Metric comparison → Benchmark plots: writes chart