How basava-code/agri-lm works
Agri-LM is a collection of agricultural data ingestion, knowledge-dataset preparation, monograph extraction, model fine-tuning, and evaluation tools. The user-facing workflow begins with KCC/API or policy scraping and institutional book inputs; ETL and extraction prepare training material, fine-tuning creates adapted models, and evaluation benchmarks them. The README documents additional generation and training capabilities not represented by sampled implementations; those are shown at their documented enclosing paths. Links between independently described stages are included only where the README explicitly documents the transfer.
Main entry points: book-extraction/main.py Book pipeline · eval_cpt_ollama_vllm.py Dual-engine evaluation · finetuning/main.py Fine-tuning pipeline · kcc_scrapper.py KCC API scraper
Components 29
Source ingestion
- KCC API scraper
web-scrapping/kcc/kcc_scrapper.py - Resumable downloader
kcc-pipeline/downloader/cli.py - PIB scraper
web-scrapping/pib/scrapper.py
KCC data preparation
- Noise classifier
kcc-pipeline/java-filter/src/main/java/com/agri/LlmNoiseFilter.java - CSV cleaner
kcc-pipeline/cleaning/cleaner.py - Dataset normalizer
kcc-pipeline/normalization/dataset_normalizer.py - Rule synthesis
kcc-pipeline/data-dictionary/pipeline.py - Dialogue generation
kcc-pipeline/generation - Data validation
kcc-pipeline/generation
Monograph extraction
- TOC detection
book-extraction/toc_detector.py - TOC alignment
book-extraction/toc_extractor.py - Chapter chunking
book-extraction/chunker.py - Visual table repair
book-extraction/table_repair.py - CPT synthesis
book-extraction/cpt_generator.py - Density validation
book-extraction/density_validator.py - Book pipeline
book-extraction/main.py
Model training
- Fine-tuning pipeline
finetuning/main.py - CPT trainer
finetuning/unsloth_cpt_trainer.py - SFT trainer
finetuning/unsloth_sft_trainer.py
Evaluation and export
- Dual-engine evaluation
evaluation/eval_cpt_ollama_vllm.py - GGUF exporter
evaluation/export_gguf.py
Other components
- KCC records
- PIB releases
- Agricultural books
- Prepared datasets
- Trained model
- Data.gov.in
- PIB website
- LLM service
Connections 47
- KCC records → KCC API scraper: provides records
- Data.gov.in → KCC API scraper: serves data
- Data.gov.in → Resumable downloader: serves pages
- PIB releases → PIB scraper: provides releases
- PIB website → PIB scraper: serves pages
- Agricultural books → Book pipeline: supplies PDFs
- KCC API scraper → Prepared datasets: writes batches
- Resumable downloader → Prepared datasets: writes CSV parts
- PIB scraper → Prepared datasets: writes releases
- Prepared datasets → Noise classifier: reads CSV
- LLM service → Noise classifier: classifies queries
- Noise classifier → CSV cleaner: filtered records
- CSV cleaner → Prepared datasets: writes cleaned data
- Prepared datasets → Dataset normalizer: reads dataset
- Dataset normalizer → Prepared datasets: writes normalized data
- Prepared datasets → Rule synthesis: samples queries
- LLM service → Rule synthesis: generates rules
- Rule synthesis → Prepared datasets: updates rules
- Prepared datasets → Dialogue generation: reads records
- LLM service → Dialogue generation: synthesizes text
- Dialogue generation → Prepared datasets: writes training data
- Prepared datasets → Data validation: reads generated data
- Data validation → Prepared datasets: writes validated data
- Book pipeline → TOC detection: runs stage
- Book pipeline → TOC alignment: runs stage
- Book pipeline → Chapter chunking: runs stage
- Book pipeline → Visual table repair: runs stage
- Book pipeline → CPT synthesis: runs stage
- Book pipeline → Density validation: runs stage
- TOC detection → Book pipeline: returns page data
- TOC alignment → Book pipeline: returns aligned TOC
- Chapter chunking → Book pipeline: returns chapters
- Visual table repair → Book pipeline: returns repaired tables
- CPT synthesis → Book pipeline: returns examples
- Density validation → Book pipeline: returns quality result
- Book pipeline → Prepared datasets: writes extraction data
- Prepared datasets → Fine-tuning pipeline: provides training data
- Fine-tuning pipeline → CPT trainer: selects CPT path
- Fine-tuning pipeline → SFT trainer: runs SFT
- CPT trainer → Trained model: writes adapter
- SFT trainer → Trained model: writes model
- Trained model → Dual-engine evaluation: candidate model
- Dual-engine evaluation → LLM service: queries candidate
- Dual-engine evaluation → LLM service: queries examiner
- Dual-engine evaluation → Prepared datasets: writes reports
- Trained model → GGUF exporter: reads model
- GGUF exporter → Trained model: writes GGUF