# basava-code/agri-lm architecture

> Architecture diagram of the GitHub repository basava-code/agri-lm, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-29). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/basava-code/agri-lm
- Repository: https://github.com/basava-code/agri-lm

## Overview

Agri-LM is a collection of agricultural data ingestion, knowledge-dataset preparation, monograph extraction, model fine-tuning, and evaluation tools. The user-facing workflow begins with KCC/API or policy scraping and institutional book inputs; ETL and extraction prepare training material, fine-tuning creates adapted models, and evaluation benchmarks them. The README documents additional generation and training capabilities not represented by sampled implementations; those are shown at their documented enclosing paths. Links between independently described stages are included only where the README explicitly documents the transfer.

Main entry points: [`book-extraction/main.py`](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/main.py) (Book pipeline), [`evaluation/eval_cpt_ollama_vllm.py`](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/evaluation/eval_cpt_ollama_vllm.py) (Dual-engine evaluation), [`finetuning/main.py`](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/finetuning/main.py) (Fine-tuning pipeline), [`web-scrapping/kcc/kcc_scrapper.py`](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/web-scrapping/kcc/kcc_scrapper.py) (KCC API scraper)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_ingest["Source ingestion"]
  node_kcc_scrape["KCC API scraper<br/>[kcc_scrapper.py]"]
  node_kcc_download["Resumable downloader<br/>[cli.py]"]
  node_pib_scrape["PIB scraper<br/>[scrapper.py]"]
end

subgraph group_etl["KCC data preparation"]
  node_noise_filter["Noise classifier"]
  node_cleaner["CSV cleaner<br/>[cleaner.py]"]
  node_normalizer["Dataset normalizer"]
  node_dictionary["Rule synthesis<br/>[pipeline.py]"]
  node_dialogue_generation["Dialogue generation"]
  node_duplicate_validation["Data validation"]
end

subgraph group_books["Monograph extraction"]
  node_toc_detection["TOC detection<br/>[toc_detector.py]"]
  node_toc_alignment["TOC alignment<br/>[toc_extractor.py]"]
  node_chunking["Chapter chunking<br/>[chunker.py]"]
  node_table_repair["Visual table repair<br/>[table_repair.py]"]
  node_cpt_generation["CPT synthesis<br/>[cpt_generator.py]"]
  node_density_gate["Density validation"]
  node_book_orchestrator["Book pipeline<br/>[main.py]"]
end

subgraph group_train["Model training"]
  node_finetuning["Fine-tuning pipeline<br/>[main.py]"]
  node_cpt_trainer["CPT trainer"]
  node_sft_trainer["SFT trainer"]
end

subgraph group_eval["Evaluation and export"]
  node_evaluation["Dual-engine evaluation"]
  node_gguf_export["GGUF exporter<br/>[export_gguf.py]"]
end

node_farmer_data(("KCC records"))
node_policy_source(("PIB releases"))
node_monographs(("Agricultural books"))
node_data_store[("Prepared datasets")]
node_model_store[("Trained model")]
node_gov_api(("Data.gov.in"))
node_pib_web(("PIB website"))
node_llm_service(("LLM service"))

node_farmer_data -->|"provides records"| node_kcc_scrape
node_gov_api -->|"serves data"| node_kcc_scrape
node_gov_api -->|"serves pages"| node_kcc_download
node_policy_source -->|"provides releases"| node_pib_scrape
node_pib_web -->|"serves pages"| node_pib_scrape
node_monographs -->|"supplies PDFs"| node_book_orchestrator
node_kcc_scrape -->|"writes batches"| node_data_store
node_kcc_download -->|"writes CSV parts"| node_data_store
node_pib_scrape -->|"writes releases"| node_data_store
node_data_store -->|"reads CSV"| node_noise_filter
node_llm_service -.->|"classifies queries"| node_noise_filter
node_noise_filter -->|"filtered records"| node_cleaner
node_cleaner -->|"writes cleaned data"| node_data_store
node_data_store -->|"reads dataset"| node_normalizer
node_normalizer -->|"writes normalized data"| node_data_store
node_data_store -->|"samples queries"| node_dictionary
node_llm_service -.->|"generates rules"| node_dictionary
node_dictionary -->|"updates rules"| node_data_store
node_data_store -->|"reads records"| node_dialogue_generation
node_llm_service -.->|"synthesizes text"| node_dialogue_generation
node_dialogue_generation -->|"writes training data"| node_data_store
node_data_store -->|"reads generated data"| node_duplicate_validation
node_duplicate_validation -->|"writes validated data"| node_data_store
node_book_orchestrator -->|"runs stage"| node_toc_detection
node_book_orchestrator -->|"runs stage"| node_toc_alignment
node_book_orchestrator -->|"runs stage"| node_chunking
node_book_orchestrator -->|"runs stage"| node_table_repair
node_book_orchestrator -->|"runs stage"| node_cpt_generation
node_book_orchestrator -->|"runs stage"| node_density_gate
node_toc_detection -->|"returns page data"| node_book_orchestrator
node_toc_alignment -->|"returns aligned TOC"| node_book_orchestrator
node_chunking -->|"returns chapters"| node_book_orchestrator
node_table_repair -->|"returns repaired tables"| node_book_orchestrator
node_cpt_generation -->|"returns examples"| node_book_orchestrator
node_density_gate -->|"returns quality result"| node_book_orchestrator
node_book_orchestrator -->|"writes extraction data"| node_data_store
node_data_store -->|"provides training data"| node_finetuning
node_finetuning -.->|"selects CPT path"| node_cpt_trainer
node_finetuning -->|"runs SFT"| node_sft_trainer
node_cpt_trainer -->|"writes adapter"| node_model_store
node_sft_trainer -->|"writes model"| node_model_store
node_model_store -->|"candidate model"| node_evaluation
node_evaluation -.->|"queries candidate"| node_llm_service
node_evaluation -.->|"queries examiner"| node_llm_service
node_evaluation -->|"writes reports"| node_data_store
node_model_store -->|"reads model"| node_gguf_export
node_gguf_export -->|"writes GGUF"| node_model_store

click node_kcc_scrape "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/web-scrapping/kcc/kcc_scrapper.py"
click node_kcc_download "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/kcc-pipeline/downloader/cli.py"
click node_pib_scrape "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/web-scrapping/pib/scrapper.py"
click node_noise_filter "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/kcc-pipeline/java-filter/src/main/java/com/agri/LlmNoiseFilter.java"
click node_cleaner "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/kcc-pipeline/cleaning/cleaner.py"
click node_normalizer "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/kcc-pipeline/normalization/dataset_normalizer.py"
click node_dictionary "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/kcc-pipeline/data-dictionary/pipeline.py"
click node_dialogue_generation "https://github.com/basava-code/agri-lm/tree/agri-lm-v1/kcc-pipeline/generation"
click node_duplicate_validation "https://github.com/basava-code/agri-lm/tree/agri-lm-v1/kcc-pipeline/generation"
click node_toc_detection "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/toc_detector.py"
click node_toc_alignment "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/toc_extractor.py"
click node_chunking "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/chunker.py"
click node_table_repair "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/table_repair.py"
click node_cpt_generation "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/cpt_generator.py"
click node_density_gate "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/density_validator.py"
click node_book_orchestrator "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/main.py"
click node_finetuning "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/finetuning/main.py"
click node_cpt_trainer "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/finetuning/unsloth_cpt_trainer.py"
click node_sft_trainer "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/finetuning/unsloth_sft_trainer.py"
click node_evaluation "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/evaluation/eval_cpt_ollama_vllm.py"
click node_gguf_export "https://github.com/basava-code/agri-lm/blob/agri-lm-v1/evaluation/export_gguf.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_kcc_scrape,node_kcc_download,node_pib_scrape toneBlue
class node_noise_filter,node_cleaner,node_normalizer,node_dictionary,node_dialogue_generation,node_duplicate_validation,node_data_store,node_model_store toneAmber
class node_toc_detection,node_toc_alignment,node_chunking,node_table_repair,node_cpt_generation,node_density_gate,node_book_orchestrator toneMint
class node_finetuning,node_cpt_trainer,node_sft_trainer toneRose
class node_evaluation,node_gguf_export,node_farmer_data,node_policy_source,node_monographs,node_gov_api,node_pib_web,node_llm_service toneIndigo
```

## Components

### Source ingestion

- [KCC API scraper](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/web-scrapping/kcc/kcc_scrapper.py): `web-scrapping/kcc/kcc_scrapper.py`
- [Resumable downloader](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/kcc-pipeline/downloader/cli.py): `kcc-pipeline/downloader/cli.py`
- [PIB scraper](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/web-scrapping/pib/scrapper.py): `web-scrapping/pib/scrapper.py`

### KCC data preparation

- [Noise classifier](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/kcc-pipeline/java-filter/src/main/java/com/agri/LlmNoiseFilter.java): `kcc-pipeline/java-filter/src/main/java/com/agri/LlmNoiseFilter.java`
- [CSV cleaner](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/kcc-pipeline/cleaning/cleaner.py): `kcc-pipeline/cleaning/cleaner.py`
- [Dataset normalizer](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/kcc-pipeline/normalization/dataset_normalizer.py): `kcc-pipeline/normalization/dataset_normalizer.py`
- [Rule synthesis](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/kcc-pipeline/data-dictionary/pipeline.py): `kcc-pipeline/data-dictionary/pipeline.py`
- [Dialogue generation](https://github.com/basava-code/agri-lm/tree/agri-lm-v1/kcc-pipeline/generation): `kcc-pipeline/generation`
- [Data validation](https://github.com/basava-code/agri-lm/tree/agri-lm-v1/kcc-pipeline/generation): `kcc-pipeline/generation`

### Monograph extraction

- [TOC detection](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/toc_detector.py): `book-extraction/toc_detector.py`
- [TOC alignment](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/toc_extractor.py): `book-extraction/toc_extractor.py`
- [Chapter chunking](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/chunker.py): `book-extraction/chunker.py`
- [Visual table repair](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/table_repair.py): `book-extraction/table_repair.py`
- [CPT synthesis](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/cpt_generator.py): `book-extraction/cpt_generator.py`
- [Density validation](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/density_validator.py): `book-extraction/density_validator.py`
- [Book pipeline](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/book-extraction/main.py): `book-extraction/main.py`

### Model training

- [Fine-tuning pipeline](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/finetuning/main.py): `finetuning/main.py`
- [CPT trainer](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/finetuning/unsloth_cpt_trainer.py): `finetuning/unsloth_cpt_trainer.py`
- [SFT trainer](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/finetuning/unsloth_sft_trainer.py): `finetuning/unsloth_sft_trainer.py`

### Evaluation and export

- [Dual-engine evaluation](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/evaluation/eval_cpt_ollama_vllm.py): `evaluation/eval_cpt_ollama_vllm.py`
- [GGUF exporter](https://github.com/basava-code/agri-lm/blob/agri-lm-v1/evaluation/export_gguf.py): `evaluation/export_gguf.py`

### Other components

- **KCC records**
- **PIB releases**
- **Agricultural books**
- **Prepared datasets**
- **Trained model**
- **Data.gov.in**
- **PIB website**
- **LLM service**

## Connections

- KCC records → KCC API scraper: provides records
- Data.gov.in → KCC API scraper: serves data
- Data.gov.in → Resumable downloader: serves pages
- PIB releases → PIB scraper: provides releases
- PIB website → PIB scraper: serves pages
- Agricultural books → Book pipeline: supplies PDFs
- KCC API scraper → Prepared datasets: writes batches
- Resumable downloader → Prepared datasets: writes CSV parts
- PIB scraper → Prepared datasets: writes releases
- Prepared datasets → Noise classifier: reads CSV
- LLM service → Noise classifier: classifies queries
- Noise classifier → CSV cleaner: filtered records
- CSV cleaner → Prepared datasets: writes cleaned data
- Prepared datasets → Dataset normalizer: reads dataset
- Dataset normalizer → Prepared datasets: writes normalized data
- Prepared datasets → Rule synthesis: samples queries
- LLM service → Rule synthesis: generates rules
- Rule synthesis → Prepared datasets: updates rules
- Prepared datasets → Dialogue generation: reads records
- LLM service → Dialogue generation: synthesizes text
- Dialogue generation → Prepared datasets: writes training data
- Prepared datasets → Data validation: reads generated data
- Data validation → Prepared datasets: writes validated data
- Book pipeline → TOC detection: runs stage
- Book pipeline → TOC alignment: runs stage
- Book pipeline → Chapter chunking: runs stage
- Book pipeline → Visual table repair: runs stage
- Book pipeline → CPT synthesis: runs stage
- Book pipeline → Density validation: runs stage
- TOC detection → Book pipeline: returns page data
- TOC alignment → Book pipeline: returns aligned TOC
- Chapter chunking → Book pipeline: returns chapters
- Visual table repair → Book pipeline: returns repaired tables
- CPT synthesis → Book pipeline: returns examples
- Density validation → Book pipeline: returns quality result
- Book pipeline → Prepared datasets: writes extraction data
- Prepared datasets → Fine-tuning pipeline: provides training data
- Fine-tuning pipeline → CPT trainer: selects CPT path
- Fine-tuning pipeline → SFT trainer: runs SFT
- CPT trainer → Trained model: writes adapter
- SFT trainer → Trained model: writes model
- Trained model → Dual-engine evaluation: candidate model
- Dual-engine evaluation → LLM service: queries candidate
- Dual-engine evaluation → LLM service: queries examiner
- Dual-engine evaluation → Prepared datasets: writes reports
- Trained model → GGUF exporter: reads model
- GGUF exporter → Trained model: writes GGUF