# kshandilya19/animalrag architecture

> Architecture diagram of the GitHub repository kshandilya19/animalrag, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-11-26). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kshandilya19/animalrag
- Repository: https://github.com/kshandilya19/animalrag

## Overview

##### Project type and purpose
- ANIMALNET is an end-to-end video analytics and narration pipeline. It ingests a video, detects and tracks animals, classifies their species and scene context, then generates a one-sentence narration grounded in a retrieval-augmented generation (RAG) knowledge base.

##### File structure
- Top-level folders:
- AnimalClassification/ – YOLOv8 detector/tracker, ConvNeXt species classifier, orchestration scripts
- SceneClassification/ – ConvNeXt scene classifier weights & metadata
- RAG/ – FAISS index, knowledge corpus, vectorization & generation code
- pipeline/ – full_pipeline.py entrypoint (ties all modules together)
- ANIMALNET_release/ (per README) – requirements, run_pipeline.py (user entrypoint)
- Models (.pth, .pt), data files (.json, .csv), config (.yaml), cache/indices (.faiss) indicate a modular, stages-in-series architecture.

##### README for additional insights
- Confirms a single-process pipeline with stages: detection→classification→RAG.
- External dependency: Google Gemini API.
- Data stores: FAISS index + JSON corpora.
- Outputs: tracked video, JSON result.

a. Main components:
- Video Ingest (input)
- Detection & Tracking Module (YOLOv8 + custom_tracker.yaml)
- Species Classification Module (ConvNeXt; train_awa2.py / testmodel.py)
- Scene Classification Module (best_scene_model.pth + scene_classes.json)
- RAG Module
– Vectorizer (sentence-transformers → eduwild_corpus.faiss)
– Retriever (FAISS index)
– Generator (Google Gemini via RAG/rag.py)
- Orchestrator (pipeline/full_pipeline.py or ANIMALNET_release/run_pipeline.py)
- Output Writer (writes JSON, video overlays)
b. Relationships & data flow:
1. Raw video → Detector & Tracker
3. Video frames → Scene Classifier (parallel)
4. Species + Scene + optional “behavior” → RAG Module
– Query vectorizer → FAISS retrieve → Gemini LLM → narration
5. Orchestrator aggregates all stage outputs → JSON & final video
c. Architectural patterns & principles:
- Modular pipeline (each stage decoupled as its own component)
- Data-oriented flow: intermediate files/models cached for reproducibility
- External service integration (Google Gemini API)
d. Key technologies & libraries:
- Python 3.10–3.12, PyTorch, timm, YOLOv8 (ultralytics), faiss-cpu, sentence-transformers, google-generativeai

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Input
    VideoIngest["Video Ingest (Input Loader)"]:::io
    click VideoIngest "https://github.com/kshandilya19/animalrag/blob/main/pipeline/full_pipeline.py"

    %% Detection & Tracking
    subgraph "Detection & Tracking Module"
        Detection["video_pipeline.py"]:::compute
        TrackerConfig["custom_tracker.yaml"]:::data
        YOLOModel["yolov8n.pt"]:::data
    end
    click Detection "https://github.com/kshandilya19/animalrag/blob/main/AnimalClassification/video_pipeline.py"
    click TrackerConfig "https://github.com/kshandilya19/animalrag/blob/main/AnimalClassification/custom_tracker.yaml"
    click YOLOModel "https://github.com/kshandilya19/animalrag/blob/main/AnimalClassification/yolov8n.pt"

    %% Species Classification
    subgraph "Species Classification Module"
        TrainAWA2["train_awa2.py"]:::compute
        TestModel["testmodel.py"]:::compute
        Classes["classes.py"]:::compute
        ConvNextBest["awa2_convnext_best.pth"]:::data
        ConvNextFinal["awa2_convnext_final.pth"]:::data
        ConvNextTiny["awa2_convnext_tiny.pth"]:::data
    end
    click TrainAWA2 "https://github.com/kshandilya19/animalrag/blob/main/AnimalClassification/train_awa2.py"
    click TestModel "https://github.com/kshandilya19/animalrag/blob/main/AnimalClassification/testmodel.py"
    click Classes "https://github.com/kshandilya19/animalrag/blob/main/AnimalClassification/classes.py"
    click ConvNextBest "https://github.com/kshandilya19/animalrag/blob/main/AnimalClassification/models/awa2_convnext_best.pth"
    click ConvNextFinal "https://github.com/kshandilya19/animalrag/blob/main/AnimalClassification/models/awa2_convnext_final.pth"
    click ConvNextTiny "https://github.com/kshandilya19/animalrag/blob/main/AnimalClassification/models/awa2_convnext_tiny.pth"

    %% Scene Classification
    subgraph "Scene Classification Module"
        SceneModel["best_scene_model.pth"]:::data
        SceneClasses["scene_classes.json"]:::data
    end
    click SceneModel "https://github.com/kshandilya19/animalrag/blob/main/SceneClassification/best_scene_model.pth"
    click SceneClasses "https://github.com/kshandilya19/animalrag/blob/main/SceneClassification/scene_classes.json"

    %% RAG Module
    subgraph "RAG Module"
        Vectorize["vectorize.py"]:::compute
        Retriever["Retriever (FAISS)"]:::compute
        FAISSIndex["eduwild_corpus.faiss"]:::data
        CorpusData["eduwild_corpus_data.json"]:::data
        Data1["eduwild_data.json"]:::data
        CorpusExp["eduwild_knowledge_corpus_expanded.json"]:::data
        FactsLoader["facts.py"]:::compute
        AnimalFactsCSV["animal_facts_fixed.csv"]:::data
        AnimalFactsJSON["animal_facts_fixed.json"]:::data
        Generator["rag.py"]:::compute
        Gemini["Google Gemini API"]:::external
    end
    click Vectorize "https://github.com/kshandilya19/animalrag/blob/main/RAG/vectorize.py"
    click FAISSIndex "https://github.com/kshandilya19/animalrag/blob/main/RAG/eduwild_corpus.faiss"
    click CorpusData "https://github.com/kshandilya19/animalrag/blob/main/RAG/eduwild_corpus_data.json"
    click Data1 "https://github.com/kshandilya19/animalrag/blob/main/RAG/eduwild_data.json"
    click CorpusExp "https://github.com/kshandilya19/animalrag/blob/main/RAG/eduwild_knowledge_corpus_expanded.json"
    click FactsLoader "https://github.com/kshandilya19/animalrag/blob/main/RAG/facts.py"
    click AnimalFactsCSV "https://github.com/kshandilya19/animalrag/blob/main/RAG/animal_facts_fixed.csv"
    click AnimalFactsJSON "https://github.com/kshandilya19/animalrag/blob/main/RAG/animal_facts_fixed.json"
    click Generator "https://github.com/kshandilya19/animalrag/blob/main/RAG/rag.py"
    click Gemini "https://github.com/kshandilya19/animalrag/blob/main/RAG/gemini.py"

    %% Output
    OutputWriter["Output Writer (JSON & Video)"]:::io
    click OutputWriter "https://github.com/kshandilya19/animalrag/blob/main/pipeline/full_pipeline.py"

    %% Data Flow
    VideoIngest --> Detection
    Detection --> TestModel
    TestModel --> ConvNextBest
    TestModel --> ConvNextFinal
    TestModel --> ConvNextTiny
    VideoIngest --> SceneModel
    SceneModel --> SceneClasses

    TestModel --> Vectorize
    SceneModel --> Vectorize
    Vectorize --> Retriever
    Retriever --> FAISSIndex
    Retriever --> CorpusData
    Retriever --> Data1
    Retriever --> CorpusExp
    Retriever --> Generator
    FactsLoader --> Generator
    AnimalFactsCSV --> FactsLoader
    AnimalFactsJSON --> FactsLoader
    Generator --> Gemini
    Generator --> OutputWriter

    classDef compute fill:#a8d0e6,stroke:#000;
    classDef data fill:#a8e6a8,stroke:#000;
    classDef external fill:#f4a261,stroke:#000;
    classDef io fill:#cdb4db,stroke:#000;
```
