How kshandilya19/animalrag works
Project type and purpose
• ANIMALNET is an end-to-end video analytics and narration pipeline. It ingests a video, detects and tracks animals, classifies their species and scene context, then generates a one-sentence narration grounded in a retrieval-augmented generation (RAG) knowledge base.
File structure
• Top-level folders:
• AnimalClassification/ – YOLOv8 detector/tracker, ConvNeXt species classifier, orchestration scripts
• SceneClassification/ – ConvNeXt scene classifier weights & metadata
• RAG/ – FAISS index, knowledge corpus, vectorization & generation code
• pipeline/ – full_pipeline.py entrypoint (ties all modules together)
• ANIMALNET_release/ (per README) – requirements, run_pipeline.py (user entrypoint)
• Models (.pth, .pt), data files (.json, .csv), config (.yaml), cache/indices (.faiss) indicate a modular, stages-in-series architecture.
README for additional insights
• Confirms a single-process pipeline with stages: detection→classification→RAG.
• External dependency: Google Gemini API.
• Data stores: FAISS index + JSON corpora.
• Outputs: tracked video, JSON result.
a. Main components:
• Video Ingest (input)
• Detection & Tracking Module (YOLOv8 + custom_tracker.yaml)
• Species Classification Module (ConvNeXt; train_awa2.py / testmodel.py)
• Scene Classification Module (best_scene_model.pth + scene_classes.json)
• RAG Module
– Vectorizer (sentence-transformers → eduwild_corpus.faiss)
– Retriever (FAISS index)
– Generator (Google Gemini via RAG/rag.py)
• Orchestrator (pipeline/full_pipeline.py or ANIMALNET_release/run_pipeline.py)
• Output Writer (writes JSON, video overlays)
b. Relationships & data flow:
1. Raw video → Detector & Tracker
3. Video frames → Scene Classifier (parallel)
4. Species + Scene + optional “behavior” → RAG Module
– Query vectorizer → FAISS retrieve → Gemini LLM → narration
5. Orchestrator aggregates all stage outputs → JSON & final video
c. Architectural patterns & principles:
• Modular pipeline (each stage decoupled as its own component)
• Data-oriented flow: intermediate files/models cached for reproducibility
• External service integration (Google Gemini API)
d. Key technologies & libraries:
• Python 3.10–3.12, PyTorch, timm, YOLOv8 (ultralytics), faiss-cpu, sentence-transformers, google-generativeai