How anion-coder/cdac_image_captioning works
Project Type and Purpose
• Key goal: Enable blind/visually impaired students to “hear” detailed yet simplified descriptions of visual educational content.
File Structure Analysis
• Only two top‐level files:
• Image_captioning_with_TTS.ipynb – the Colab notebook containing all code, model loading, and inference logic.
• README.md – documents pipeline stages, models, dependencies, and usage.
• No separate frontend/backend directories: it’s a data-and-models pipeline orchestrated in Jupyter.
README Insights
• The architecture is described as a linear, modular pipeline with six core components:
1. Vision‐Language Model
2. Text Complexity Classifier
3. Sentence Simplifier
4. Domain Term Annotator
5. Punctuation Restorer
6. Parler‐TTS (Text-to-Speech)
• Dependencies: PyTorch, Transformers, Sentence-Transformers, KeyBERT, NLTK, spaCy, WordFreq, Wikipedia API, Parler-TTS.
1. Input: Chart Image (e.g., ocean temperature depth chart)
2. Vision-Language Model (vLLM: Meta-LLaMA-4-Maverick)
3. Text Complexity Classifier (TCM-NN: BERT + handcrafted readability features)
4. Sentence Simplifier (Complex Word ID + BERT-MLM + Zipf frequency)
5. Domain Term Annotator (KeyBERT → Wikipedia/WordNet definitions)
6. Punctuation Restorer (for prosody)
7. Parler-TTS Engine (Indian‐accented audio output)
8. Output: Accessible Audio + Simplified Text (for debugging/preview)
• Image → Vision-Language Model → Raw Description
• Raw Description → Text Complexity Classifier → Complexity Scores
• Raw Description + Scores → Sentence Simplifier → Simplified Text
• Simplified Text → Domain Term Annotator → Annotated Text
• Annotated Text → Punctuation Restorer → Punctuated Text
• Punctuated Text → Parler-TTS → Audio Output
c. Architectural Patterns & Design Principles:
• Modularity: Each stage is a standalone module with a well‐defined input/output interface.
• Pipeline (linear): Sequential transformation ensures clarity, easy debugging, and per‐module evaluation.
• Data Validation/Logging: Implicitly log intermediate outputs (text, scores) for QA and metrics.
d. Technologies & Frameworks (annotate each component with its stack):
• Vision-Language: PyTorch + HuggingFace Transformers (Meta-LLaMA)
• Complexity Classifier: PyTorch, Sentence-Transformers, sklearn (for handcrafted feature pipeline)
• Simplification: HuggingFace BERT MLM, WordFreq (Zipf), custom BiLSTM for CWI
• Term Annotator: KeyBERT, Wikipedia API, NLTK/WordNet
• Punctuation Restorer: seq2seq or rule‐based punctuation model (NLTK/spacy)
• TTS: Parler-TTS library wrapping PyTorch models