# anion-coder/cdac_image_captioning architecture

> Architecture diagram of the GitHub repository anion-coder/cdac_image_captioning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-20). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anion-coder/cdac_image_captioning
- Repository: https://github.com/anion-coder/cdac_image_captioning

## Overview

##### Project Type and Purpose
- Key goal: Enable blind/visually impaired students to “hear” detailed yet simplified descriptions of visual educational content.

##### File Structure Analysis
- Only two top‐level files:
- Image_captioning_with_TTS.ipynb – the Colab notebook containing all code, model loading, and inference logic.
- README.md – documents pipeline stages, models, dependencies, and usage.
- No separate frontend/backend directories: it’s a data-and-models pipeline orchestrated in Jupyter.

##### README Insights
- The architecture is described as a linear, modular pipeline with six core components:
1. Vision‐Language Model
2. Text Complexity Classifier
3. Sentence Simplifier
4. Domain Term Annotator
5. Punctuation Restorer
6. Parler‐TTS (Text-to-Speech)
- Dependencies: PyTorch, Transformers, Sentence-Transformers, KeyBERT, NLTK, spaCy, WordFreq, Wikipedia API, Parler-TTS.

1. Input: Chart Image (e.g., ocean temperature depth chart)
2. Vision-Language Model (vLLM: Meta-LLaMA-4-Maverick)
3. Text Complexity Classifier (TCM-NN: BERT + handcrafted readability features)
4. Sentence Simplifier (Complex Word ID + BERT-MLM + Zipf frequency)
5. Domain Term Annotator (KeyBERT → Wikipedia/WordNet definitions)
6. Punctuation Restorer (for prosody)
7. Parler-TTS Engine (Indian‐accented audio output)
8. Output: Accessible Audio + Simplified Text (for debugging/preview)

- Image → Vision-Language Model → Raw Description
- Raw Description → Text Complexity Classifier → Complexity Scores
- Raw Description + Scores → Sentence Simplifier → Simplified Text
- Simplified Text → Domain Term Annotator → Annotated Text
- Annotated Text → Punctuation Restorer → Punctuated Text
- Punctuated Text → Parler-TTS → Audio Output

c. Architectural Patterns & Design Principles:
- **Modularity**: Each stage is a standalone module with a well‐defined input/output interface.
- **Pipeline (linear)**: Sequential transformation ensures clarity, easy debugging, and per‐module evaluation.
- **Data Validation/Logging**: Implicitly log intermediate outputs (text, scores) for QA and metrics.

d. Technologies & Frameworks (annotate each component with its stack):
- Vision-Language: PyTorch + HuggingFace Transformers (Meta-LLaMA)
- Complexity Classifier: PyTorch, Sentence-Transformers, sklearn (for handcrafted feature pipeline)
- Simplification: HuggingFace BERT MLM, WordFreq (Zipf), custom BiLSTM for CWI
- Term Annotator: KeyBERT, Wikipedia API, NLTK/WordNet
- Punctuation Restorer: seq2seq or rule‐based punctuation model (NLTK/spacy)
- TTS: Parler-TTS library wrapping PyTorch models

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Inputs and Side Info
    SideBox["Environment Setup\n(pip install…, nltk/spacy downloads)"]:::io
    README["Pipeline Documentation\n& Dependencies"]:::io

    %% Main Pipeline
    Input["Chart Image"]:::io
    Vision["Vision-Language Model\n(PyTorch + HF Transformers\nMeta-LLaMA-4-Maverick)"]:::visionblue
    Complexity["Text Complexity Classifier\n(PyTorch, Sentence-Transformers,\nsklearn features)"]:::textproc
    Simplifier["Sentence Simplifier\n(BERT-MLM, Zipf frequency,\nComplex Word ID)"]:::textproc
    Annotator["Domain Term Annotator\n(KeyBERT, Wikipedia API,\nNLTK/WordNet)"]:::textproc
    PunctRestorer["Punctuation Restorer\n(seq2seq/rule-based\nNLTK/spacy)"]:::textproc
    TTS["Parler-TTS Engine\n(Indian-accented, PyTorch)"]:::tts
    OutputText["Simplified Text\n(for preview/debug)"]:::io
    OutputAudio["Accessible Audio"]:::io

    %% Layout
    SideBox --> Input
    SideBox --> Vision
    SideBox --> Complexity
    SideBox --> Simplifier
    SideBox --> Annotator
    SideBox --> PunctRestorer
    SideBox --> TTS
    README -.-> Vision
    README -.-> Complexity
    README -.-> Simplifier
    README -.-> Annotator
    README -.-> PunctRestorer
    README -.-> TTS

    Input --> Vision
    Vision -->|"raw description"| Complexity
    Complexity -->|"complexity scores"| Simplifier
    Simplifier -->|"simplified text"| Annotator
    Annotator -->|"annotated text"| PunctRestorer
    PunctRestorer -->|"punctuated text"| TTS
    PunctRestorer -->|"punctuated text"| OutputText
    TTS -->|"audio output"| OutputAudio

    %% Click Events
    click Vision "https://github.com/anion-coder/cdac_image_captioning/blob/main/Image_captioning_with_TTS.ipynb"
    click Complexity "https://github.com/anion-coder/cdac_image_captioning/blob/main/Image_captioning_with_TTS.ipynb"
    click Simplifier "https://github.com/anion-coder/cdac_image_captioning/blob/main/Image_captioning_with_TTS.ipynb"
    click Annotator "https://github.com/anion-coder/cdac_image_captioning/blob/main/Image_captioning_with_TTS.ipynb"
    click PunctRestorer "https://github.com/anion-coder/cdac_image_captioning/blob/main/Image_captioning_with_TTS.ipynb"
    click TTS "https://github.com/anion-coder/cdac_image_captioning/blob/main/Image_captioning_with_TTS.ipynb"
    click README "https://github.com/anion-coder/cdac_image_captioning/blob/main/README.md"

    %% Styles
    classDef visionblue fill:#D0E8FF,stroke:#000,color:#000;
    classDef textproc fill:#D0FFD0,stroke:#000,color:#000;
    classDef tts fill:#FFE0B2,stroke:#000,color:#000;
    classDef io fill:#F0F0F0,stroke:#000,color:#000;
```
