# rishabhpathak93/image_captioning architecture

> Architecture diagram of the GitHub repository rishabhpathak93/image_captioning, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-10). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rishabhpathak93/image_captioning
- Repository: https://github.com/rishabhpathak93/image_captioning

## Overview

##### Main Components
- Inference Service (app.py): This is likely the entry point for generating captions on new images. It acts as the interface for external input (images) and returns generated captions.
- Model Artifacts (models/ directory): Contains the saved CNN and LSTM models as well as associated data (e.g., tokenizer, feature cache, max_length, and other pickled objects). These are key components for both feature extraction and caption generation.
- Training & Evaluation Pipelines (src/ directory):
– train.py: Handles training of the deep learning models.
– preprocessing.py: Takes care of data ingestion, image processing, and preparing the text data for model training.
– evaluate.py: Used to assess model performance, ensuring that the system is generating accurate captions.

##### Relationships and Interactions
- Data Flow for Training:
– Data (images and possibly captions) enters the preprocessing module.
– Preprocessing outputs processed images and textual data which are then fed into the training pipeline.
– Training generates/updates model artifacts (CNN and LSTM models, tokenizers, etc.) and saves them in the models directory.
– Evaluation reads from these models and validates the output.
- Inference Flow:
– The app.py service accepts new image inputs.
– It loads the pre-trained CNN and LSTM models from the models directory.
– The image is preprocessed (possibly using routines similar to those in preprocessing.py) and passed through the CNN to extract features.
– These features are used by the LSTM to generate a caption.
– The final caption is returned as a response through the service.

##### Architectural Patterns and Design Principles
- Modular Design: The separation between training (src/) and serving (app.py) ensures clear responsibilities for model development and deployment.
- Reusability: Model artifacts (e.g., CNN, LSTM, tokenizer) are stored separately, making them easy to update, version, or replace without impacting the service interface.
- Pipeline Flow: The structure naturally follows a machine learning pipeline — preprocessing, training, evaluation, and deployment/inference.

##### Relevant Technologies and Frameworks
- Deep Learning Frameworks: Although not explicitly mentioned, the use of .h5 models hints at Keras/TensorFlow.
- Serialization/Processing: Pickle files for features and tokenizers suggest use of Python’s pickle module for saving and loading intermediate computations.
- Python-based Service: The presence of app.py implies that an HTTP server or API endpoint (possibly Flask or FastAPI) is used to serve the model predictions.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Client
    Client["External Client (Image Input)"]:::external

    %% Inference Flow
    InferenceService["Inference Service (app.py)"]:::inference
    CaptionOutput["Caption Output (Image Caption)"]:::external

    %% Model Artifacts Subgraph
    subgraph "Model Artifacts"
        CNN["CNN Model (.h5)"]:::artifact
        LSTM["LSTM Model (.h5)"]:::artifact
        Features["Features (.pkl)"]:::artifact
        MaxLength["Max Length (.pkl)"]:::artifact
        Tokenizer["Tokenizer (.pkl)"]:::artifact
    end

    %% Training Pipeline Subgraph
    subgraph "Training Pipeline"
        Preproc["Preprocessing Module (src/preprocessing.py)"]:::training
        TrainModule["Training Module (src/train.py)"]:::training
        EvalModule["Evaluation Module (src/evaluate.py)"]:::training
    end

    %% Inference Data Flow
    Client -->|"sends image"| InferenceService
    InferenceService -->|"loads_CNN_model"| CNN
    InferenceService -->|"loads_LSTM_model"| LSTM
    InferenceService -->|"loads_Features"| Features
    InferenceService -->|"loads_MaxLength"| MaxLength
    InferenceService -->|"loads_Tokenizer"| Tokenizer
    InferenceService -->|"returns_caption"| CaptionOutput

    %% Training Data Flow
    Preproc -->|"processes_data"| TrainModule
    TrainModule -->|"updates_CNN"| CNN
    TrainModule -->|"updates_LSTM"| LSTM
    TrainModule -->|"updates_Features"| Features
    TrainModule -->|"updates_MaxLength"| MaxLength
    TrainModule -->|"updates_Tokenizer"| Tokenizer
    CNN ---|"evaluated_by"| EvalModule
    LSTM ---|"evaluated_by"| EvalModule
    EvalModule -->|"feedback_loop"| TrainModule

    %% Click Events
    click InferenceService "https://github.com/rishabhpathak93/image_captioning/blob/main/app.py"
    click CNN "https://github.com/rishabhpathak93/image_captioning/blob/main/models/cnn_model.h5"
    click LSTM "https://github.com/rishabhpathak93/image_captioning/blob/main/models/lstm_model.h5"
    click Features "https://github.com/rishabhpathak93/image_captioning/blob/main/models/features.pkl"
    click MaxLength "https://github.com/rishabhpathak93/image_captioning/blob/main/models/max_length.pkl"
    click Tokenizer "https://github.com/rishabhpathak93/image_captioning/blob/main/models/tokenizer.pkl"
    click Preproc "https://github.com/rishabhpathak93/image_captioning/blob/main/src/preprocessing.py"
    click TrainModule "https://github.com/rishabhpathak93/image_captioning/blob/main/src/train.py"
    click EvalModule "https://github.com/rishabhpathak93/image_captioning/blob/main/src/evaluate.py"

    %% Styles
    classDef external fill:#FFF79A,stroke:#D6D600,stroke-width:2px;
    classDef inference fill:#B9F6CA,stroke:#00C853,stroke-width:2px;
    classDef training fill:#B3E5FC,stroke:#039BE5,stroke-width:2px;
    classDef artifact fill:#E1BEE7,stroke:#8E24AA,stroke-width:2px;
```
