How niharikachaudhari13/image-captioning-generator works
Project type and purpose
• This is a machine‐learning application with both a Web UI and a command‐line interface.
• Purpose: generate natural‐language captions for user‐provided images using a CNN+LSTM model, support multiple decoding strategies, translation, and text‐to‐speech.
File structure
• Top level:
• main.py → training pipeline (load data, train Xception+LSTM, save model)
• app.py → Streamlit web front end (user uploads, caption display, download options, TTS)
• test.py → CLI test harness (batch or single‐image captioning using different strategies)
• requirements.txt → dependency list (Streamlit, TensorFlow/Keras, gTTS, numpy, pandas…)
• README.md → high‐level project description and usage
• No nested directories; conceptual modules live inside these scripts.
• External dependencies: pretrained Xception weights, Flickr8k dataset, gTTS service.
README for additional insights
• Architecture: encoder (Xception without top) extracts 2048D features → decoder (Embedding + LSTM + Dense).
• Captioning options: greedy search, beam search, nucleus sampling.
• Translation to four languages.
• TTS via gTTS.
• UI based on Streamlit.
• Workflow: download data → run main.py to train → run test.py or app.py.
a. Main components
• Data Store: Flickr8k images + captions
• Training Pipeline (main.py)
• Trained Model Artifacts (encoder weights + decoder weights + tokenizer)
• Feature Extraction Service (Xception)
• Caption Generation Service (LSTM with decoding strategies)
• Translation Module
• Text‐to‐Speech Module (gTTS API)
• Streamlit UI (app.py)
• CLI Test Module (test.py)
b. Relationships and interactions
1. Training: Data Store → Training Pipeline → Model Artifacts
2. Inference (UI): User uploads image → Feature Extraction → Caption Generation → (optional) Translation → (optional) TTS → UI display/download
3. Inference (CLI): test.py loads Model Artifacts → runs same Feature Extraction & Caption Generation → outputs .txt/.csv
c. Architectural patterns/design principles
• Separation of concerns: Training vs. Inference vs. Presentation
• Modular pipeline: distinct services for extraction, decoding, translation, TTS
• Reuse of pretrained CNN as encoder
d. Key technologies/frameworks
• TensorFlow/Keras (Xception, LSTM)
• Streamlit (UI)
• gTTS (TTS)
• Python scripting for CLI and training
Details to capture
• Input formats (raw image → tensor → 2048D feature vector).
• Decoding strategies as parallel paths or a decision node.
• Model artifacts storage location.
• Deployment context: local execution for both UI and CLI.