How mouadbra/multimodal-app works
This is a browser-based multimodal generation and evaluation product. In the demo workflow, a user records audio; the frontend submits it for transcription, then requests image generation, image analysis, and speech synthesis, presenting the results. A separate evaluation workflow launches prompt batches, evaluates generated images, persists results and metrics, and displays batch dashboards. The backend excerpts show the endpoint implementations and SQLite access; evaluation orchestration details are partly unsampled, so those links follow the README’s documented flow.
Main entry points: App.tsx Pipeline & Dashboard UI · main.py Batch Endpoint · common.py Modal Volume
Components 21
Frontend
- Pipeline & Dashboard UI
frontend_service/src/App.tsx
Demo Pipeline
- Whisper Transcription
backend_service/src/modal_app/main.py - Image Generation
backend_service/src/modal_app/main.py - Image Analysis
backend_service/src/modal_app/main.py - Speech Synthesis
backend_service/src/modal_app/main.py
Evaluation
- Batch Endpoint
backend_service/src/modal_app/main.py - Iteration Evaluation
backend_service/src/modal_app/main.py - Objective Evaluation
backend_service/src/modal_app/main.py - Image Feedback
backend_service/src/modal_app/main.py - Metric Aggregation
backend_service/src/modal_app/main.py - Batch Listing
backend_service/src/modal_app/main.py - Batch Results
backend_service/src/modal_app/main.py
Persistence
- Evaluation SQLite
backend_service/src/modal_app/main.py
Other components
- User
- Microphone Audio
- Azure Whisper
- Azure FLUX
- CLIP Similarity
- GPT-4o Vision
- Azure TTS
- Modal Volume
backend_service/src/modal_app/common.py
Connections 33
- User → Pipeline & Dashboard UI: uses
- Pipeline & Dashboard UI → Microphone Audio: records
- Microphone Audio → Pipeline & Dashboard UI: provides audio
- Pipeline & Dashboard UI → Whisper Transcription: POST audio
- Whisper Transcription → Azure Whisper: transcribes
- Whisper Transcription → Pipeline & Dashboard UI: returns transcript
- Pipeline & Dashboard UI → Image Generation: requests image
- Image Generation → Azure FLUX: generates image
- Image Generation → Pipeline & Dashboard UI: returns image
- Pipeline & Dashboard UI → Image Analysis: submits image
- Image Analysis → CLIP Similarity: scores similarity
- Image Analysis → GPT-4o Vision: describes image
- Image Analysis → Pipeline & Dashboard UI: returns analysis
- Pipeline & Dashboard UI → Speech Synthesis: submits description
- Speech Synthesis → Azure TTS: synthesizes speech
- Speech Synthesis → Pipeline & Dashboard UI: returns audio
- Pipeline & Dashboard UI → Batch Endpoint: launches batch
- Batch Endpoint → Iteration Evaluation: runs iterations
- Iteration Evaluation → Image Generation: generates image
- Iteration Evaluation → Image Analysis: scores image
- Iteration Evaluation → Objective Evaluation: evaluates criteria
- Iteration Evaluation → Image Feedback: gets feedback
- Batch Endpoint → Evaluation SQLite: writes results
- Batch Endpoint → Metric Aggregation: aggregates scores
- Metric Aggregation → Evaluation SQLite: reads results
- Pipeline & Dashboard UI → Batch Listing: requests batches
- Batch Listing → Evaluation SQLite: reads batches
- Pipeline & Dashboard UI → Batch Results: requests batch
- Batch Results → Evaluation SQLite: reads results
- Evaluation SQLite → Modal Volume: persists on
- Batch Endpoint → Pipeline & Dashboard UI: returns evaluation
- Batch Listing → Pipeline & Dashboard UI: returns batches
- Batch Results → Pipeline & Dashboard UI: returns dashboard data