# mouadbra/multimodal-app architecture

> Architecture diagram of the GitHub repository mouadbra/multimodal-app, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/mouadbra/multimodal-app
- Repository: https://github.com/mouadbra/multimodal-app

## Overview

This is a browser-based multimodal generation and evaluation product. In the demo workflow, a user records audio; the frontend submits it for transcription, then requests image generation, image analysis, and speech synthesis, presenting the results. A separate evaluation workflow launches prompt batches, evaluates generated images, persists results and metrics, and displays batch dashboards. The backend excerpts show the endpoint implementations and SQLite access; evaluation orchestration details are partly unsampled, so those links follow the README’s documented flow.

Main entry points: [`frontend_service/src/App.tsx`](https://github.com/mouadbra/multimodal-app/blob/main/frontend_service/src/App.tsx) (Pipeline & Dashboard UI), [`backend_service/src/modal_app/main.py`](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py) (Batch Endpoint), [`backend_service/src/modal_app/common.py`](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/common.py) (Modal Volume)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Frontend"]
  node_app["Pipeline &amp; Dashboard UI<br/>[App.tsx]"]
end

subgraph group_pipeline["Demo Pipeline"]
  node_transcribe["Whisper Transcription<br/>[main.py]"]
  node_generate["Image Generation<br/>[main.py]"]
  node_analyze["Image Analysis<br/>[main.py]"]
  node_speech["Speech Synthesis<br/>[main.py]"]
end

subgraph group_evaluation["Evaluation"]
  node_evaluate["Batch Endpoint<br/>[main.py]"]
  node_iteration["Iteration Evaluation<br/>[main.py]"]
  node_objective["Objective Evaluation<br/>[main.py]"]
  node_describe["Image Feedback<br/>[main.py]"]
  node_metrics["Metric Aggregation<br/>[main.py]"]
  node_batchlist["Batch Listing<br/>[main.py]"]
  node_batchdetail["Batch Results<br/>[main.py]"]
end

subgraph group_storage["Persistence"]
  node_sqlite[("Evaluation SQLite<br/>[main.py]")]
end

node_user(("User"))
node_audio["Microphone Audio"]
node_whisper["Azure Whisper"]
node_flux["Azure FLUX"]
node_clip["CLIP Similarity"]
node_vision["GPT-4o Vision"]
node_tts["Azure TTS"]
node_modalvolume[("Modal Volume<br/>[common.py]")]

node_user -->|"uses"| node_app
node_app -->|"records"| node_audio
node_audio -->|"provides audio"| node_app
node_app -->|"POST audio"| node_transcribe
node_transcribe -->|"transcribes"| node_whisper
node_transcribe -->|"returns transcript"| node_app
node_app -->|"requests image"| node_generate
node_generate -->|"generates image"| node_flux
node_generate -->|"returns image"| node_app
node_app -->|"submits image"| node_analyze
node_analyze -->|"scores similarity"| node_clip
node_analyze -->|"describes image"| node_vision
node_analyze -->|"returns analysis"| node_app
node_app -->|"submits description"| node_speech
node_speech -->|"synthesizes speech"| node_tts
node_speech -->|"returns audio"| node_app
node_app -->|"launches batch"| node_evaluate
node_evaluate -->|"runs iterations"| node_iteration
node_iteration -->|"generates image"| node_generate
node_iteration -->|"scores image"| node_analyze
node_iteration -->|"evaluates criteria"| node_objective
node_iteration -->|"gets feedback"| node_describe
node_evaluate -->|"writes results"| node_sqlite
node_evaluate -->|"aggregates scores"| node_metrics
node_metrics -->|"reads results"| node_sqlite
node_app -->|"requests batches"| node_batchlist
node_batchlist -->|"reads batches"| node_sqlite
node_app -->|"requests batch"| node_batchdetail
node_batchdetail -->|"reads results"| node_sqlite
node_sqlite -->|"persists on"| node_modalvolume
node_evaluate -->|"returns evaluation"| node_app
node_batchlist -->|"returns batches"| node_app
node_batchdetail -->|"returns dashboard data"| node_app

click node_app "https://github.com/mouadbra/multimodal-app/blob/main/frontend_service/src/App.tsx"
click node_transcribe "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_generate "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_analyze "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_speech "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_evaluate "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_iteration "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_objective "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_describe "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_metrics "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_batchlist "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_batchdetail "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_sqlite "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py"
click node_modalvolume "https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/common.py"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_app,node_user toneBlue
class node_transcribe,node_generate,node_analyze,node_speech,node_modalvolume toneAmber
class node_evaluate,node_iteration,node_objective,node_describe,node_metrics,node_batchlist,node_batchdetail toneMint
class node_sqlite toneRose
class node_audio,node_whisper,node_flux,node_clip,node_vision,node_tts toneIndigo
```

## Components

### Frontend

- [Pipeline & Dashboard UI](https://github.com/mouadbra/multimodal-app/blob/main/frontend_service/src/App.tsx): `frontend_service/src/App.tsx`

### Demo Pipeline

- [Whisper Transcription](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Image Generation](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Image Analysis](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Speech Synthesis](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`

### Evaluation

- [Batch Endpoint](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Iteration Evaluation](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Objective Evaluation](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Image Feedback](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Metric Aggregation](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Batch Listing](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`
- [Batch Results](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`

### Persistence

- [Evaluation SQLite](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/main.py): `backend_service/src/modal_app/main.py`

### Other components

- **User**
- **Microphone Audio**
- **Azure Whisper**
- **Azure FLUX**
- **CLIP Similarity**
- **GPT-4o Vision**
- **Azure TTS**
- [Modal Volume](https://github.com/mouadbra/multimodal-app/blob/main/backend_service/src/modal_app/common.py): `backend_service/src/modal_app/common.py`

## Connections

- User → Pipeline & Dashboard UI: uses
- Pipeline & Dashboard UI → Microphone Audio: records
- Microphone Audio → Pipeline & Dashboard UI: provides audio
- Pipeline & Dashboard UI → Whisper Transcription: POST audio
- Whisper Transcription → Azure Whisper: transcribes
- Whisper Transcription → Pipeline & Dashboard UI: returns transcript
- Pipeline & Dashboard UI → Image Generation: requests image
- Image Generation → Azure FLUX: generates image
- Image Generation → Pipeline & Dashboard UI: returns image
- Pipeline & Dashboard UI → Image Analysis: submits image
- Image Analysis → CLIP Similarity: scores similarity
- Image Analysis → GPT-4o Vision: describes image
- Image Analysis → Pipeline & Dashboard UI: returns analysis
- Pipeline & Dashboard UI → Speech Synthesis: submits description
- Speech Synthesis → Azure TTS: synthesizes speech
- Speech Synthesis → Pipeline & Dashboard UI: returns audio
- Pipeline & Dashboard UI → Batch Endpoint: launches batch
- Batch Endpoint → Iteration Evaluation: runs iterations
- Iteration Evaluation → Image Generation: generates image
- Iteration Evaluation → Image Analysis: scores image
- Iteration Evaluation → Objective Evaluation: evaluates criteria
- Iteration Evaluation → Image Feedback: gets feedback
- Batch Endpoint → Evaluation SQLite: writes results
- Batch Endpoint → Metric Aggregation: aggregates scores
- Metric Aggregation → Evaluation SQLite: reads results
- Pipeline & Dashboard UI → Batch Listing: requests batches
- Batch Listing → Evaluation SQLite: reads batches
- Pipeline & Dashboard UI → Batch Results: requests batch
- Batch Results → Evaluation SQLite: reads results
- Evaluation SQLite → Modal Volume: persists on
- Batch Endpoint → Pipeline & Dashboard UI: returns evaluation
- Batch Listing → Pipeline & Dashboard UI: returns batches
- Batch Results → Pipeline & Dashboard UI: returns dashboard data