# abi/screenshot-to-code architecture

> Architecture diagram of the GitHub repository abi/screenshot-to-code, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-09-17). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/abi/screenshot-to-code
- Repository: https://github.com/abi/screenshot-to-code

## Overview

#### Purpose and Entry Points
Screenshot-to-Code is a local/self-hostable AI application that converts screenshots, mockups, Figma-like inputs, URLs, imported code, and recordings into editable web implementations across HTML, React, Vue, Bootstrap, and Ionic stacks. The React/Vite client starts at `frontend/src/main.tsx` and composes the application in `frontend/src/App.tsx`. The FastAPI service is exposed from `backend/main.py`; container deployment is coordinated by `docker-compose.yml`.

#### Client Workspace and Interaction Model
The frontend is a browser-based generation workspace: users submit inputs, select output/model settings, observe generation activity, and inspect or edit generated code and preview output. The main input surface is `frontend/src/components/unified-input/UnifiedInputPane.tsx`; generation orchestration is initiated through `frontend/src/generateCode.ts`. Project state, generated variants, history, and UI persistence are centralized in `frontend/src/store/project-store.ts` and `frontend/src/store/app-store.ts`.

#### API and Realtime Boundary
FastAPI route modules provide the backend’s HTTP/WebSocket-facing contract. Code generation is served by `backend/routes/generate_code.py`; browser capabilities and screenshot rendering are surfaced through `backend/routes/capabilities.py` and `backend/routes/screenshot.py`. The frontend connects using configured HTTP and WebSocket backend URLs. WebSocket status/event constants live in `backend/ws/constants.py`, supporting streamed progress for long-running model and agent operations rather than a request-only workflow.

#### Generation Pipeline and Prompt Construction
Request parsing, stack-specific instructions, policies, planning, and multimodal prompt assembly are separated under `backend/prompts/`. `backend/prompts/pipeline.py` is the central prompt pipeline, with input normalization in `backend/prompts/request_parsing.py` and model message construction in `backend/prompts/message_builder.py`. Create and update flows have distinct prompts under `backend/prompts/create/` and `backend/prompts/update/`, allowing iterative revisions from prompt history or current file snapshots.

#### Agentic Code Production
The agent subsystem executes iterative model/tool loops rather than only single-pass completion. `backend/agent/runner.py` drives runs, `backend/agent/engine.py` coordinates model decisions and state transitions, and `backend/agent/state.py` holds run state. Tools in `backend/agent/tools/` let the agent inspect assets, parse outputs, produce summaries, and invoke runtime functionality. This is the primary control boundary between untrusted/provider-generated instructions and controlled local capabilities.

#### Model, Asset, and Media Integrations
A provider abstraction selects available OpenAI, Anthropic, or Gemini models: `backend/agent/providers/base.py`, `backend/agent/providers/factory.py`, and provider implementations such as `backend/agent/providers/openai.py`. Gemini-backed asset extraction is implemented through `backend/asset_extraction.py`; uploaded assets are stored and made available to prompts/tools via `backend/uploaded_assets/store.py`. Replicate-backed image generation/editing is isolated in `backend/image_generation/replicate.py`. Video support is represented by `backend/video/`.

#### Visual Validation, Evaluation, and Operations
Generated pages can be rendered in headless Chromium for agent visual feedback through the preview abstraction in `backend/preview_screenshot/registry.py` and Playwright implementation `backend/preview_screenshot/playwright_backend.py`. Evaluation datasets, execution, and persisted sessions are organized in `backend/evals/runner.py`, `backend/evals/sets.py`, and `backend/evals/sessions.py`, with corresponding operator pages in `frontend/src/components/evals/`. Token usage and pricing are tracked separately under `backend/costs/`, while filesystem logging under `backend/fs_logging/` supports run inspection and prompt reporting.

Main entry points: [`backend/prompts/pipeline.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/pipeline.py) (Prompt pipeline), [`backend/agent/runner.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/runner.py) (Agent runner), [`frontend/src/App.tsx`](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/App.tsx) (Application shell), [`backend/agent/engine.py`](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/engine.py) (Agent engine & state)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_frontend["Browser Workspace"]
  node_client_entry["React/Vite entry<br/>client entry<br/>[main.tsx]"]
  node_app_shell["Application shell<br/>React composition<br/>[App.tsx]"]
  node_input_pane["Unified input pane<br/>input workspace"]
  node_generation_client["Generation client<br/>frontend orchestration<br/>[generateCode.ts]"]
  node_project_store["Project state<br/>client state<br/>[project-store.ts]"]
  node_app_store["App state<br/>client state<br/>[app-store.ts]"]
end

subgraph group_api["FastAPI Service"]
  node_api_entry["FastAPI application<br/>API entry<br/>[main.py]"]
  node_generation_route["Generation API<br/>HTTP/WebSocket route<br/>[generate_code.py]"]
  node_browser_routes["Browser &amp; screenshot APIs<br/>route modules<br/>[capabilities.py]"]
  node_ws_protocol["WebSocket events<br/>realtime protocol<br/>[constants.py]"]
end

subgraph group_generation["Generation Engine"]
  node_prompt_pipeline["Prompt pipeline<br/>[pipeline.py]"]
  node_request_parsing["Request normalization<br/>prompt input layer<br/>[request_parsing.py]"]
  node_message_builder["Model message builder<br/>multimodal prompt layer<br/>[message_builder.py]"]
  node_agent_runner["Agent runner<br/>iterative run control<br/>[runner.py]"]
  node_agent_engine["Agent engine &amp; state<br/>agent coordination<br/>[engine.py]"]
  node_agent_tools{{"Controlled agent tools<br/>tool boundary"}}
end

subgraph group_integrations["Models &amp; Media"]
  node_provider_factory["Model provider selection<br/>provider abstraction<br/>[factory.py]"]
  node_asset_store[("Uploaded asset store<br/>asset storage<br/>[store.py]")]
  node_gemini_extraction["Gemini asset extraction<br/>asset integration"]
  node_replicate_images["Replicate image generation<br/>media integration<br/>[replicate.py]"]
end

subgraph group_operations["Validation &amp; Operations"]
  node_preview_renderer["Playwright visual preview<br/>headless rendering"]
  node_evaluations["Evaluation runner<br/>evaluation execution<br/>[runner.py]"]
  node_cost_tracking["Cost tracking<br/>usage accounting"]
  node_run_logging["Filesystem run logging<br/>operator inspection"]
end

node_deployment["Container deployment<br/>Docker Compose<br/>[docker-compose.yml]"]

node_client_entry -->|"mounts"| node_app_shell
node_app_shell -->|"hosts"| node_input_pane
node_input_pane -->|"submits inputs"| node_generation_client
node_app_shell -->|"reads and updates"| node_project_store
node_app_shell -->|"reads and updates"| node_app_store
node_generation_client -->|"HTTP/WebSocket"| node_generation_route
node_api_entry -->|"registers"| node_generation_route
node_api_entry -->|"registers"| node_browser_routes
node_generation_route -->|"streams events"| node_ws_protocol
node_generation_route -->|"builds request prompt"| node_prompt_pipeline
node_prompt_pipeline -->|"normalizes input"| node_request_parsing
node_prompt_pipeline -->|"assembles messages"| node_message_builder
node_prompt_pipeline -->|"starts run"| node_agent_runner
node_agent_runner -->|"drives"| node_agent_engine
node_agent_engine -->|"uses controlled tools"| node_agent_tools
node_agent_engine -->|"requests model"| node_provider_factory
node_agent_tools -->|"reads assets"| node_asset_store
node_prompt_pipeline -->|"includes assets"| node_asset_store
node_asset_store -->|"extracts asset context"| node_gemini_extraction
node_agent_tools -->|"generates or edits images"| node_replicate_images
node_agent_engine -->|"requests visual feedback"| node_preview_renderer
node_agent_runner -.->|"records usage"| node_cost_tracking
node_agent_runner -.->|"records run artifacts"| node_run_logging
node_evaluations -->|"executes generation runs"| node_agent_runner
node_deployment -.->|"deploys"| node_api_entry
node_deployment -.->|"deploys"| node_client_entry

click node_client_entry "https://github.com/abi/screenshot-to-code/blob/main/frontend/src/main.tsx"
click node_app_shell "https://github.com/abi/screenshot-to-code/blob/main/frontend/src/App.tsx"
click node_input_pane "https://github.com/abi/screenshot-to-code/blob/main/frontend/src/components/unified-input/UnifiedInputPane.tsx"
click node_generation_client "https://github.com/abi/screenshot-to-code/blob/main/frontend/src/generateCode.ts"
click node_project_store "https://github.com/abi/screenshot-to-code/blob/main/frontend/src/store/project-store.ts"
click node_app_store "https://github.com/abi/screenshot-to-code/blob/main/frontend/src/store/app-store.ts"
click node_api_entry "https://github.com/abi/screenshot-to-code/blob/main/backend/main.py"
click node_generation_route "https://github.com/abi/screenshot-to-code/blob/main/backend/routes/generate_code.py"
click node_browser_routes "https://github.com/abi/screenshot-to-code/blob/main/backend/routes/capabilities.py"
click node_ws_protocol "https://github.com/abi/screenshot-to-code/blob/main/backend/ws/constants.py"
click node_prompt_pipeline "https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/pipeline.py"
click node_request_parsing "https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/request_parsing.py"
click node_message_builder "https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/message_builder.py"
click node_agent_runner "https://github.com/abi/screenshot-to-code/blob/main/backend/agent/runner.py"
click node_agent_engine "https://github.com/abi/screenshot-to-code/blob/main/backend/agent/engine.py"
click node_provider_factory "https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/factory.py"
click node_asset_store "https://github.com/abi/screenshot-to-code/blob/main/backend/uploaded_assets/store.py"
click node_gemini_extraction "https://github.com/abi/screenshot-to-code/blob/main/backend/asset_extraction.py"
click node_replicate_images "https://github.com/abi/screenshot-to-code/blob/main/backend/image_generation/replicate.py"
click node_preview_renderer "https://github.com/abi/screenshot-to-code/blob/main/backend/preview_screenshot/playwright_backend.py"
click node_evaluations "https://github.com/abi/screenshot-to-code/blob/main/backend/evals/runner.py"
click node_deployment "https://github.com/abi/screenshot-to-code/blob/main/docker-compose.yml"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_client_entry,node_app_shell,node_input_pane,node_generation_client,node_project_store,node_app_store toneBlue
class node_api_entry,node_generation_route,node_browser_routes,node_ws_protocol toneAmber
class node_prompt_pipeline,node_request_parsing,node_message_builder,node_agent_runner,node_agent_engine,node_agent_tools toneMint
class node_provider_factory,node_asset_store,node_gemini_extraction,node_replicate_images toneRose
class node_preview_renderer,node_evaluations,node_cost_tracking,node_run_logging toneIndigo
class node_deployment toneNeutral
```

## Components

### Browser Workspace

- [React/Vite entry](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/main.tsx): client entry · `frontend/src/main.tsx`
- [Application shell](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/App.tsx): React composition · `frontend/src/App.tsx`
- [Unified input pane](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/components/unified-input/UnifiedInputPane.tsx): input workspace · Accepts screenshots, URLs, imports, and recordings. · `frontend/src/components/unified-input/UnifiedInputPane.tsx`
- [Generation client](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/generateCode.ts): frontend orchestration · `frontend/src/generateCode.ts`
- [Project state](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/store/project-store.ts): client state · Stores variants, history, generated code, and UI persistence. · `frontend/src/store/project-store.ts`
- [App state](https://github.com/abi/screenshot-to-code/blob/main/frontend/src/store/app-store.ts): client state · `frontend/src/store/app-store.ts`

### FastAPI Service

- [FastAPI application](https://github.com/abi/screenshot-to-code/blob/main/backend/main.py): API entry · `backend/main.py`
- [Generation API](https://github.com/abi/screenshot-to-code/blob/main/backend/routes/generate_code.py): HTTP/WebSocket route · Starts generation and streams long-running progress. · `backend/routes/generate_code.py`
- [Browser & screenshot APIs](https://github.com/abi/screenshot-to-code/blob/main/backend/routes/capabilities.py): route modules · `backend/routes/capabilities.py`
- [WebSocket events](https://github.com/abi/screenshot-to-code/blob/main/backend/ws/constants.py): realtime protocol · `backend/ws/constants.py`

### Generation Engine

- [Prompt pipeline](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/pipeline.py): request-to-prompt pipeline · Applies parsing, stack instructions, policies, and create/update prompts. · `backend/prompts/pipeline.py`
- [Request normalization](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/request_parsing.py): prompt input layer · `backend/prompts/request_parsing.py`
- [Model message builder](https://github.com/abi/screenshot-to-code/blob/main/backend/prompts/message_builder.py): multimodal prompt layer · `backend/prompts/message_builder.py`
- [Agent runner](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/runner.py): iterative run control · Drives model and tool loops. · `backend/agent/runner.py`
- [Agent engine & state](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/engine.py): agent coordination · Controls model decisions and run-state transitions. · `backend/agent/engine.py`
- **Controlled agent tools**: tool boundary · Inspects assets, parses output, summarizes, and invokes runtime capabilities.

### Models & Media

- [Model provider selection](https://github.com/abi/screenshot-to-code/blob/main/backend/agent/providers/factory.py): provider abstraction · Selects OpenAI, Anthropic, or Gemini implementations. · `backend/agent/providers/factory.py`
- [Uploaded asset store](https://github.com/abi/screenshot-to-code/blob/main/backend/uploaded_assets/store.py): asset storage · Makes uploaded assets available to prompts and tools. · `backend/uploaded_assets/store.py`
- [Gemini asset extraction](https://github.com/abi/screenshot-to-code/blob/main/backend/asset_extraction.py): asset integration · `backend/asset_extraction.py`
- [Replicate image generation](https://github.com/abi/screenshot-to-code/blob/main/backend/image_generation/replicate.py): media integration · `backend/image_generation/replicate.py`

### Validation & Operations

- [Playwright visual preview](https://github.com/abi/screenshot-to-code/blob/main/backend/preview_screenshot/playwright_backend.py): headless rendering · Renders generated pages for agent visual feedback. · `backend/preview_screenshot/playwright_backend.py`
- [Evaluation runner](https://github.com/abi/screenshot-to-code/blob/main/backend/evals/runner.py): evaluation execution · Runs datasets and persists evaluation sessions. · `backend/evals/runner.py`
- **Cost tracking**: usage accounting
- **Filesystem run logging**: operator inspection

### Other components

- [Container deployment](https://github.com/abi/screenshot-to-code/blob/main/docker-compose.yml): Docker Compose · `docker-compose.yml`

## Connections

- React/Vite entry → Application shell: mounts
- Application shell → Unified input pane: hosts
- Unified input pane → Generation client: submits inputs
- Application shell → Project state: reads and updates
- Application shell → App state: reads and updates
- Generation client → Generation API: HTTP/WebSocket
- FastAPI application → Generation API: registers
- FastAPI application → Browser & screenshot APIs: registers
- Generation API → WebSocket events: streams events
- Generation API → Prompt pipeline: builds request prompt
- Prompt pipeline → Request normalization: normalizes input
- Prompt pipeline → Model message builder: assembles messages
- Prompt pipeline → Agent runner: starts run
- Agent runner → Agent engine & state: drives
- Agent engine & state → Controlled agent tools: uses controlled tools
- Agent engine & state → Model provider selection: requests model
- Controlled agent tools → Uploaded asset store: reads assets
- Prompt pipeline → Uploaded asset store: includes assets
- Uploaded asset store → Gemini asset extraction: extracts asset context
- Controlled agent tools → Replicate image generation: generates or edits images
- Agent engine & state → Playwright visual preview: requests visual feedback
- Agent runner → Cost tracking: records usage
- Agent runner → Filesystem run logging: records run artifacts
- Evaluation runner → Agent runner: executes generation runs
- Container deployment → FastAPI application: deploys
- Container deployment → React/Vite entry: deploys
