How abi/screenshot-to-code works
Purpose and Entry Points
Screenshot-to-Code is a local/self-hostable AI application that converts screenshots, mockups, Figma-like inputs, URLs, imported code, and recordings into editable web implementations across HTML, React, Vue, Bootstrap, and Ionic stacks. The React/Vite client starts at frontend/src/main.tsx and composes the application in frontend/src/App.tsx. The FastAPI service is exposed from backend/main.py; container deployment is coordinated by docker-compose.yml.
Client Workspace and Interaction Model
The frontend is a browser-based generation workspace: users submit inputs, select output/model settings, observe generation activity, and inspect or edit generated code and preview output. The main input surface is frontend/src/components/unified-input/UnifiedInputPane.tsx; generation orchestration is initiated through frontend/src/generateCode.ts. Project state, generated variants, history, and UI persistence are centralized in frontend/src/store/project-store.ts and frontend/src/store/app-store.ts.
API and Realtime Boundary
FastAPI route modules provide the backend’s HTTP/WebSocket-facing contract. Code generation is served by backend/routes/generate_code.py; browser capabilities and screenshot rendering are surfaced through backend/routes/capabilities.py and backend/routes/screenshot.py. The frontend connects using configured HTTP and WebSocket backend URLs. WebSocket status/event constants live in backend/ws/constants.py, supporting streamed progress for long-running model and agent operations rather than a request-only workflow.
Generation Pipeline and Prompt Construction
Request parsing, stack-specific instructions, policies, planning, and multimodal prompt assembly are separated under backend/prompts/. backend/prompts/pipeline.py is the central prompt pipeline, with input normalization in backend/prompts/request_parsing.py and model message construction in backend/prompts/message_builder.py. Create and update flows have distinct prompts under backend/prompts/create/ and backend/prompts/update/, allowing iterative revisions from prompt history or current file snapshots.
Agentic Code Production
The agent subsystem executes iterative model/tool loops rather than only single-pass completion. backend/agent/runner.py drives runs, backend/agent/engine.py coordinates model decisions and state transitions, and backend/agent/state.py holds run state. Tools in backend/agent/tools/ let the agent inspect assets, parse outputs, produce summaries, and invoke runtime functionality. This is the primary control boundary between untrusted/provider-generated instructions and controlled local capabilities.
Model, Asset, and Media Integrations
A provider abstraction selects available OpenAI, Anthropic, or Gemini models: backend/agent/providers/base.py, backend/agent/providers/factory.py, and provider implementations such as backend/agent/providers/openai.py. Gemini-backed asset extraction is implemented through backend/asset_extraction.py; uploaded assets are stored and made available to prompts/tools via backend/uploaded_assets/store.py. Replicate-backed image generation/editing is isolated in backend/image_generation/replicate.py. Video support is represented by backend/video/.
Visual Validation, Evaluation, and Operations
Generated pages can be rendered in headless Chromium for agent visual feedback through the preview abstraction in backend/preview_screenshot/registry.py and Playwright implementation backend/preview_screenshot/playwright_backend.py. Evaluation datasets, execution, and persisted sessions are organized in backend/evals/runner.py, backend/evals/sets.py, and backend/evals/sessions.py, with corresponding operator pages in frontend/src/components/evals/. Token usage and pricing are tracked separately under backend/costs/, while filesystem logging under backend/fs_logging/ supports run inspection and prompt reporting.
Main entry points: pipeline.py Prompt pipeline · runner.py Agent runner · App.tsx Application shell · engine.py Agent engine & state
Components 25
Browser Workspace
- React/Vite entry
frontend/src/main.tsx - Application shell
frontend/src/App.tsx - Unified input pane · Accepts screenshots, URLs, imports, and recordings.
frontend/src/components/unified-input/UnifiedInputPane.tsx - Generation client
frontend/src/generateCode.ts - Project state · Stores variants, history, generated code, and UI persistence.
frontend/src/store/project-store.ts - App state
frontend/src/store/app-store.ts
FastAPI Service
- FastAPI application
backend/main.py - Generation API · Starts generation and streams long-running progress.
backend/routes/generate_code.py - Browser & screenshot APIs
backend/routes/capabilities.py - WebSocket events
backend/ws/constants.py
Generation Engine
- Prompt pipeline · Applies parsing, stack instructions, policies, and create/update prompts.
backend/prompts/pipeline.py - Request normalization
backend/prompts/request_parsing.py - Model message builder
backend/prompts/message_builder.py - Agent runner · Drives model and tool loops.
backend/agent/runner.py - Agent engine & state · Controls model decisions and run-state transitions.
backend/agent/engine.py - Controlled agent tools · Inspects assets, parses output, summarizes, and invokes runtime capabilities.
Models & Media
- Model provider selection · Selects OpenAI, Anthropic, or Gemini implementations.
backend/agent/providers/factory.py - Uploaded asset store · Makes uploaded assets available to prompts and tools.
backend/uploaded_assets/store.py - Gemini asset extraction
backend/asset_extraction.py - Replicate image generation
backend/image_generation/replicate.py
Validation & Operations
- Playwright visual preview · Renders generated pages for agent visual feedback.
backend/preview_screenshot/playwright_backend.py - Evaluation runner · Runs datasets and persists evaluation sessions.
backend/evals/runner.py - Cost tracking
- Filesystem run logging
Other components
- Container deployment
docker-compose.yml
Connections 26
- React/Vite entry → Application shell: mounts
- Application shell → Unified input pane: hosts
- Unified input pane → Generation client: submits inputs
- Application shell → Project state: reads and updates
- Application shell → App state: reads and updates
- Generation client → Generation API: HTTP/WebSocket
- FastAPI application → Generation API: registers
- FastAPI application → Browser & screenshot APIs: registers
- Generation API → WebSocket events: streams events
- Generation API → Prompt pipeline: builds request prompt
- Prompt pipeline → Request normalization: normalizes input
- Prompt pipeline → Model message builder: assembles messages
- Prompt pipeline → Agent runner: starts run
- Agent runner → Agent engine & state: drives
- Agent engine & state → Controlled agent tools: uses controlled tools
- Agent engine & state → Model provider selection: requests model
- Controlled agent tools → Uploaded asset store: reads assets
- Prompt pipeline → Uploaded asset store: includes assets
- Uploaded asset store → Gemini asset extraction: extracts asset context
- Controlled agent tools → Replicate image generation: generates or edits images
- Agent engine & state → Playwright visual preview: requests visual feedback
- Agent runner → Cost tracking: records usage
- Agent runner → Filesystem run logging: records run artifacts
- Evaluation runner → Agent runner: executes generation runs
- Container deployment → FastAPI application: deploys
- Container deployment → React/Vite entry: deploys