How sirantonysir/editor works
1. Overall shape
- This is a React + TypeScript photo editor with a Python backend for AI-assisted features.
- The core editing experience runs in the frontend; the backend is a separate service for analysis, refinement, segmentation, session APIs, and MCP exposure.
- There is also an Electron main/preload layer, so the same app can be packaged as a desktop shell.
2. Frontend presentation layer
- The UI is organized around a canvas workspace and a right-side inspector.
- Main UI areas include: canvas/workspace, layers/panels, toolbar, command palette, inspector tabs, and widget editors.
- The workspace renders image nodes, widget nodes, tether/overlay visuals, crop overlays, and segment/object indicators.
3. Editor state and boundaries
- App state is split into Zustand slices: document, layers, selection, tool, viewport, workspace, backend status, segmentation, and suggestions UI.
- Undo/redo is handled as part of the local editor state model, not by mutating pixels directly.
- The key boundary is between ephemeral UI state in Zustand and pixel data / session artifacts stored elsewhere.
4. Non-destructive image model
- The editor is built around non-destructive editing: adjustments are metadata, not baked into pixels.
- Crop, layer transforms, masks, and widget parameters are all modeled as state attached to documents/layers/widgets.
- Original or pre-crop image data is preserved so edits can be revisited and re-rendered.
5. Render and processing pipeline
- Image rendering uses Fabric.js for canvas interaction plus a custom WebGL shader pipeline for filters/adjustments.
- The rendering path is layered: each image/layer gets its own adjustment pipeline, then results are composited into the final output.
- CPU-heavy work is offloaded through workers and pipeline helpers rather than being done directly in React render flow.
6. Tool system
- Tools are modeled as self-contained definitions and discovered through a registry.
- There are atomic tools for selection, masks, image context, widget ops, and analysis, plus fused tools that bundle multiple adjustment operations into higher-level editing actions.
- This is an important architectural boundary: new editor capabilities are added by registering tool modules rather than wiring special cases throughout the app.
7. Registry-driven adjustment library
- The project has a shared registry of ops and presets under shared/registry, consumed by both frontend and backend.
- JSON schema files and generated shared types define the contract for operation graphs, widgets, contexts, and session snapshots.
- The UI, tool manifests, and backend registry loader all depend on this common schema layer, which keeps the editor’s controls and AI tool surface consistent.
8. AI-assisted editing loop
- The frontend has hooks for backend session status, image context, and auto-suggested actions.
- The backend uses an Anthropic client and services for context assembly, autonomous suggestions, and refinement workflows.
- Data flow is: image/document state → context extraction/analysis → backend suggestion/refinement → frontend UI updates and tool actions.
9. Segmentation and object mode
- Objects-mode is powered by SAM/MobileSAM on the frontend and supported by backend segmentation endpoints.
- The app manages masks, named regions, object extraction, region precomputation, and object ownership as first-class state.
- This subsystem feeds both the workspace overlay and the inspector panels, and it also supports AI tools that need region-aware context.
10. Session persistence and lifecycle
- Sessions are handled through a dedicated session layer with history, checkpointers, persistence, revive, and migrations.
- There are both frontend persistence helpers and backend session APIs, suggesting a split between local editor continuity and server-backed AI/session continuity.
- The architecture supports restoring editor state, undo/redo cycles, and migration from older snapshot formats.
11. Backend service surface
- The Python backend exposes API modules for analyze, refine, segment, session, state, telemetry, admin, panel, and tools REST.
- It also contains an MCP server and rate limiting, indicating it can act as a tool provider for external AI/agent workflows.
- In diagram terms, the backend is not the rendering engine; it is the analysis and orchestration boundary around the editor.
12. Shared contracts and validation
- shared/schemas and shared/types define the contract between frontend, backend, and generated tool manifests.
- Tests are extensive across registry, session, state, tools, and contract boundaries, which reinforces that the architecture depends heavily on stable cross-layer schemas.
- For an architecture diagram, the most important arrows are between: document/layer state → render pipeline, image/context state → backend analysis, and registry schemas → UI/tool generation.
Main entry points: main.py API surface · index.ts Editor state · main.tsx App shell · document.ts Doc model
Components 22
Frontend app
React + TypeScript editor UI, state, rendering, and local processing.
- App shell · Bootstraps the editor and top-level providers.
src/main.tsx - Workspace · Canvas-centric editing surface for image nodes, widgets, overlays, and tethers.
src/components/workspace/CanvasWorkspace.tsx - Inspector · Right-side panel for layers, crop, info, adjustments, and widget editing.
src/components/inspector/InspectorPanel.tsx - Toolbar · History, menu actions, shortcuts, and command palette entry points.
src/components/toolbar/MenuBar.tsx - Editor state · Document, layers, selection, tool, viewport, workspace, backend, segmentation, and suggestions state.
src/store/index.ts - Doc model · Document, layer, widget, crop, mask, and transform metadata used for undoable editing.
src/core/document.ts - History & save · Undo/redo, persistence, restore, and pixel source/session lifecycle helpers.
src/core/editor-state-persistence.ts - Render pipeline · Compositing and per-layer rendering path combining Fabric.js interaction with shader-based processing.
src/lib/image-node-renderer.ts - Shaders · GPU adjustment and blend shaders for image processing.
src/shaders/pipeline.ts - Workers · Offloads CPU-heavy processing and pipeline work from the React flow.
src/workers/processing.worker.ts - Tool system · Self-contained tools and toolrail behavior for selection, masks, analysis, and fused actions.
src/engine/registry.ts - Registry UI · Inspector controls, widget shells, and tool manifests generated from shared registry definitions.
src/lib/tool-manifest/index.ts - Segmentation · SAM/MobileSAM-backed selection, masks, regions, and object ownership flows.
src/lib/segmentation/mobile-sam-client.ts - AI hooks · Frontend hooks for session status, image context, and auto-suggested actions.
src/hooks/useAutoTetherAiSuggestions.ts
Backend service
Python AI and session orchestration service.
- API surface · Analyze, refine, segment, session, state, telemetry, admin, panel, and tools REST endpoints.
backend/app/main.py - AI services · Anthropic client, LLM context assembly, autonomous suggestions, image validation, and SAM client.
backend/app/services/anthropic_client.py - Session engine · History, checkpoints, persistence, revive, disk I/O, and migrations for session continuity.
backend/app/session/persistence.py - Registry backend · Loads, interpolates, and resolves registry ops, presets, and schema contracts.
backend/app/registry/loader.py - Backend state · Canonical document state, snapshots, events, and preview rendering for server-side workflows.
backend/app/state/document.py
Shared contracts
Registry, schemas, and generated types shared across runtimes.
- Shared registry · Common adjustment ops and presets consumed by both frontend and backend.
shared/registry/schema.ts - Shared schemas · JSON schemas and generated types for contexts, widgets, operation graphs, and session snapshots.
shared/schemas/combined.schema.json
Desktop shell
Electron main and preload layer.
- Electron shell · Desktop main and preload process for packaging the app as a shell.
electron/main.cjs
Connections 26
- App shell → Workspace: mounts · Renders the main editing canvas.
- App shell → Inspector: mounts · Renders the side inspector.
- App shell → Toolbar: mounts · Provides top-level actions and shortcuts.
- Workspace → Editor state: reads state · Consumes document, selection, tool, viewport, and workspace slices.
- Inspector → Editor state: reads state · Consumes layer, segmentation, suggestions, and backend status slices.
- Toolbar → Editor state: dispatches · Triggers history, palette, and tool actions.
- Editor state → Doc model: hosts · Keeps the non-destructive editor model in local state.
- Doc model → History & save: snapshots · Document changes feed undo/redo and persistence.
- Doc model → Render pipeline: renders · Layer, crop, widget, and mask state drives compositing.
- Render pipeline → Shaders: uses · Applies GPU adjustment and blending stages.
- Render pipeline → Workers: offloads · Sends heavy processing away from React render flow.
- Tool system → Editor state: updates · Tools mutate editor state through controlled actions.
- Registry UI → Shared schemas: binds to · Builds controls from shared schemas and types.
- Registry UI → Shared registry: reads · Uses the common adjustment/preset registry.
- Tool system → Shared registry: discovers tools · Tool manifests and fused actions are registry-driven.
- Segmentation → Editor state: writes masks · Updates object mode, regions, and selection state.
- AI hooks → API surface: calls · Requests session status, analysis, refine, and suggestions.
- API surface → AI services: orchestrates · Routes requests into AI and analysis services.
- API surface → Session engine: persists · Creates, restores, checkpoints, and revives sessions.
- API surface → Backend state: serves state · Exposes canonical document and preview data.
- AI services → Registry backend: resolves ops · Uses registry data to assemble analysis and refinement workflows.
- Registry backend → Shared registry: loads from · Shares the op and preset definitions with the frontend.
- Backend state → Shared schemas: validates against · Uses shared schemas for snapshots and operation graphs.
- API surface → Shared schemas: contracts · API payloads are shaped by the shared schema layer.
- Electron shell → App shell: hosts · Wraps the frontend as a desktop app.
- Electron shell → API surface: bridges to · Desktop packaging can talk to the backend service.