# frankstew1/mrsurvey architecture

> Architecture diagram of the GitHub repository frankstew1/mrsurvey, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-06-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/frankstew1/mrsurvey
- Repository: https://github.com/frankstew1/mrsurvey

## Overview

1. **Overall shape**
   SurveyBot Enterprise is a two-plane system: a synchronous **API/control plane** and an asynchronous **worker/execution plane**. The API handles session creation, auth, and real-time interaction, while workers perform heavier browser, vision, and analysis jobs.

2. **Primary runtime entrypoints**
   The repository is centered on a **FastAPI application** for the API service and a separate **worker image** for background processing. `docker-compose.yml`, `Dockerfile`, and `Dockerfile.worker` show that these are deployed as distinct containers, not a single monolith.

3. **Orchestration core**
   The core runtime is built around **SessionManager**, **RuntimeController**, **EventBus**, and **RuntimeState**. This layer coordinates survey runs, emits domain events, manages session lifecycle, and wraps recovery/retry behavior. It is the main hub between API requests and lower-level automation components.

4. **Browser automation boundary**
   The browser layer manages **Playwright-based** browser/context/page lifecycles. It sits beneath orchestration and above page-specific logic, so it is the component that actually interacts with live survey pages. Retry, checkpoint/restore, and recovery strategies live close to this layer because they need to react to browser failures.

5. **Survey platform abstraction**
   A dedicated **adapter subsystem** normalizes different survey platforms: Google Forms, Typeform, Qualtrics, SurveyMonkey, Jotform, Microsoft Forms, LimeSurvey, and generic HTML forms. `AdapterRegistry` handles auto-detection, so the rest of the system can operate against a common interface instead of platform-specific code.

6. **Page understanding and self-healing**
   The DOM subsystem provides **DOM graph building, selector generation/healing, mutation tracking, and accessibility parsing**. This is the architectural bridge between raw browser pages and reliable automation, especially when survey UIs change dynamically or selectors become brittle.

7. **AI and reasoning layer**
   The AI subsystem adds **semantic question interpretation, persona-based response generation, sentiment/intent classification, and multi-provider LLM support**. This layer influences what answers get generated, but it is logically separate from the browser so reasoning can evolve independently of automation mechanics.

8. **Vision and multimodal analysis**
   The vision layer uses **OpenCV-based detection** plus **OCR engines** to interpret screenshots and hard-to-parse controls. A multimodal reasoner combines visual evidence with DOM/semantic context when text-only parsing is insufficient, such as sliders, ratings, or image-heavy pages.

9. **Memory, learning, and persistence**
   Long-term behavior is split across **vector/memory components** and **database persistence**. The project uses **async SQLAlchemy** plus migration support (`alembic.ini`, `migrations/`) for structured state, and a **PGVector-backed store** for embeddings, knowledge, and adaptive learning data. This is where historical survey outcomes feed future strategy.

10. **Distributed execution**
    Background work is offloaded to **Celery workers** with **Redis** as the queue/broker layer. The worker deployment in Kubernetes and the worker-specific Dockerfile indicate that tasks like screenshot analysis, embedding generation, and session analysis are expected to scale independently from the API.

11. **Plugin and event-driven extensibility**
    The architecture is intentionally **event-driven** and **plugin-based**. Hooks exist around session/page/question/submit/error lifecycle points, which makes custom logging, analytics, or transformations additive rather than invasive. The EventBus is the main decoupling mechanism here.

12. **Observability stack**
    Observability is a first-class subsystem: **structlog** for structured logs, **OpenTelemetry** for tracing, and **Prometheus** metrics. The repo also includes ready-made **Grafana, Loki, and Tempo** configs and dashboards, which means runtime visibility is designed into the deployment rather than added later.

13. **Deployment targets**
    The project supports three deployment modes: local **Docker Compose**, minimal **Kubernetes manifests** in `k8s/`, and a full **Helm chart** in `helm/surveybot`. The Helm chart splits API and worker deployments, exposes ingress/service configuration, and includes config/secrets/horizontal scaling resources.

14. **Quality and failure testing**
    The test suite includes normal integration tests plus **chaos tests** for Redis failures, worker crashes, and replay storms. That tells you the system is expected to tolerate partial failures in the queue/worker layer and recover session execution without losing the whole workflow.

15. **What to show in an architecture diagram**
    A good diagram should emphasize: client/API entry → session orchestration → event bus → browser/DOM/adapter pipeline, with side branches into AI, vision, persistence/memory, and Celery/Redis workers; plus an observability layer spanning everything and deployment boundaries around API, worker, Postgres, Redis, and monitoring.

Main entry points: [`Dockerfile`](https://github.com/frankstew1/mrsurvey/tree/main/Dockerfile) (API service), [`Dockerfile.worker`](https://github.com/frankstew1/mrsurvey/blob/main/Dockerfile.worker) (Worker service), [`helm/surveybot`](https://github.com/frankstew1/mrsurvey/tree/main/helm/surveybot) (Helm chart), [`configs`](https://github.com/frankstew1/mrsurvey/tree/main/configs) (Telemetry stack)

## Diagram (Mermaid)

```mermaid
flowchart TD

subgraph group_runtime["Runtime"]
  node_api["API service<br/>fastapi app"]
  node_worker["Worker service<br/>celery worker<br/>[Dockerfile.worker]"]
end

subgraph group_core["Core logic"]
  node_session_mgr["Session manager<br/>orchestration"]
  node_runtime_ctrl["Runtime control<br/>runtime coordinator"]
  node_event_bus["Event bus<br/>domain events"]
  node_browser["Browser layer<br/>playwright automation"]
  node_dom["DOM engine<br/>selector healing"]
  node_adapters["Adapters<br/>survey platforms"]
  node_ai["AI reasoning<br/>llm pipeline"]
  node_vision["Vision layer<br/>ocr / cv"]
end

subgraph group_platform["Platform"]
  node_memory[("Memory store<br/>embeddings store")]
  node_db[("Database<br/>async sqlalchemy")]
  node_queue["Redis queue<br/>broker"]
  node_tests["Chaos tests<br/>resilience tests"]
end

subgraph group_deploy["Deployment"]
  node_compose["Compose setup<br/>local runtime<br/>[docker-compose.yml]"]
  node_k8s["Kubernetes<br/>manifests"]
  node_helm["Helm chart"]
end

subgraph group_ops["Observability"]
  node_observability["Telemetry stack<br/>metrics/logs/traces"]
end

node_api -->|"handles"| node_session_mgr
node_session_mgr -->|"drives"| node_runtime_ctrl
node_runtime_ctrl -->|"emits"| node_event_bus
node_event_bus -->|"triggers"| node_browser
node_browser -->|"feeds"| node_dom
node_browser -->|"routes"| node_adapters
node_dom -->|"normalizes"| node_adapters
node_dom -->|"augments"| node_ai
node_ai -->|"combines"| node_vision
node_vision -->|"reads"| node_browser
node_ai -->|"writes"| node_memory
node_session_mgr -->|"persists"| node_db
node_memory -->|"syncs"| node_db
node_worker -->|"consumes"| node_queue
node_queue -.->|"stressed by"| node_tests
node_observability -.->|"watches"| node_api
node_observability -.->|"watches"| node_worker
node_compose -->|"defines"| node_api
node_compose -->|"defines"| node_worker
node_k8s -->|"deploys"| node_api
node_k8s -->|"deploys"| node_worker
node_helm -->|"packages"| node_api
node_helm -->|"packages"| node_worker
node_helm -->|"configures"| node_observability

click node_api "https://github.com/frankstew1/mrsurvey/tree/main/Dockerfile"
click node_worker "https://github.com/frankstew1/mrsurvey/blob/main/Dockerfile.worker"
click node_db "https://github.com/frankstew1/mrsurvey/tree/main/migrations"
click node_queue "https://github.com/frankstew1/mrsurvey/blob/main/k8s/redis-deployment.yaml"
click node_observability "https://github.com/frankstew1/mrsurvey/tree/main/configs"
click node_compose "https://github.com/frankstew1/mrsurvey/blob/main/docker-compose.yml"
click node_k8s "https://github.com/frankstew1/mrsurvey/tree/main/k8s"
click node_helm "https://github.com/frankstew1/mrsurvey/tree/main/helm/surveybot"
click node_tests "https://github.com/frankstew1/mrsurvey/tree/main/tests/chaos"

classDef toneNeutral fill:#f8fafc,stroke:#334155,stroke-width:1.5px,color:#0f172a
classDef toneBlue fill:#dbeafe,stroke:#2563eb,stroke-width:1.5px,color:#172554
classDef toneAmber fill:#fef3c7,stroke:#d97706,stroke-width:1.5px,color:#78350f
classDef toneMint fill:#dcfce7,stroke:#16a34a,stroke-width:1.5px,color:#14532d
classDef toneRose fill:#ffe4e6,stroke:#e11d48,stroke-width:1.5px,color:#881337
classDef toneIndigo fill:#e0e7ff,stroke:#4f46e5,stroke-width:1.5px,color:#312e81
classDef toneTeal fill:#ccfbf1,stroke:#0f766e,stroke-width:1.5px,color:#134e4a
class node_api,node_worker toneBlue
class node_session_mgr,node_runtime_ctrl,node_event_bus,node_browser,node_dom,node_adapters,node_ai,node_vision toneAmber
class node_memory,node_db,node_queue,node_tests toneMint
class node_compose,node_k8s,node_helm toneRose
class node_observability toneIndigo
```

## Components

### Runtime

API/control plane and background execution plane

- [API service](https://github.com/frankstew1/mrsurvey/tree/main/Dockerfile): fastapi app · Synchronous control plane for sessions, auth, and live interaction · `Dockerfile`
- [Worker service](https://github.com/frankstew1/mrsurvey/blob/main/Dockerfile.worker): celery worker · Asynchronous execution plane for heavy jobs and analysis · `Dockerfile.worker`

### Core logic

Orchestration, browser, adapters, AI, and memory

- **Session manager**: orchestration · Coordinates survey session lifecycle and runtime flow
- **Runtime control**: runtime coordinator · Wraps retries, recovery, and execution state transitions
- **Event bus**: domain events · Decouples lifecycle hooks and plugin reactions
- **Browser layer**: playwright automation · Manages browser, context, and page lifecycles
- **DOM engine**: selector healing · Builds DOM graphs, tracks mutations, and heals selectors
- **Adapters**: survey platforms · Normalizes Google Forms, Typeform, Qualtrics, and other platforms
- **AI reasoning**: llm pipeline · Interprets questions and generates persona-aware responses
- **Vision layer**: ocr / cv · Uses screenshots, OCR, and visual detection for hard UI cases

### Platform

Persistence, queueing, and operational stack

- **Memory store**: embeddings store · Persists adaptive knowledge and historical survey signals
- [Database](https://github.com/frankstew1/mrsurvey/tree/main/migrations): async sqlalchemy · Structured persistence with migrations and runtime state · `migrations`
- [Redis queue](https://github.com/frankstew1/mrsurvey/blob/main/k8s/redis-deployment.yaml): broker · Backs distributed task execution and worker coordination · `k8s/redis-deployment.yaml`
- [Chaos tests](https://github.com/frankstew1/mrsurvey/tree/main/tests/chaos): resilience tests · Exercises Redis failures, worker crashes, and replay storms · `tests/chaos`

### Deployment

Container, Kubernetes, and Helm packaging

- [Compose setup](https://github.com/frankstew1/mrsurvey/blob/main/docker-compose.yml): local runtime · Local multi-container wiring for API, worker, and dependencies · `docker-compose.yml`
- [Kubernetes](https://github.com/frankstew1/mrsurvey/tree/main/k8s): manifests · Base manifests for api, worker, postgres, redis, and monitoring · `k8s`
- [Helm chart](https://github.com/frankstew1/mrsurvey/tree/main/helm/surveybot): package · Parameterized deployment for api, worker, ingress, config, and scaling · `helm/surveybot`

### Observability

Logging, metrics, tracing, and dashboards

- [Telemetry stack](https://github.com/frankstew1/mrsurvey/tree/main/configs): metrics/logs/traces · Structured logging, tracing, metrics, and dashboards · `configs`

## Connections

- API service → Session manager: handles · API requests enter session orchestration
- Session manager → Runtime control: drives · Session flow is coordinated through runtime control
- Runtime control → Event bus: emits · Lifecycle transitions publish domain events
- Event bus → Browser layer: triggers · Events fan out into browser execution
- Browser layer → DOM engine: feeds · Page state is analyzed and healed through DOM logic
- Browser layer → Adapters: routes · Platform-specific handling is selected via adapter registry
- DOM engine → Adapters: normalizes · DOM and selector abstractions support common platform behavior
- DOM engine → AI reasoning: augments · Semantic understanding improves question interpretation
- AI reasoning → Vision layer: combines · Reasoning can incorporate visual evidence when needed
- Vision layer → Browser layer: reads · Screenshots and page captures are taken from browser context
- AI reasoning → Memory store: writes · Learning and adaptive signals are persisted in memory
- Session manager → Database: persists · Structured session state is stored in the database
- Memory store → Database: syncs · Embedding and historical data sit alongside structured persistence
- Worker service → Redis queue: consumes · Background tasks are pulled from Redis-backed queues
- Redis queue → Chaos tests: stressed by · Chaos coverage targets broker and worker failures
- Telemetry stack → API service: watches · Telemetry spans the control plane
- Telemetry stack → Worker service: watches · Telemetry spans the execution plane
- Compose setup → API service: defines · Local compose deploys the API container
- Compose setup → Worker service: defines · Local compose deploys the worker container
- Kubernetes → API service: deploys · Kubernetes manifests include the API deployment
- Kubernetes → Worker service: deploys · Kubernetes manifests include the worker deployment
- Helm chart → API service: packages · Helm chart templates the API release
- Helm chart → Worker service: packages · Helm chart templates the worker release
- Helm chart → Telemetry stack: configures · Chart supports config and operational wiring