frankstew1/mrsurvey

1. Overall shape

SurveyBot Enterprise is a two-plane system: a synchronous API/control plane and an asynchronous worker/execution plane. The API handles session creation, auth, and real-time interaction, while workers perform heavier browser, vision, and analysis jobs.

2. Primary runtime entrypoints

The repository is centered on a FastAPI application for the API service and a separate worker image for background processing. docker-compose.yml, Dockerfile, and Dockerfile.worker show that these are deployed as distinct containers, not a single monolith.

3. Orchestration core

The core runtime is built around SessionManager, RuntimeController, EventBus, and RuntimeState. This layer coordinates survey runs, emits domain events, manages session lifecycle, and wraps recovery/retry behavior. It is the main hub between API requests and lower-level automation components.

4. Browser automation boundary

The browser layer manages Playwright-based browser/context/page lifecycles. It sits beneath orchestration and above page-specific logic, so it is the component that actually interacts with live survey pages. Retry, checkpoint/restore, and recovery strategies live close to this layer because they need to react to browser failures.

5. Survey platform abstraction

A dedicated adapter subsystem normalizes different survey platforms: Google Forms, Typeform, Qualtrics, SurveyMonkey, Jotform, Microsoft Forms, LimeSurvey, and generic HTML forms. AdapterRegistry handles auto-detection, so the rest of the system can operate against a common interface instead of platform-specific code.

6. Page understanding and self-healing

The DOM subsystem provides DOM graph building, selector generation/healing, mutation tracking, and accessibility parsing. This is the architectural bridge between raw browser pages and reliable automation, especially when survey UIs change dynamically or selectors become brittle.

7. AI and reasoning layer

The AI subsystem adds semantic question interpretation, persona-based response generation, sentiment/intent classification, and multi-provider LLM support. This layer influences what answers get generated, but it is logically separate from the browser so reasoning can evolve independently of automation mechanics.

8. Vision and multimodal analysis

The vision layer uses OpenCV-based detection plus OCR engines to interpret screenshots and hard-to-parse controls. A multimodal reasoner combines visual evidence with DOM/semantic context when text-only parsing is insufficient, such as sliders, ratings, or image-heavy pages.

9. Memory, learning, and persistence

Long-term behavior is split across vector/memory components and database persistence. The project uses async SQLAlchemy plus migration support (alembic.ini, migrations/) for structured state, and a PGVector-backed store for embeddings, knowledge, and adaptive learning data. This is where historical survey outcomes feed future strategy.

10. Distributed execution

Background work is offloaded to Celery workers with Redis as the queue/broker layer. The worker deployment in Kubernetes and the worker-specific Dockerfile indicate that tasks like screenshot analysis, embedding generation, and session analysis are expected to scale independently from the API.

11. Plugin and event-driven extensibility

The architecture is intentionally event-driven and plugin-based. Hooks exist around session/page/question/submit/error lifecycle points, which makes custom logging, analytics, or transformations additive rather than invasive. The EventBus is the main decoupling mechanism here.

12. Observability stack

Observability is a first-class subsystem: structlog for structured logs, OpenTelemetry for tracing, and Prometheus metrics. The repo also includes ready-made Grafana, Loki, and Tempo configs and dashboards, which means runtime visibility is designed into the deployment rather than added later.

13. Deployment targets

The project supports three deployment modes: local Docker Compose, minimal Kubernetes manifests in k8s/, and a full Helm chart in helm/surveybot. The Helm chart splits API and worker deployments, exposes ingress/service configuration, and includes config/secrets/horizontal scaling resources.

14. Quality and failure testing

The test suite includes normal integration tests plus chaos tests for Redis failures, worker crashes, and replay storms. That tells you the system is expected to tolerate partial failures in the queue/worker layer and recover session execution without losing the whole workflow.

15. What to show in an architecture diagram

A good diagram should emphasize: client/API entry → session orchestration → event bus → browser/DOM/adapter pipeline, with side branches into AI, vision, persistence/memory, and Celery/Redis workers; plus an observability layer spanning everything and deployment boundaries around API, worker, Postgres, Redis, and monitoring.