# getgather-hub/getgather architecture

> Architecture diagram of the GitHub repository getgather-hub/getgather, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-07-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/getgather-hub/getgather
- Repository: https://github.com/getgather-hub/getgather

## Overview

##### Project Type and Purpose
- Project type: A Python-based backend API service (monolithic service)
- Purpose: Expose HTTP endpoints that orchestrate headless browser sessions against brand websites (via connector specs) to detect, extract, and parse data on demand

1. Clients
– HTTP clients (curl, Postman, any consumer) or the built-in static front-end (docs at `/docs`)
2. API Layer (FastAPI)
– `getgather/api/main.py`
– Route handlers under `getgather/api/routes/...`
3. Orchestration Layer
– AuthOrchestrator (`auth_orchestrator.py`)
– ExtractOrchestrator (`extract_orchestrator.py`)
– Flow controller (`flow.py`, `flow_state.py`)
4. Connector Framework
– Spec Loader (`connectors/spec_loader.py`)
– Spec Models (`connectors/spec_models.py`)
– Brand specs (YAML files in `connectors/brand_specs/...`)
5. Browser Automation Engine
– Browser session management (`browser/session.py`)
– Freezable model support (`browser/freezable_model.py`)
– Profile handling (`browser/profile.py`)
6. Data Processing
– Detection (`detect.py`)
– Extraction (`extract.py`—through orchestrator)
– Parsing (`parse.py`)
– Analytics (`analytics.py`)
7. Infrastructure & Cross-Cutting Concerns
– Configuration (`config.py`, `.env.template`, `.envrc`)
– Logging (`logs.py`)
– Error tracking (Sentry integration via `sentry.py`)
8. Deployment / CI-CD
– Docker (Dockerfile, entrypoint.sh)
– GitHub Actions workflows (`.github/workflows/…`)
– Hosted targets (Fly.io, Railway; deployment docs in `docs/`)

1. Client → HTTP Request → API Layer
2. API Layer → calls → Orchestrator (auth or extract)
3. Orchestrator → loads → Connector Framework (loads the right brand spec)
4. Orchestrator → drives → Browser Automation Engine
5. Browser Engine → produces → raw page data → Data Processing
6. Data Processing → outputs → structured result → Orchestrator → returned → API Layer
7. Throughout each step, Logging and Sentry capture events/exceptions
8. Analytics module can be tapped anywhere to record metrics

##### Architectural Patterns & Design Principles
- Orchestrator pattern: separates high-level workflow (`auth_orchestrator`, `extract_orchestrator`) from implementation details
- Plugin-style connectors: each brand spec is a self-contained YAML describing how to interact with that site
- Layered architecture: API → Orchestration → Connectors → Automation → Processing
- Twelve-factor app readiness: config via env, dev/prod parity, logs as event streams

##### Technology Stack Labels
- Python 3.x + FastAPI (HTTP framework)
- Playwright (headless browser automation)
- YAML (brand specs via `yaml_utils.py`)
- Docker (containerization)
- GitHub Actions (CI/CD)
- Sentry (error monitoring)

##### Final Checklist
– Configuration and deployment elements sit outside the main data flow but are visibly connected

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Clients Layer
    subgraph "Clients"
        Docs["Docs Directory"]:::external
        HTTPClients["HTTP Clients"]:::external
        APIFrontend["Swagger UI (index.html)"]:::external
    end

    %% API Layer
    subgraph "API Layer"
        API_Main["getgather/api/main.py"]:::compute
        AuthEndpoints["Auth Endpoints"]:::compute
        BrandsEndpoints["Brands Endpoints"]:::compute
        SharedTypes["getgather/api/types.py"]:::compute
    end

    %% Orchestration Layer
    subgraph "Orchestration Layer"
        AuthOrch["getgather/auth_orchestrator.py"]:::compute
        ExtractOrch["getgather/extract_orchestrator.py"]:::compute
        Flow["Flow Control"]:::compute
    end

    %% Connector Framework
    subgraph "Connector Framework"
        SpecLoader["getgather/connectors/spec_loader.py"]:::data
        SpecModels["getgather/connectors/spec_models.py"]:::data
        BrandSpecs["Brand Specs (YAML)"]:::data
    end

    %% Browser Automation Engine
    subgraph "Browser Automation Engine"
        Session["getgather/browser/session.py"]:::compute
        Freezable["getgather/browser/freezable_model.py"]:::compute
        Profile["getgather/browser/profile.py"]:::compute
    end

    %% Data Processing
    subgraph "Data Processing"
        Detect["getgather/detect.py"]:::compute
        Parse["getgather/parse.py"]:::compute
        Analytics["getgather/analytics.py"]:::compute
    end

    %% Infrastructure & Cross-Cutting Concerns
    subgraph "Infra & Cross-Cutting Concerns"
        Config["getgather/config.py"]:::infra
        Env[".env.template/.envrc"]:::infra
        Logs["getgather/logs.py"]:::infra
        Sentry["getgather/sentry.py"]:::infra
    end

    %% Deployment / CI-CD
    subgraph "Deployment / CI-CD"
        Docker["Dockerfile & entrypoint.sh"]:::infra
        GHWorkflows[".github/workflows"]:::infra
        GHActions[".github/actions/prepare-backend"]:::infra
        DeployDocs["Deployment Guides"]:::infra
    end

    %% Data Flow
    HTTPClients -->|"HTTP request"| API_Main
    Docs -->|"view docs"| APIFrontend
    APIFrontend -->|"HTTP request"| API_Main
    API_Main -->|"route to auth"| AuthEndpoints
    API_Main -->|"route to brands"| BrandsEndpoints
    AuthEndpoints -->|"invoke auth"| AuthOrch
    BrandsEndpoints -->|"invoke extract"| ExtractOrch
    AuthOrch -->|"load spec"| SpecLoader
    ExtractOrch -->|"load spec"| SpecLoader
    SpecLoader -->|"uses models"| SpecModels
    SpecLoader -->|"loads files"| BrandSpecs
    AuthOrch -->|"start session"| Session
    ExtractOrch -->|"start session"| Session
    Session -->|"raw page data"| Detect
    Session -->|"raw page data"| Parse
    Detect -->|"detection result"| ExtractOrch
    Parse -->|"parsed JSON"| ExtractOrch
    ExtractOrch -->|"return result"| API_Main

    %% Cross-cutting
    API_Main -.-> Logs
    AuthOrch -.-> Logs
    ExtractOrch -.-> Logs
    Session -.-> Logs
    Detect -.-> Logs
    Parse -.-> Logs

    API_Main -.-> Sentry
    AuthOrch -.-> Sentry
    ExtractOrch -.-> Sentry
    Session -.-> Sentry
    Detect -.-> Sentry
    Parse -.-> Sentry

    API_Main -.-> Analytics
    AuthOrch -.-> Analytics
    ExtractOrch -.-> Analytics
    Detect -.-> Analytics
    Parse -.-> Analytics

    Config -.-> API_Main
    Config -.-> AuthOrch
    Config -.-> ExtractOrch

    %% Deployment Connections
    Docker -.-> API_Main
    GHWorkflows -.-> Docker
    GHActions -.-> GHWorkflows
    DeployDocs -.-> Docker

    %% Click Events
    click Docs "https://github.com/getgather-hub/getgather/tree/main//docs"
    click HTTPClients "https://github.com/getgather-hub/getgather/tree/main//docs"
    click APIFrontend "https://github.com/getgather-hub/getgather/blob/main/getgather/api/frontend/index.html"
    click API_Main "https://github.com/getgather-hub/getgather/blob/main/getgather/api/main.py"
    click AuthEndpoints "https://github.com/getgather-hub/getgather/blob/main/getgather/api/routes/auth/endpoints.py"
    click BrandsEndpoints "https://github.com/getgather-hub/getgather/blob/main/getgather/api/routes/brands/endpoints.py"
    click SharedTypes "https://github.com/getgather-hub/getgather/blob/main/getgather/api/types.py"
    click AuthOrch "https://github.com/getgather-hub/getgather/blob/main/getgather/auth_orchestrator.py"
    click ExtractOrch "https://github.com/getgather-hub/getgather/blob/main/getgather/extract_orchestrator.py"
    click Flow "https://github.com/getgather-hub/getgather/blob/main/getgather/flow.py"
    click SpecLoader "https://github.com/getgather-hub/getgather/blob/main/getgather/connectors/spec_loader.py"
    click SpecModels "https://github.com/getgather-hub/getgather/blob/main/getgather/connectors/spec_models.py"
    click BrandSpecs "https://github.com/getgather-hub/getgather/tree/main/getgather/connectors/brand_specs/"
    click Session "https://github.com/getgather-hub/getgather/blob/main/getgather/browser/session.py"
    click Freezable "https://github.com/getgather-hub/getgather/blob/main/getgather/browser/freezable_model.py"
    click Profile "https://github.com/getgather-hub/getgather/blob/main/getgather/browser/profile.py"
    click Detect "https://github.com/getgather-hub/getgather/blob/main/getgather/detect.py"
    click Parse "https://github.com/getgather-hub/getgather/blob/main/getgather/parse.py"
    click Analytics "https://github.com/getgather-hub/getgather/blob/main/getgather/analytics.py"
    click Config "https://github.com/getgather-hub/getgather/blob/main/getgather/config.py"
    click Env "https://github.com/getgather-hub/getgather/blob/main/.env.template"
    click Logs "https://github.com/getgather-hub/getgather/blob/main/getgather/logs.py"
    click Sentry "https://github.com/getgather-hub/getgather/blob/main/getgather/sentry.py"
    click Docker "https://github.com/getgather-hub/getgather/tree/main/Dockerfile"
    click GHWorkflows "https://github.com/getgather-hub/getgather/blob/main/.github/workflows/container.yml"
    click GHActions "https://github.com/getgather-hub/getgather/blob/main/.github/actions/prepare-backend/action.yaml"
    click DeployDocs "https://github.com/getgather-hub/getgather/blob/main/docs/deploy_fly.md"

    %% Styles
    classDef compute fill:#D0E6FF,stroke:#0366d6,color:#000;
    classDef data fill:#E0F8D8,stroke:#28a745,color:#000;
    classDef infra fill:#FFEDD5,stroke:#d15704,color:#000;
    classDef external fill:#F0F0F0,stroke:#999,color:#000;
```
