How getgather-hub/getgather works
Project Type and Purpose
• Project type: A Python-based backend API service (monolithic service)
• Purpose: Expose HTTP endpoints that orchestrate headless browser sessions against brand websites (via connector specs) to detect, extract, and parse data on demand
1. Clients
– HTTP clients (curl, Postman, any consumer) or the built-in static front-end (docs at /docs)
2. API Layer (FastAPI)
– getgather/api/main.py
– Route handlers under getgather/api/routes/...
3. Orchestration Layer
– AuthOrchestrator (auth_orchestrator.py)
– ExtractOrchestrator (extract_orchestrator.py)
– Flow controller (flow.py, flow_state.py)
4. Connector Framework
– Spec Loader (connectors/spec_loader.py)
– Spec Models (connectors/spec_models.py)
– Brand specs (YAML files in connectors/brand_specs/...)
5. Browser Automation Engine
– Browser session management (browser/session.py)
– Freezable model support (browser/freezable_model.py)
– Profile handling (browser/profile.py)
6. Data Processing
– Detection (detect.py)
– Extraction (extract.py—through orchestrator)
– Parsing (parse.py)
– Analytics (analytics.py)
7. Infrastructure & Cross-Cutting Concerns
– Configuration (config.py, .env.template, .envrc)
– Logging (logs.py)
– Error tracking (Sentry integration via sentry.py)
8. Deployment / CI-CD
– Docker (Dockerfile, entrypoint.sh)
– GitHub Actions workflows (.github/workflows/…)
– Hosted targets (Fly.io, Railway; deployment docs in docs/)
1. Client → HTTP Request → API Layer
2. API Layer → calls → Orchestrator (auth or extract)
3. Orchestrator → loads → Connector Framework (loads the right brand spec)
4. Orchestrator → drives → Browser Automation Engine
5. Browser Engine → produces → raw page data → Data Processing
6. Data Processing → outputs → structured result → Orchestrator → returned → API Layer
7. Throughout each step, Logging and Sentry capture events/exceptions
8. Analytics module can be tapped anywhere to record metrics
Architectural Patterns & Design Principles
• Orchestrator pattern: separates high-level workflow (auth_orchestrator, extract_orchestrator) from implementation details
• Plugin-style connectors: each brand spec is a self-contained YAML describing how to interact with that site
• Layered architecture: API → Orchestration → Connectors → Automation → Processing
• Twelve-factor app readiness: config via env, dev/prod parity, logs as event streams
Technology Stack Labels
• Python 3.x + FastAPI (HTTP framework)
• Playwright (headless browser automation)
• YAML (brand specs via yaml_utils.py)
• Docker (containerization)
• GitHub Actions (CI/CD)
• Sentry (error monitoring)
Final Checklist
– Configuration and deployment elements sit outside the main data flow but are visibly connected