# pinterest/querybook architecture

> Architecture diagram of the GitHub repository pinterest/querybook, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/pinterest/querybook
- Repository: https://github.com/pinterest/querybook

## Overview

The Querybook project is a full‐stack Big Data IDE that supports composing, scheduling, and visualizing data queries. Its purpose is to allow users to manage data analyses by writing queries, annotating them in “DataDocs” (rich text documents that include charts), and enabling collaboration. It also supports integrations with various query engines (like Presto, Hive, Snowflake, etc.), authentication methods (user/password, OAuth variants, LDAP), and external storage for query results and notifications.

##### Main Components
- Frontend (Web Application): Represented by the “querybook/webapp” folder. This is built using React (with Redux for state management) and styled components. Include core UI components such as the Query Editor, Dashboard views, and supporting libraries (Storybook components, UI elements).
- Backend (Server-side Application): Represented by the “querybook/server” directory. This includes the main web server (Flask or similar Python framework), authentication modules, API endpoints, job handling for query execution and scheduling, plugins for query review, export, and other integrations.
- Database and Storage: There is evidence of database migration scripts and configuration files (in folders like “querybook/migrations” and configuration YAML files). This component holds metadata (user details, query histories, shared analyses) as well as logs and result stores (MySQL, Postgres, S3, etc.).
- Query Engines Integration: The backend has adapters and plugins (in “plugins/” and “querybook/server/datasources”) to integrate with a variety of query engines like Presto, Hive, Snowflake, etc. Represent each external query engine as separate blocks.
- External Services and Integrations:
- Authentication Providers (OAuth, LDAP, Auth0, GitHub, etc.)
- Metastore integrations (Hive, AWS Glue, and SQLAlchemy Inspect)
- Notification services (Email, Slack)
- Result export services (Google Sheets, Python-based export)
- Containerization and Orchestration: Based on Dockerfiles, Docker Compose files (in “containers” and “docker-compose.yml”), Helm charts, and Kubernetes YAML files, include a deployment environment component that orchestrates the service containers.
- Message Queues and Task Scheduling: Represent Celery or any asynchronous task processing components found in “querybook/server/tasks” and “querybook/server/lib/celery”. This is used for asynchronous query execution, cleanup tasks, and event logging.

##### Relationships and Interactions
- The Backend communicates with the Database for persisting metadata, query execution history, configuration, and user management.
- Authentication modules in the Backend communicate with external providers (OAuth, LDAP, etc.).
- The Query Execution tasks (managed via Celery/Task Scheduler) interact with the backend and database to update query status.
- The Notification system in the backend sends alerts via Email or Slack upon query completion or errors.

##### Architectural Patterns and Design Principles
- A layered architecture:
- Presentation Layer: Frontend React application with Redux and Storybook UI components.
- API/Service Layer: Backend services handling business logic, authentication, and plugin integrations.
- Data Access Layer: Database interactions, data migrations, and persistence mechanisms.
- Plugin Architecture: Modules in “plugins/” extend functionality (e.g., AI Assistant, authentication plugins, metastore plugins).
- Use clear directional data flow (from front-end input to backend processing, then to query engines, followed by logging and notifications).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        direction TB
        FE["React Webapp"]:::frontendStyle
        QE["Query Editor"]:::uiComponent
        DASH["Dashboard Views"]:::uiComponent
        FE --> QE
        FE --> DASH
    end

    %% Backend & Integrations
    subgraph "Backend & Integrations"
        direction TB
        BE["API Server"]:::backendStyle
        AUTH["Authentication Module"]:::backendStyle
        QEINT["Query Engine Integration"]:::backendStyle
        TASK["Task Queue (Celery)"]:::backendStyle
        NOTIF["Notification & Logging"]:::backendStyle
        BE -->|"apis_auth"| AUTH
        BE -->|"apis_query"| QEINT
        BE -->|"apis_tasks"| TASK
        BE -->|"apis_notify"| NOTIF
    end

    %% Data & Storage Layer
    subgraph "Data & Storage"
        direction TB
        DB["MySQL/Postgres Database"]:::databaseStyle
        MIG["Migration Scripts"]:::databaseStyle
        CONF["Configuration Files"]:::databaseStyle
        DB --> MIG
        DB --> CONF
    end

    %% External Services
    subgraph "External Services"
        direction TB
        OAUTH["OAuth Provider"]:::externalStyle
        LDAP["LDAP Provider"]:::externalStyle
        QEEXT["External Query Engines"]:::externalStyle
        ALERT["Email/Slack Notifications"]:::externalStyle
        AUTH -->|"calls"| OAUTH
        AUTH -->|"calls"| LDAP
        QEINT -->|"sends_queries"| QEEXT
        NOTIF -->|"sends_alerts"| ALERT
    end

    %% Deployment Environment
    subgraph "Deployment Environment"
        direction TB
        DEPLOY["Deployment Orchestrator"]:::containerStyle
    end

    %% User Interaction and Data Flow
    USER["End User"]:::externalStyle
    USER -->|"interacts"| FE
    FE -->|"REST_API_calls"| BE
    TASK -->|"updates_status"| DB

    %% Click Events
    click FE "https://github.com/pinterest/querybook/tree/master/querybook/webapp"
    click BE "https://github.com/pinterest/querybook/tree/master/querybook/server"
    click MIG "https://github.com/pinterest/querybook/tree/master/querybook/migrations"
    click CONF "https://github.com/pinterest/querybook/tree/master/querybook/config"
    click QEINT "https://github.com/pinterest/querybook/tree/master/plugins/"
    click QEINT "https://github.com/pinterest/querybook/tree/master/querybook/server/datasources"
    click AUTH "https://github.com/pinterest/querybook/tree/master/querybook/server/app/auth"
    click AUTH "https://github.com/pinterest/querybook/tree/master/plugins/auth_plugin"
    click NOTIF "https://github.com/pinterest/querybook/tree/master/querybook/notification_templates"
    click NOTIF "https://github.com/pinterest/querybook/tree/master/plugins/notifier_plugin"
    click DEPLOY "https://github.com/pinterest/querybook/tree/master/containers/"
    click DEPLOY "https://github.com/pinterest/querybook/blob/master/docker-compose.yml"
    click DEPLOY "https://github.com/pinterest/querybook/tree/master/k8s/"
    click TASK "https://github.com/pinterest/querybook/tree/master/querybook/server/tasks"
    click TASK "https://github.com/pinterest/querybook/tree/master/querybook/server/lib/celery"

    %% Styles
    classDef frontendStyle fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef uiComponent fill:#87CEFA,stroke:#000,stroke-width:1px;
    classDef backendStyle fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef databaseStyle fill:#FFFFE0,stroke:#000,stroke-width:2px;
    classDef externalStyle fill:#FFB6C1,stroke:#000,stroke-width:2px;
    classDef containerStyle fill:#D3D3D3,stroke:#000,stroke-width:2px;
```
