# quesmaorg/quesma architecture

> Architecture diagram of the GitHub repository quesmaorg/quesma, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/quesmaorg/quesma
- Repository: https://github.com/quesmaorg/quesma

## Overview

Based on our review of the file tree and README, Quesma is a specialized database gateway written in Go that intercepts queries from front-end clients (such as Kibana or log generators) and translates them into the query languages of various backend databases (for example, Clickhouse and Elasticsearch). This middleware allows you to migrate between databases without changing your client query logic.

##### Main Components
- External Clients – Represent “Kibana,” “log-generator” (and any end-user client) as the sources of HTTP/TCP queries.
o Frontend Connectors (in “quesma/frontend_connectors”) which handle incoming requests using various protocols (HTTP, TCP, etc.),
o Query Parsing/Translation Modules (present in “quesma/parsers” and “quesma/model”) that convert the incoming query into an internal representation,
o Processors (in “quesma/processors”) that manage further transformation logic (including dual writes, transparent proxy behavior, and AB testing),
o Backend Connectors (in “quesma/backend_connectors” and “quesma/connectors”) that are responsible for sending the translated queries to the target databases.
- Ingestion & Logging – There are additional sub-systems such as “ch_ingest,” “ingest” directories, and the “logger” components that manage data ingestion, logging, and telemetry.
- Configuration, Licensing, and Monitoring – Include components for configuration management (using config files, environment-based configuration from “quesma/config”), licensing (“quesma/licensing”), health-checks, and telemetry (within “quesma/telemetry” and “ci” directories for testing pipelines).

##### Relationships and Data Flow
- Request Receipt → Query Parser/Translator → Query Processors → Backend Connectors.
- A feedback loop where responses from backend databases are routed back through the connector and returned to clients.
- Separate flows for logging, monitoring, and telemetry that collect data from various components.
- Optionally, represent AB Testing and dual-write flows (as seen in the “ab_testing” directory) to illustrate experimental processing or failover for rolling migrations.

##### Architectural Patterns and Design Principles
- Separation of Concerns: Distinguish between request intake (frontend connectors), business logic (parsing, translation, processing), and backend communication (backend connectors).
- Modularity: Modules like query parsers, licensing, logging, and telecommunication are separated into distinct directories, enabling easier extensibility and testing.
- Scalability: The use of different pipelines (for ingestion, transparent proxying, dual write configurations) suggests a design that supports high throughput and reliability.

##### Relevant Technologies
- Languages/Frameworks: Indicate that the system is built in Golang.
- Deployment: Note the usage of Docker (Dockerfiles in “bin” and “quesma” directories, plus deployment automation under “.github/workflows” and the helm-chart).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% External Clients Layer
    subgraph "External Clients"
        kibana["Kibana"]:::external
        loggen["Log-generator"]:::external
    end

    %% Quesma Gateway Layer
    subgraph "Quesma Gateway"
        fc["Frontend Connector"]:::gateway
        qp["Query Parser & Translator"]:::gateway
        proc["Processors"]:::gateway
        abt["AB Testing Pipeline"]:::gateway
        bc["Backend Connector"]:::gateway
    end

    %% Ingestion & Logging Layer
    subgraph "Ingestion & Logging"
        ing["Ingestion Module"]:::ingestion
        log["Logger"]:::ingestion
    end

    %% Configuration, Licensing & Monitoring Layer
    subgraph "Config, Licensing & Monitoring"
        cfg["Configuration"]:::config
        linc["Licensing"]:::config
        telem["Telemetry & Health-checks"]:::config
    end

    %% Backend Databases Layer
    subgraph "Backend Databases"
        clickhouse["ClickHouse"]:::db
        elasticsearch["Elasticsearch"]:::db
        mysql["MySQL"]:::db
        postgres["Postgres"]:::db
    end

    %% Data Flow from External Clients to Quesma Gateway
    kibana -->|"HTTP/TCPQuery"| fc
    loggen -->|"HTTP/TCPQuery"| fc

    %% Internal Quesma Gateway Flow
    fc -->|"ParseRequest"| qp
    qp -->|"Transform"| proc
    proc -->|"ABTest"| abt
    abt -->|"Result"| proc
    proc -->|"ForwardQuery"| bc

    %% Return Flow from Backend Databases to Frontend Connector (Response Loop)
    bc -->|"SendQuery"| clickhouse
    bc -->|"SendQuery"| elasticsearch
    bc -->|"SendQuery"| mysql
    bc -->|"SendQuery"| postgres
    clickhouse -->|"Response"| fc
    elasticsearch -->|"Response"| fc
    mysql -->|"Response"| fc
    postgres -->|"Response"| fc

    %% Telemetry, Logging and Ingestion Hooks
    fc -->|"DataIngest"| ing
    fc -->|"TelemetryLog"| log
    proc -->|"LogData"| log

    %% Configuration and Licensing Hooks
    fc -->|"ConfigCheck"| cfg
    proc -->|"LicenseVerify"| linc
    proc -->|"Metrics"| telem

    %% Click Events for Component Mapping
    click kibana "https://github.com/quesmaorg/quesma/tree/main/ci/kibana"
    click kibana "https://github.com/quesmaorg/quesma/tree/main/examples/kibana-sample-data/kibana"
    click loggen "https://github.com/quesmaorg/quesma/tree/main/ci/log-generator"
    click loggen "https://github.com/quesmaorg/quesma/tree/main/examples/kibana-sample-data/log-generator"
    click fc "https://github.com/quesmaorg/quesma/tree/main/quesma/frontend_connectors"
    click qp "https://github.com/quesmaorg/quesma/tree/main/quesma/parsers"
    click qp "https://github.com/quesmaorg/quesma/tree/main/quesma/model"
    click proc "https://github.com/quesmaorg/quesma/tree/main/quesma/processors"
    click bc "https://github.com/quesmaorg/quesma/tree/main/quesma/backend_connectors"
    click bc "https://github.com/quesmaorg/quesma/tree/main/quesma/connectors"
    click ing "https://github.com/quesmaorg/quesma/tree/main/quesma/ingest"
    click ing "https://github.com/quesmaorg/quesma/tree/main/bin/ingest-clickhouse"
    click log "https://github.com/quesmaorg/quesma/tree/main/quesma/logger"
    click cfg "https://github.com/quesmaorg/quesma/blob/main/quesma/config.yaml.template"
    click cfg "https://github.com/quesmaorg/quesma/tree/main/quesma/quesma/config"
    click linc "https://github.com/quesmaorg/quesma/tree/main/quesma/licensing"
    click telem "https://github.com/quesmaorg/quesma/tree/main/quesma/telemetry"
    click abt "https://github.com/quesmaorg/quesma/tree/main/quesma/ab_testing"

    %% Styles
    classDef external fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef gateway fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef db fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef ingestion fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef config fill:#DDA0DD,stroke:#000,stroke-width:2px;
```
