How quesmaorg/quesma works
Based on our review of the file tree and README, Quesma is a specialized database gateway written in Go that intercepts queries from front-end clients (such as Kibana or log generators) and translates them into the query languages of various backend databases (for example, Clickhouse and Elasticsearch). This middleware allows you to migrate between databases without changing your client query logic.
Main Components
• External Clients – Represent “Kibana,” “log-generator” (and any end-user client) as the sources of HTTP/TCP queries.
o Frontend Connectors (in “quesma/frontend_connectors”) which handle incoming requests using various protocols (HTTP, TCP, etc.),
o Query Parsing/Translation Modules (present in “quesma/parsers” and “quesma/model”) that convert the incoming query into an internal representation,
o Processors (in “quesma/processors”) that manage further transformation logic (including dual writes, transparent proxy behavior, and AB testing),
o Backend Connectors (in “quesma/backend_connectors” and “quesma/connectors”) that are responsible for sending the translated queries to the target databases.
• Ingestion & Logging – There are additional sub-systems such as “ch_ingest,” “ingest” directories, and the “logger” components that manage data ingestion, logging, and telemetry.
• Configuration, Licensing, and Monitoring – Include components for configuration management (using config files, environment-based configuration from “quesma/config”), licensing (“quesma/licensing”), health-checks, and telemetry (within “quesma/telemetry” and “ci” directories for testing pipelines).
Relationships and Data Flow
• Request Receipt → Query Parser/Translator → Query Processors → Backend Connectors.
• A feedback loop where responses from backend databases are routed back through the connector and returned to clients.
• Separate flows for logging, monitoring, and telemetry that collect data from various components.
• Optionally, represent AB Testing and dual-write flows (as seen in the “ab_testing” directory) to illustrate experimental processing or failover for rolling migrations.
Architectural Patterns and Design Principles
• Separation of Concerns: Distinguish between request intake (frontend connectors), business logic (parsing, translation, processing), and backend communication (backend connectors).
• Modularity: Modules like query parsers, licensing, logging, and telecommunication are separated into distinct directories, enabling easier extensibility and testing.
• Scalability: The use of different pipelines (for ingestion, transparent proxying, dual write configurations) suggests a design that supports high throughput and reliability.
Relevant Technologies
• Languages/Frameworks: Indicate that the system is built in Golang.
• Deployment: Note the usage of Docker (Dockerfiles in “bin” and “quesma” directories, plus deployment automation under “.github/workflows” and the helm-chart).