# vinaykhanchi712/onql-vk architecture

> Architecture diagram of the GitHub repository vinaykhanchi712/onql-vk, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-30). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/vinaykhanchi712/onql-vk
- Repository: https://github.com/vinaykhanchi712/onql-vk

## Overview

##### Project Type and Purpose
- ONQL is a backend‐focused, API‐first, context‐aware database engine written in Go with a custom DSL parser, evaluator, pluggable protocol extensions (in Python), and support for multiple storage backends.
- Its goal is to allow clients and services to query related data without manual joins by using an “Object Notation Query Language” and context‐aware protocols.

##### File Structure Analysis
- cmd/server: Entry point – starts the TCP/HTTP server.
- config: Environment and configuration loader.
- router: Routes incoming DSL/API requests to handlers (CUD, DSL, streaming, extensions).
- dsl: Parser (lexer, parser) and evaluator of the ONQL language.
- database & storemanager: Schema definitions, CRUD operations, validation, setup; abstractions over physical stores.
- engine: Adapters for BadgerDB, CacheDB, NATS messaging.
- extensions: Python‐based protocol and schema plugins that define context and relations via JSON + request handlers.
- server: Low‐level TCP server implementation.
- utils: Shared helpers (ULID, string utilities).
- docker & .github: Deployment and CI configuration.

##### Key README Insights
- Emphasizes a layered, plugin‐ready architecture: Core engine interprets DSL → consults protocol definitions → executes against storage backends → returns results via API.
- Supports context‐aware security, migrations, CLI, drivers, ORMs, and plugin SDK.

##### Main Components
a. Client / CLI / API Consumer
- Sends ONQL queries or commands
b. Entry Server Layer
- cmd/server → server/tcp_server.go
- router/setup.go & route.go
c. DSL Subsystem
- parser (dsl/parser)
- evaluator (dsl/evaluator)
d. Protocol Manager / Extension Host
- extensions/protocol & extensions/schema (JSON protocol files + Python handlers)
e. Query Engine & Validation
- database/* (CRUD, schema, validation)
- storemanager/* (data synchronization, indexing)
f. Storage Backends
- engine/badgerdb.go
- engine/cashedb.go
- message broker: engine/nats.go
g. Configuration
- config/config.go & .env files
h. Deployment
- docker/compose.build.yml, Dockerfiles, stack.yaml

##### Relationships & Interactions
- Client → Server Layer (TCP/HTTP) → Router
- Router → DSL Parser → DSL Evaluator
- Evaluator → Protocol Manager (loads JSON, calls Python request_handler via IPC/HTTP)
- Evaluator → Query Engine → Database Layer → Storemanager → Storage Adapters
- Engine adapters read/write to BadgerDB or in‐memory cache, publish events to NATS
- Results bubble back through evaluator → router → client

##### Architectural Patterns & Principles
- Layered Architecture: clear separation of transport, routing, DSL, engine, storage.
- Plugin/Extension Pattern: protocol and schema extensibility via external Python services.
- Adapter Pattern: storage backends abstracted behind engine/adapter interfaces.
- Microservice‐ready: NATS messaging for eventing, Docker for containerization.

##### Essential Elements to Capture
- Config loader feeding all layers with environment values.
- Router’s multiple handlers: CUD, DSL queries, extensions, streams.
- Flow from client to persistent storage and back.
- Extension protocols as first‐class citizens in the request pipeline.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Configuration Layer
    subgraph "Configuration" 
        direction TB
        CFG["Config Loader"]:::config
        CFG_ENV[".env"]:::config
    end

    %% Transport/Server Layer
    subgraph "Transport/Server Layer"
        direction TB
        CMD["cmd/server/main.go"]:::transport
        TCP["server/tcp_server.go"]:::transport
    end

    %% Router Layer
    subgraph "Router" 
        direction TB
        SETUP["router/setup.go"]:::transport
        ROUTE["router/route.go"]:::transport
        CUD["router/cud.go"]:::transport
        DSL_R["router/dsl.go"]:::transport
        EXT_R["router/ext.go"]:::transport
        DB_R["router/database.go"]:::transport
        STRM["router/stream.go"]:::transport
    end

    %% DSL Subsystem
    subgraph "DSL Subsystem"
        direction TB
        DSL_MAIN["dsl/main.go"]:::dsl
        LEX["dsl/parser/lexer.go"]:::dsl
        PARSER["dsl/parser/parser.go"]:::dsl
        EVAL["dsl/evaluator/evaluator.go"]:::dsl
    end

    %% Protocol Manager / Extensions Host
    subgraph "Extensions Host"
        direction TB
        PROTO_REG["extensions/registry.json"]:::extension
        subgraph "Protocol Plugins"
            direction TB
            PROTO_JSON["extensions/protocol/protocol.py"]:::extension
            PROTO_REQ["extensions/protocol/request_handler.py"]:::extension
        end
        subgraph "Schema Plugins"
            direction TB
            SCHEMA_JSON["extensions/schema/schema.py"]:::extension
            SCHEMA_REQ["extensions/schema/request_handler.py"]:::extension
        end
    end

    %% Core Engine & Validation
    subgraph "Core Engine"
        direction TB
        DB_SCHEMA["database/schema.go"]:::engine
        DB_PROTO["database/protocol.go"]:::engine
        DB_SETUP["database/setup.go"]:::engine
        DB_VAL["database/validate.go"]:::engine
    end

    %% Store Manager
    subgraph "Store Manager"
        direction TB
        SM_SETUP["storemanager/setup.go"]:::engine
        SM_DATA["storemanager/data.go"]:::engine
        SM_SYNC["storemanager/data_sync.go"]:::engine
        SM_IDX["storemanager/index.go"]:::engine
        SM_IDX_SYNC["storemanager/index_sync.go"]:::engine
        SM_CASHE["storemanager/cashe.go"]:::engine
        SM_PROTO["storemanager/protocol.go"]:::engine
        SM_SCHEMA["storemanager/schema.go"]:::engine
    end

    %% Storage Backends & Broker
    subgraph "Storage Backends"
        direction TB
        BADGER["engine/badgerdb.go"]:::storage
        CACHEDB["engine/cashedb.go"]:::storage
        NATS["engine/nats.go"]:::storage
    end

    %% Deployment & CI
    subgraph "Deployment & CI"
        direction TB
        COMPOSE["docker/compose.build.yml"]:::deployment
        DOCKER_ONQL["docker/onql.dockerfile"]:::deployment
        DOCKER_EXT["docker/ext.dockerfile"]:::deployment
        STACK["docker/stack.yaml"]:::deployment
        CI[".github/workflows/build_and_deploy_image.yaml"]:::deployment
    end

    %% Utilities
    UTIL_STR["utils/string.go"]:::util
    UTIL_ULID["utils/ulid.go"]:::util

    %% External Client
    CLIENT["Client / CLI"]:::external

    %% Connections
    CLIENT -->|"ONQL query / command"| CMD
    CMD --> TCP
    TCP --> SETUP
    SETUP --> ROUTE
    ROUTE --> CUD
    ROUTE --> DSL_R
    ROUTE --> EXT_R
    ROUTE --> DB_R
    ROUTE --> STRM

    DSL_R --> DSL_MAIN
    DSL_MAIN --> LEX
    LEX --> PARSER
    PARSER --> EVAL

    EVAL -->|"Protocol JSON"| PROTO_REG
    PROTO_REG --> PROTO_JSON
    PROTO_JSON --> PROTO_REQ
    PROTO_REG --> SCHEMA_JSON
    SCHEMA_JSON --> SCHEMA_REQ

    EVAL --> DB_SCHEMA
    DB_SCHEMA --> DB_PROTO
    DB_PROTO --> DB_SETUP
    DB_SETUP --> DB_VAL

    DB_VAL --> SM_SETUP
    SM_SETUP --> SM_DATA
    SM_DATA --> SM_SYNC
    SM_DATA --> SM_IDX
    SM_IDX --> SM_IDX_SYNC
    SM_DATA --> SM_CASHE
    SM_PROTO --> SM_SCHEMA
    SM_SCHEMA --> SM_SETUP

    SM_SETUP --> BADGER
    SM_SETUP --> CACHEDB
    SM_SETUP --> NATS

    NATS -.->|"event publish"| CLIENT

    CFG --> CMD
    CFG --> TCP
    CFG --> SETUP
    CFG --> DB_SETUP
    CFG --> SM_SETUP
    CFG --> BADGER
    CFG_ENV --> CFG

    UTIL_STR --> DSL_MAIN
    UTIL_ULID --> SM_DATA

    COMPOSE --> CMD
    COMPOSE --> PROTO_JSON
    COMPOSE --> SCHEMA_JSON
    DOCKER_ONQL --> CMD
    DOCKER_EXT --> PROTO_JSON
    STACK --> COMPOSE
    CI --> DOCKER_ONQL
    CI --> DOCKER_EXT

    %% Click Events
    click CMD "https://github.com/vinaykhanchi712/onql-vk/blob/main/cmd/server/main.go"
    click TCP "https://github.com/vinaykhanchi712/onql-vk/blob/main/server/tcp_server.go"
    click SETUP "https://github.com/vinaykhanchi712/onql-vk/blob/main/router/setup.go"
    click ROUTE "https://github.com/vinaykhanchi712/onql-vk/blob/main/router/route.go"
    click CUD "https://github.com/vinaykhanchi712/onql-vk/blob/main/router/cud.go"
    click DSL_R "https://github.com/vinaykhanchi712/onql-vk/blob/main/router/dsl.go"
    click EXT_R "https://github.com/vinaykhanchi712/onql-vk/blob/main/router/ext.go"
    click DB_R "https://github.com/vinaykhanchi712/onql-vk/blob/main/router/database.go"
    click STRM "https://github.com/vinaykhanchi712/onql-vk/blob/main/router/stream.go"
    click DSL_MAIN "https://github.com/vinaykhanchi712/onql-vk/blob/main/dsl/main.go"
    click LEX "https://github.com/vinaykhanchi712/onql-vk/blob/main/dsl/parser/lexer.go"
    click PARSER "https://github.com/vinaykhanchi712/onql-vk/blob/main/dsl/parser/parser.go"
    click EVAL "https://github.com/vinaykhanchi712/onql-vk/blob/main/dsl/evaluator/evaluator.go"
    click DB_SCHEMA "https://github.com/vinaykhanchi712/onql-vk/blob/main/database/schema.go"
    click DB_PROTO "https://github.com/vinaykhanchi712/onql-vk/blob/main/database/protocol.go"
    click DB_SETUP "https://github.com/vinaykhanchi712/onql-vk/blob/main/database/setup.go"
    click DB_VAL "https://github.com/vinaykhanchi712/onql-vk/blob/main/database/validate.go"
    click SM_SETUP "https://github.com/vinaykhanchi712/onql-vk/blob/main/storemanager/setup.go"
    click SM_DATA "https://github.com/vinaykhanchi712/onql-vk/blob/main/storemanager/data.go"
    click SM_SYNC "https://github.com/vinaykhanchi712/onql-vk/blob/main/storemanager/data_sync.go"
    click SM_IDX "https://github.com/vinaykhanchi712/onql-vk/blob/main/storemanager/index.go"
    click SM_IDX_SYNC "https://github.com/vinaykhanchi712/onql-vk/blob/main/storemanager/index_sync.go"
    click SM_CASHE "https://github.com/vinaykhanchi712/onql-vk/blob/main/storemanager/cashe.go"
    click SM_PROTO "https://github.com/vinaykhanchi712/onql-vk/blob/main/storemanager/protocol.go"
    click SM_SCHEMA "https://github.com/vinaykhanchi712/onql-vk/blob/main/storemanager/schema.go"
    click BADGER "https://github.com/vinaykhanchi712/onql-vk/blob/main/engine/badgerdb.go"
    click CACHEDB "https://github.com/vinaykhanchi712/onql-vk/blob/main/engine/cashedb.go"
    click NATS "https://github.com/vinaykhanchi712/onql-vk/blob/main/engine/nats.go"
    click CFG "https://github.com/vinaykhanchi712/onql-vk/blob/main/config/config.go"
    click CFG_ENV "https://github.com/vinaykhanchi712/onql-vk/blob/main/config/.env"
    click PROTO_JSON "https://github.com/vinaykhanchi712/onql-vk/blob/main/extensions/protocol/protocol.py"
    click PROTO_REQ "https://github.com/vinaykhanchi712/onql-vk/blob/main/extensions/protocol/request_handler.py"
    click SCHEMA_JSON "https://github.com/vinaykhanchi712/onql-vk/blob/main/extensions/schema/schema.py"
    click SCHEMA_REQ "https://github.com/vinaykhanchi712/onql-vk/blob/main/extensions/schema/request_handler.py"
    click PROTO_REG "https://github.com/vinaykhanchi712/onql-vk/blob/main/extensions/registry.json"
    click UTIL_STR "https://github.com/vinaykhanchi712/onql-vk/blob/main/utils/string.go"
    click UTIL_ULID "https://github.com/vinaykhanchi712/onql-vk/blob/main/utils/ulid.go"
    click COMPOSE "https://github.com/vinaykhanchi712/onql-vk/blob/main/docker/compose.build.yml"
    click DOCKER_ONQL "https://github.com/vinaykhanchi712/onql-vk/blob/main/docker/onql.dockerfile"
    click DOCKER_EXT "https://github.com/vinaykhanchi712/onql-vk/blob/main/docker/ext.dockerfile"
    click STACK "https://github.com/vinaykhanchi712/onql-vk/blob/main/docker/stack.yaml"
    click CI "https://github.com/vinaykhanchi712/onql-vk/blob/main/.github/workflows/build_and_deploy_image.yaml"

    %% Styles
    classDef transport fill:#D0E6FF,stroke:#0066CC;
    classDef dsl fill:#FFE0B2,stroke:#FF9900;
    classDef engine fill:#D0FFD6,stroke:#33AA33;
    classDef storage fill:#E3D2FF,stroke:#884ECC;
    classDef extension fill:#FFF3B0,stroke:#E6B800;
    classDef config fill:#CCE5FF,stroke:#3385FF,stroke-dasharray: 5 5;
    classDef deployment fill:#F2F2F2,stroke:#999999;
    classDef util fill:#F0F0F0,stroke:#666666;
    classDef external fill:#FFFFFF,stroke:#000000,stroke-dasharray: 2 2;
```
