# cubeflow-ai/calmcore architecture

> Architecture diagram of the GitHub repository cubeflow-ai/calmcore, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-16). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/cubeflow-ai/calmcore
- Repository: https://github.com/cubeflow-ai/calmcore

## Overview

This project is a fast search core library called “calmcore” designed for efficient text and vector search capabilities. It’s structured as a modular Rust codebase with several components handling distinct aspects of search functionality (from text analysis to multiple indexing techniques) as well as integrations for external search protocols and networking.

##### Project Type and Purpose
- The README clearly states that calmcore is a “fast search core library.” Its purpose includes handling robust text and vector search. The project is centered on indexing data, processing queries, and analyzing textual input.
- It is not a full-stack application per se but the core search engine library, with additional client-facing network services built atop it (e.g., GraphQL and gRPC endpoints in the src/ folder).

##### File Structure
- The top-level directory includes key items like Cargo.toml, build.rs, configuration (config/), documentation (docs/), and examples.
- The calmcore directory contains the main crate with its own Cargo.toml, a rich README, and layered source code in src/ that shows:
- Analyzers (in calmcore/src/analyzer): responsible for tokenizing and dictionary management.
- Index Store (in calmcore/src/index_store): with multiple submodules for fulltext, term, and vector indexes. Notice that there is further separation between reading, writing, and handling streams.
- Persistence and storage functionality (persist/ and store/ directories), supporting both disk-based and memory-based storage.
- Protocols (in calmcore/src/protocols): integrating external interfaces like Elasticsearch and SQL, which indicates versatility in supporting various search-related APIs.
- The engine (calmcore/src/engine.rs) that likely ties together query processing, analysis, and index lookup.
- There are additional supportive libraries:
- libs/hora_lib: Contains core algorithms for approximate nearest neighbor search, k-means clustering, graph search (hnsw, ssg), etc.
- libs/mem_btree: Implements an in-memory B-tree for efficient data indexing.
- The proto directory further defines protocol buffers and generation scripts that hint at external communication (e.g., for a calmserver or related services).

##### README for Additional Insights
- The calmcore README lists new features like enhanced field types, support for vector search, batch indexing updates, schema adjustments, etc.
- Documentation in the docs/ folder (including an architecture overview) emphasizes distributed components, deployment strategies, and the internal design choices.

a. Main Components:
- Core Library (calmcore):
- Analyzer Module: Responsible for text processing, tokenization, and dictionary management.
- Index Store: Split into submodules for fulltext, term, and vector indexes. Further subdivided into reading, writing, storage & streaming components.
- Engine & Query Processor: Orchestrates search operations and query evaluations using the index store.
- Persistence Layer: Handles schema, storage persistence, and segment merging.
- Auxiliary Libraries:
- hora_lib: Implements advanced indexing algorithms and ANN (approximate nearest neighbor) methods.
- mem_btree: Provides in-memory B-tree data structure support for indexing.
- Communication Interfaces:
- Protocol Integration: Support for Elasticsearch and SQL protocols.
- External API Layers: Defined in src/graphql and src/grpc, which front the calmcore engine for client queries.
- Proto definitions: In the proto/ directory, used for structured communications (calmserver and core protocols).
- Configuration & Deployment Components:
- Config files (config/config.toml) and deployment documentation (docs/deployment.md) guide the runtime setup.

b. Relationships and Interactions:
- External clients interact through network layers: GraphQL endpoints (src/graphql) or gRPC (src/grpc) which connect to the core engine.
- The engine takes user queries, passing them through the Analyzer to tokenize and understand the query.
- Processed queries then access the appropriate index (fulltext, term, or vector) in the Index Store, which might involve both memory and disk storage layers.
- Auxiliary libraries (hora_lib, mem_btree) support specific indexing or retrieval functions, interfacing with calmcore.
- Protocols modules (within calmcore/src/protocols) translate or adapt the core search behaviors to external query standards like Elasticsearch or SQL.
- Configuration and logging or monitoring (potentially referenced in build scripts and config files) handle operational concerns.

c. Architectural Patterns or Design Principles:
- Modular Design: Clear separation between analysis, indexing, and network communication components.
- Separation of Concerns: Different modules (analyzer, index store, engine) focus on distinct responsibilities.
- Extensibility: Additional protocols and libraries (like hora_lib and mem_btree) can be updated or extended without altering the core search functionality.
- Reusability and Interoperability: Use of proto buffers and protocol-specific modules allows interfacing with multiple external systems.

d. Relevant Technologies, Frameworks, or Libraries:
- Rust: The entire codebase is implemented in Rust, leveraging Cargo for build and dependency management.
- Protocol Buffers: For defining external communication interfaces.
- GRPC / GraphQL: Found in the src/ directory for serving search queries.
- Custom indexing algorithms and in-memory data structures, particularly in hora_lib and mem_btree.

##### Final Detail
- Do not oversimplify—each module should be representatively broken down so that an engineer gets a clear view of the overall architecture.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Core Library Components
    subgraph "Core Library"
        direction TB
        Analyzer["Analyzer"]
        IndexStore["Index Store"]
        Engine["Engine & Query Processor"]
        Persistence["Persistence Layer"]
        Protocols["Protocol Integrations"]
    end

    %% External API Interfaces
    subgraph "External API Interfaces"
        direction TB
        GraphQL["GraphQL API"]
        gRPC["gRPC API"]
    end

    %% Auxiliary Libraries
    subgraph "Auxiliary Libraries"
        direction TB
        HoraLib["Advanced Indexing/ANN"]
        MemBTree["In-memory B-tree"]
    end

    %% Configuration & Communication
    subgraph "Configuration & Communication"
        direction TB
        ConfigFile["Configuration File"]
        DeploymentDoc["Deployment Documentation"]
        Proto["Protocol Buffers"]
    end

    %% Connections from External APIs to Core Engine
    GraphQL -->|"sendsQuery"| Engine
    gRPC -->|"sendsQuery"| Engine

    %% Internal Core Connections
    Engine -->|"callsAnalyzer"| Analyzer
    Engine -->|"queriesIndex"| IndexStore
    Engine -->|"persistsData"| Persistence
    Engine -->|"protocolAdapter"| Protocols

    IndexStore -->|"storesData"| Persistence
    IndexStore -->|"integratesANN"| HoraLib
    IndexStore -->|"utilizesBTree"| MemBTree

    %% Configuration guiding the Engine
    ConfigFile -->|"configures"| Engine
    DeploymentDoc -->|"guidesSetup"| Engine
    Proto -->|"definesComm"| Protocols

    %% Style Classes
    classDef core fill:#f9f,stroke:#333,stroke-width:2px;
    classDef api fill:#ccf,stroke:#333,stroke-width:2px;
    classDef aux fill:#cfc,stroke:#333,stroke-width:2px;
    classDef config fill:#fc9,stroke:#333,stroke-width:2px;

    class Analyzer,IndexStore,Engine,Persistence,Protocols core;
    class GraphQL,gRPC api;
    class HoraLib,MemBTree aux;
    class ConfigFile,DeploymentDoc,Proto config;

    %% Click Events for component mapping
    click Analyzer "https://github.com/cubeflow-ai/calmcore/tree/main/calmcore/src/analyzer"
    click IndexStore "https://github.com/cubeflow-ai/calmcore/tree/main/calmcore/src/index_store"
    click Engine "https://github.com/cubeflow-ai/calmcore/blob/main/calmcore/src/engine.rs"
    click Persistence "https://github.com/cubeflow-ai/calmcore/tree/main/calmcore/src/persist"
    click IndexStore "https://github.com/cubeflow-ai/calmcore/tree/main/calmcore/src/index_store/store"
    click Protocols "https://github.com/cubeflow-ai/calmcore/tree/main/calmcore/src/protocols"
    click GraphQL "https://github.com/cubeflow-ai/calmcore/tree/main/src/graphql"
    click gRPC "https://github.com/cubeflow-ai/calmcore/tree/main/src/grpc"
    click HoraLib "https://github.com/cubeflow-ai/calmcore/tree/main/libs/hora_lib"
    click MemBTree "https://github.com/cubeflow-ai/calmcore/tree/main/libs/mem_btree"
    click ConfigFile "https://github.com/cubeflow-ai/calmcore/blob/main/config/config.toml"
    click DeploymentDoc "https://github.com/cubeflow-ai/calmcore/blob/main/docs/deployment.md"
    click Proto "https://github.com/cubeflow-ai/calmcore/tree/main/proto"
```
