# lotus-data/lotus architecture

> Architecture diagram of the GitHub repository lotus-data/lotus, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-03-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/lotus-data/lotus
- Repository: https://github.com/lotus-data/lotus

## Overview

##### Project Context and Overall Purpose
- LOTUS is a Python library that provides a Pandas-like API to build reasoning-based data processing pipelines. It lets users apply semantic operators such as sem_map, sem_filter, sem_extract, sem_join, etc. to manipulate both structured and unstructured data.
- Its design integrates large language models (LM), retrieval models (RM), and reranker models to facilitate natural language-driven data operations.
- The project architecture centers around a declarative semantic operator model, where users specify operations using natural language expressions (langex) and LOTUS orchestrates the underlying processing using a query engine.

##### High-Level Components
a. Client/Application Layer:
- User code or third-party applications that import LOTUS.
- This layer interacts with LOTUS through a Pandas-like API and triggers query pipelines.

b. Core Semantic Operator Engine:
- Semantic Operators Module:
▪ Contains modules in lotus/sem_ops (files such as sem_agg, sem_filter, sem_join, sem_map, etc.).
▪ Depicts how individual operators implement declarative transformations.
- Expression Interface:
▪ Handles natural language expressions (langex) that drive each operator’s logic.

c. Model Subsystems:
- Language Model (LM) Component:
▪ Found under lotus/models (e.g., lm.py, litellm interfaces).
▪ Integrates with external models via the LiteLLM library.
- Retrieval Model (RM) Component:
▪ Includes modules for retrieval tasks (e.g., rm.py, sentence transformer based RM).
- Reranker Model:
▪ Located under lotus/models (e.g., cross_encoder_reranker.py) for reordering results.
- To show interactions between the semantic operators and these model classes.

d. Data and File Handling:
- Data Connectors:
▪ Modules in lotus/data_connectors that manage connections to databases or various structured data sources.
- File Extractors:
▪ Modules in lotus/file_extractors for processing unstructured data (like PDFs or directories).
- Vector Store:
▪ Modules in lotus/vector_store wrapping Faiss, Weaviate, and similar libraries for similarity search and other vector-based operations.

e. Settings and Configuration:
- Global configuration components in lotus/settings.py which define default behaviors and model parameters.
- Utility modules (lotus/utils.py and lotus/types.py) for shared functions and type definitions.

f. External Dependencies & Interfaces:
- External libraries: FAISS (via conda installations), LiteLLM, and SentenceTransformers.
- Integrations with third-party LLM providers via API keys and external workflows (documented in the README).

g. Tests and Documentation:
- Testing modules (e.g., tests/ and .github/tests) which ensure system integrity.
- Documentation (docs/) complements the system design by explaining configurations, core concepts, and usage patterns.

##### Relationships and Data Flow
- Connection between data processing modules (data connectors, file extractors) and the vector store; for example, highlighting that data initially fetched or extracted can then be indexed or searched via the vector store.
- That configuration settings feed into several components (especially the LM/RM submodules) ensuring consistent behavior.

##### Architectural Patterns and Design Principles
- Modular design: Each core capability (semantic operators, model interfaces, data connectors) is encapsulated in its own directory/module.
- Declarative and compositional: The design promotes chaining of high-level operations similar to Pandas, enabling flexible query pipelines.
- Integration-focused: LOTUS seamlessly incorporates external AI models and vector search engines, suggesting the use of adapter patterns to couple these modules with the core functionality.

##### Final Assembly
- Leave room for potential extension points (as LOTUS is designed to support additional semantic operators or model integrations in the future).

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Client/Application Layer
    subgraph "Client/Application Layer"
        A["Client Code (examples)"]:::client
    end

    %% Core Semantic Operator Engine
    subgraph "Core Semantic Operator Engine"
        %% Semantic Operators Subgraph
        subgraph "Semantic Operators"
            OPS1["sem_map Operator"]:::operator
            OPS2["sem_filter Operator"]:::operator
            OPS3["sem_join Operator"]:::operator
            EXPR["Expression Interface"]:::expression
        end

        %% Data Handling Modules Subgraph
        subgraph "Data Handling Modules"
            DC["Data Connectors"]:::data
            subgraph "File Extractors"
                FE1["Directory Reader"]:::data
                FE2["PPTX Extractor"]:::data
            end
            subgraph "Vector Store"
                VS1["FAISS VS"]:::data
                VS2["Weaviate VS"]:::data
            end
        end

        %% Configuration and Utilities
        subgraph "Configuration & Utilities"
            S["Settings"]:::config
            U["Utilities"]:::config
            Tt["Types"]:::config
        end
    end

    %% Model Services Layer
    subgraph "Model Services Layer"
        LM["Language Model (LM)"]:::model
        subgraph "Retrieval Models"
            RM1["Retrieval Model (rm.py)"]:::model
            RM2["ColBertv2 RM"]:::model
            RM3["Sentence Transformers RM"]:::model
        end
        RR["Reranker Model"]:::model
    end

    %% External Dependencies
    subgraph "External Dependencies"
        FAISS["FAISS"]:::external
        WEAV["Weaviate"]:::external
        LLLM["LiteLLM"]:::external
        SENT["SentenceTransformers"]:::external
    end

    %% Tests and Documentation
    subgraph "Tests and Documentation"
        TST["Tests"]:::test
        DOC["Documentation"]:::doc
    end

    %% Connections between layers
    A -->|"calls"| EXPR

    EXPR -->|"triggers"| OPS1
    EXPR -->|"triggers"| OPS2
    EXPR -->|"triggers"| OPS3

    OPS1 -->|"delegates"| LM
    OPS2 -->|"delegates"| RM1
    OPS3 -->|"delegates"| RR

    DC -->|"indexes"| VS1
    DC -->|"indexes"| VS2
    FE1 -->|"extracts"| VS1
    FE2 -->|"extracts"| VS2

    S --> LM
    S --> RM1
    S --> RR

    LM -->|"integrates"| LLLM
    LM -->|"integrates"| SENT
    RM1 -->|"integrates"| SENT

    VS1 -->|"uses"| FAISS
    VS2 -->|"uses"| WEAV

    %% Styles
    classDef client fill:#FFD700,stroke:#333,stroke-width:2px;
    classDef operator fill:#ADD8E6,stroke:#333,stroke-width:2px;
    classDef expression fill:#87CEFA,stroke:#333,stroke-width:2px;
    classDef data fill:#90EE90,stroke:#333,stroke-width:2px;
    classDef config fill:#FFB6C1,stroke:#333,stroke-width:2px;
    classDef model fill:#FFE4B5,stroke:#333,stroke-width:2px;
    classDef external fill:#D3D3D3,stroke:#333,stroke-width:2px;
    classDef test fill:#E6E6FA,stroke:#333,stroke-width:2px;
    classDef doc fill:#98FB98,stroke:#333,stroke-width:2px;

    %% Click Events
    click A "https://github.com/lotus-data/lotus/tree/main/examples/"
    click OPS1 "https://github.com/lotus-data/lotus/blob/main/lotus/sem_ops/sem_map.py"
    click OPS2 "https://github.com/lotus-data/lotus/blob/main/lotus/sem_ops/sem_filter.py"
    click OPS3 "https://github.com/lotus-data/lotus/blob/main/lotus/sem_ops/sem_join.py"
    click EXPR "https://github.com/lotus-data/lotus/blob/main/lotus/nl_expression.py"
    click LM "https://github.com/lotus-data/lotus/blob/main/lotus/models/lm.py"
    click RM1 "https://github.com/lotus-data/lotus/blob/main/lotus/models/rm.py"
    click RM2 "https://github.com/lotus-data/lotus/blob/main/lotus/models/colbertv2_rm.py"
    click RM3 "https://github.com/lotus-data/lotus/blob/main/lotus/models/sentence_transformers_rm.py"
    click RR "https://github.com/lotus-data/lotus/blob/main/lotus/models/cross_encoder_reranker.py"
    click DC "https://github.com/lotus-data/lotus/blob/main/lotus/data_connectors/connectors.py"
    click FE1 "https://github.com/lotus-data/lotus/blob/main/lotus/file_extractors/directory_reader.py"
    click FE2 "https://github.com/lotus-data/lotus/blob/main/lotus/file_extractors/pptx.py"
    click VS1 "https://github.com/lotus-data/lotus/blob/main/lotus/vector_store/faiss_vs.py"
    click VS2 "https://github.com/lotus-data/lotus/blob/main/lotus/vector_store/weaviate_vs.py"
    click S "https://github.com/lotus-data/lotus/blob/main/lotus/settings.py"
    click U "https://github.com/lotus-data/lotus/blob/main/lotus/utils.py"
    click Tt "https://github.com/lotus-data/lotus/blob/main/lotus/types.py"
    click TST "https://github.com/lotus-data/lotus/tree/main/tests/"
    click DOC "https://github.com/lotus-data/lotus/tree/main/docs/"
```
