How lotus-data/lotus works
Project Context and Overall Purpose
• LOTUS is a Python library that provides a Pandas-like API to build reasoning-based data processing pipelines. It lets users apply semantic operators such as sem_map, sem_filter, sem_extract, sem_join, etc. to manipulate both structured and unstructured data.
• Its design integrates large language models (LM), retrieval models (RM), and reranker models to facilitate natural language-driven data operations.
• The project architecture centers around a declarative semantic operator model, where users specify operations using natural language expressions (langex) and LOTUS orchestrates the underlying processing using a query engine.
High-Level Components
a. Client/Application Layer:
• User code or third-party applications that import LOTUS.
• This layer interacts with LOTUS through a Pandas-like API and triggers query pipelines.
b. Core Semantic Operator Engine:
• Semantic Operators Module:
▪ Contains modules in lotus/sem_ops (files such as sem_agg, sem_filter, sem_join, sem_map, etc.).
▪ Depicts how individual operators implement declarative transformations.
• Expression Interface:
▪ Handles natural language expressions (langex) that drive each operator’s logic.
c. Model Subsystems:
• Language Model (LM) Component:
▪ Found under lotus/models (e.g., lm.py, litellm interfaces).
▪ Integrates with external models via the LiteLLM library.
• Retrieval Model (RM) Component:
▪ Includes modules for retrieval tasks (e.g., rm.py, sentence transformer based RM).
• Reranker Model:
▪ Located under lotus/models (e.g., cross_encoder_reranker.py) for reordering results.
• To show interactions between the semantic operators and these model classes.
d. Data and File Handling:
• Data Connectors:
▪ Modules in lotus/data_connectors that manage connections to databases or various structured data sources.
• File Extractors:
▪ Modules in lotus/file_extractors for processing unstructured data (like PDFs or directories).
• Vector Store:
▪ Modules in lotus/vector_store wrapping Faiss, Weaviate, and similar libraries for similarity search and other vector-based operations.
e. Settings and Configuration:
• Global configuration components in lotus/settings.py which define default behaviors and model parameters.
• Utility modules (lotus/utils.py and lotus/types.py) for shared functions and type definitions.
f. External Dependencies & Interfaces:
• External libraries: FAISS (via conda installations), LiteLLM, and SentenceTransformers.
• Integrations with third-party LLM providers via API keys and external workflows (documented in the README).
g. Tests and Documentation:
• Testing modules (e.g., tests/ and .github/tests) which ensure system integrity.
• Documentation (docs/) complements the system design by explaining configurations, core concepts, and usage patterns.
Relationships and Data Flow
• Connection between data processing modules (data connectors, file extractors) and the vector store; for example, highlighting that data initially fetched or extracted can then be indexed or searched via the vector store.
• That configuration settings feed into several components (especially the LM/RM submodules) ensuring consistent behavior.
Architectural Patterns and Design Principles
• Modular design: Each core capability (semantic operators, model interfaces, data connectors) is encapsulated in its own directory/module.
• Declarative and compositional: The design promotes chaining of high-level operations similar to Pandas, enabling flexible query pipelines.
• Integration-focused: LOTUS seamlessly incorporates external AI models and vector search engines, suggesting the use of adapter patterns to couple these modules with the core functionality.
Final Assembly
• Leave room for potential extension points (as LOTUS is designed to support additional semantic operators or model integrations in the future).