# kmwtechnology/lucille architecture

> Architecture diagram of the GitHub repository kmwtechnology/lucille, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-02). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/kmwtechnology/lucille
- Repository: https://github.com/kmwtechnology/lucille

## Overview

##### Main Components
- Core Engine (lucille-core):
– Connector Layer: These classes (in packages like com.kmwllc.lucille.connector and its subpackages) handle extracting data from files, CSVs, databases, or even cloud storage. They include specialized connectors for CSV, JSON, file, JDBC, Solr, and XML inputs.
– Document and Pipeline Processing: The core packages (com.kmwllc.lucille.core and com.kmwllc.lucille.stage) manage Document creation, transformation stages (including enrichment, filtering, field normalization, regex-based modifications, etc.), and error handling.
– Worker & Thread Management: Components like Worker, WorkerPool, Runner, and RunnerManager enable parallel and fault-tolerant processing.
– Messaging Layer: The message package implements integration with messaging systems (for example, Kafka) to support distributed operations and inter-component communication.
– Indexer Layer: Modules under com.kmwllc.lucille.indexer implement the logic to push processed documents to search engines (e.g., using SolrClient, ElasticsearchIndexer, OpenSearchIndexer, etc.).

##### Plugins and Extensions
- Lucille Plugins (e.g. lucille-api, lucille-ocr, lucille-jlama, lucille-parquet, lucille-pinecone, lucille-tika, lucille-weaviate):
– These modules extend the core system with additional features. For example, lucille-api exposes a RESTful interface for operations like liveness/readiness checks and index management; lucille-ocr integrates OCR capabilities; and other plugins support specific connectors/indexers for various external platforms.
- BOM and Parent Modules (lucille-bom, lucille-parent) manage versioning and dependency aggregation across modules.

##### External System & Deployment Components
- Data Sources: External data (e.g., files, CSVs, JSONl, databases) from which data is extracted.
- Search Engines: External targets such as Apache Solr, Elasticsearch/OpenSearch, and vector search engines (Pinecone, Weaviate) to which the transformed "Documents" are ultimately loaded.
- Distributed Infrastructure: The provided examples (e.g., distributed examples in lucille-examples) include Docker Compose, Kafka, Zookeeper, and worker setups, which illustrate distributed and production deployments.

##### Relationships and Data Flow
- Documents then pass through the Pipeline (a configurable sequence of stages), where transformation, enrichment, error handling, and data validation occur.
- The processed Documents flow into the Indexing component, which then sends data to the appropriate external search engine.
- Additionally, the Messaging layer (using Kafka integration and other local/in-memory messengers) helps coordinate processing—especially in distributed deployments.
- A separate API layer can be exposed by one of the plugins (lucille-api) to monitor, control, or reconfigure the pipeline at runtime.

– One layer for input (Connectors),
– a core processing layer (Pipelines, Stages, Workers),
– an output layer (Indexers),
– and an external integration layer (Search Engines, message queues, APIs).
- Integrability and extensibility points – for example, plugins can easily be shown as both augmenting core processing and providing alternative paths for handling additional use cases (OCR, parquet files, etc.).

## Diagram (Mermaid)

```mermaid
flowchart TD

    %% External Systems
    subgraph "External Systems"
        DS["Data Sources\n(Files, CSV, JSON, DB)"]:::external
        SE["Search Engines\n(Solr, ES, OpenSearch,\nPinecone, Weaviate)"]:::external
    end

    %% Core Engine Layer
    subgraph "Core Engine"
        subgraph "Input & Extraction"
            Connector["Connector Layer"]:::core
        end
        
        subgraph "Processing"
            CoreProcessing["Document Processing"]:::core
            PipelineStages["Pipeline Stages"]:::core
        end
        
        subgraph "Worker Management"
            Worker["Worker"]:::core
            WorkerPool["WorkerPool"]:::core
            WorkerThread["WorkerThread"]:::core
        end
        
        subgraph "Messaging & Coordination"
            Messaging["Messaging Layer"]:::core
        end
        
        subgraph "Indexing"
            Indexer["Indexer Layer"]:::core
        end
    end

    %% Plugins & Extensions
    subgraph "Plugins"
        API["Lucille API"]:::plugin
        OCR["Lucille OCR"]:::plugin
        Jlama["Lucille Jlama"]:::plugin
        Parquet["Lucille Parquet"]:::plugin
        Pinecone["Lucille Pinecone"]:::plugin
        Tika["Lucille Tika"]:::plugin
        Weaviate["Lucille Weaviate"]:::plugin
    end

    %% Supporting Modules & Configurations
    subgraph "Modules & Configs"
        BOM["BOM Modules"]:::module
        Parent["Parent Modules"]:::module
        Distributed["Distributed Infrastructure"]:::external
        CICD["CI/CD Configurations"]:::external
    end

    %% Data Flow Connections
    DS -->|"extracts"| Connector
    Connector -->|"converts to Document"| CoreProcessing
    CoreProcessing -->|"triggers transformations"| PipelineStages
    PipelineStages -->|"dispatches tasks"| WorkerPool
    WorkerPool -->|"executes in parallel"| Worker
    WorkerPool -->|"communicates"| WorkerThread
    WorkerPool --> Messaging
    Messaging --> WorkerPool
    PipelineStages -->|"prepares for indexing"| Indexer
    Indexer -->|"loads data"| SE

    %% Plugin Integrations
    OCR -->|"enhances input"| Connector
    Jlama -->|"augment connectors"| Connector
    Parquet -->|"supports file formats"| Connector
    Tika -->|"extracts metadata"| Connector
    Pinecone -->|"vector indexing"| Indexer
    Weaviate -->|"vector search"| Indexer
    API ---|"monitors & controls"| PipelineStages
    API ---|"manages indexing"| Indexer

    %% Supporting Modules Connections
    Distributed --- Messaging
    Distributed --- WorkerPool
    BOM --- CoreProcessing
    Parent --- PipelineStages
    CICD --- CoreProcessing

    %% Click Events for Core Engine
    click Connector "https://github.com/kmwtechnology/lucille/tree/main/lucille-core/src/main/java/com/kmwllc/lucille/connector"
    click CoreProcessing "https://github.com/kmwtechnology/lucille/tree/main/lucille-core/src/main/java/com/kmwllc/lucille/core"
    click PipelineStages "https://github.com/kmwtechnology/lucille/tree/main/lucille-core/src/main/java/com/kmwllc/lucille/stage"
    click Worker "https://github.com/kmwtechnology/lucille/tree/main/lucille-core/src/main/java/com/kmwllc/lucille/core/Worker"
    click WorkerPool "https://github.com/kmwtechnology/lucille/tree/main/lucille-core/src/main/java/com/kmwllc/lucille/core/WorkerPool"
    click WorkerThread "https://github.com/kmwtechnology/lucille/tree/main/lucille-core/src/main/java/com/kmwllc/lucille/core/WorkerThread"
    click Messaging "https://github.com/kmwtechnology/lucille/tree/main/lucille-core/src/main/java/com/kmwllc/lucille/message"
    click Indexer "https://github.com/kmwtechnology/lucille/tree/main/lucille-core/src/main/java/com/kmwllc/lucille/indexer"

    %% Click Events for Plugins
    click API "https://github.com/kmwtechnology/lucille/tree/main/lucille-plugins/lucille-api"
    click OCR "https://github.com/kmwtechnology/lucille/tree/main/lucille-plugins/lucille-ocr"
    click Jlama "https://github.com/kmwtechnology/lucille/tree/main/lucille-plugins/lucille-jlama"
    click Parquet "https://github.com/kmwtechnology/lucille/tree/main/lucille-plugins/lucille-parquet"
    click Pinecone "https://github.com/kmwtechnology/lucille/tree/main/lucille-plugins/lucille-pinecone"
    click Tika "https://github.com/kmwtechnology/lucille/tree/main/lucille-plugins/lucille-tika"
    click Weaviate "https://github.com/kmwtechnology/lucille/tree/main/lucille-plugins/lucille-weaviate"

    %% Click Events for Modules & Configurations
    click BOM "https://github.com/kmwtechnology/lucille/tree/main/lucille-bom"
    click Parent "https://github.com/kmwtechnology/lucille/tree/main/lucille-parent"
    click Distributed "https://github.com/kmwtechnology/lucille/tree/main/lucille-examples/lucille-distributed-example"
    click CICD "https://github.com/kmwtechnology/lucille/tree/main/.github/workflows"

    %% Styles
    classDef core fill:#cce5ff,stroke:#004085,stroke-width:2px;
    classDef plugin fill:#d4edda,stroke:#155724,stroke-width:2px;
    classDef external fill:#fff3cd,stroke:#856404,stroke-width:2px;
    classDef module fill:#f8d7da,stroke:#721c24,stroke-width:2px;
```
