How kmwtechnology/lucille works
Main Components
• Core Engine (lucille-core):
– Connector Layer: These classes (in packages like com.kmwllc.lucille.connector and its subpackages) handle extracting data from files, CSVs, databases, or even cloud storage. They include specialized connectors for CSV, JSON, file, JDBC, Solr, and XML inputs.
– Document and Pipeline Processing: The core packages (com.kmwllc.lucille.core and com.kmwllc.lucille.stage) manage Document creation, transformation stages (including enrichment, filtering, field normalization, regex-based modifications, etc.), and error handling.
– Worker & Thread Management: Components like Worker, WorkerPool, Runner, and RunnerManager enable parallel and fault-tolerant processing.
– Messaging Layer: The message package implements integration with messaging systems (for example, Kafka) to support distributed operations and inter-component communication.
– Indexer Layer: Modules under com.kmwllc.lucille.indexer implement the logic to push processed documents to search engines (e.g., using SolrClient, ElasticsearchIndexer, OpenSearchIndexer, etc.).
Plugins and Extensions
• Lucille Plugins (e.g. lucille-api, lucille-ocr, lucille-jlama, lucille-parquet, lucille-pinecone, lucille-tika, lucille-weaviate):
– These modules extend the core system with additional features. For example, lucille-api exposes a RESTful interface for operations like liveness/readiness checks and index management; lucille-ocr integrates OCR capabilities; and other plugins support specific connectors/indexers for various external platforms.
• BOM and Parent Modules (lucille-bom, lucille-parent) manage versioning and dependency aggregation across modules.
External System & Deployment Components
• Data Sources: External data (e.g., files, CSVs, JSONl, databases) from which data is extracted.
• Search Engines: External targets such as Apache Solr, Elasticsearch/OpenSearch, and vector search engines (Pinecone, Weaviate) to which the transformed "Documents" are ultimately loaded.
• Distributed Infrastructure: The provided examples (e.g., distributed examples in lucille-examples) include Docker Compose, Kafka, Zookeeper, and worker setups, which illustrate distributed and production deployments.
Relationships and Data Flow
• Documents then pass through the Pipeline (a configurable sequence of stages), where transformation, enrichment, error handling, and data validation occur.
• The processed Documents flow into the Indexing component, which then sends data to the appropriate external search engine.
• Additionally, the Messaging layer (using Kafka integration and other local/in-memory messengers) helps coordinate processing—especially in distributed deployments.
• A separate API layer can be exposed by one of the plugins (lucille-api) to monitor, control, or reconfigure the pipeline at runtime.
– One layer for input (Connectors),
– a core processing layer (Pipelines, Stages, Workers),
– an output layer (Indexers),
– and an external integration layer (Search Engines, message queues, APIs).
• Integrability and extensibility points – for example, plugins can easily be shown as both augmenting core processing and providing alternative paths for handling additional use cases (OCR, parquet files, etc.).