How sajdoann/inverted_amigos_stage3 works
Project type and purpose
• This is a distributed search engine built in Java, packaged as a multi-module Maven project and containerized with Docker.
• It ingests a corpus of Gutenberg books, builds a distributed inverted index (via Hazelcast), and exposes a REST API to serve search queries. NGINX routes incoming queries to the appropriate query node.
File structure
• Top-level aggregator (root pom.xml) with four submodules:
• Crawler – reads raw books, tokenizes them, and feeds data into the indexing subsystem.
• InvertedIndex – produces and stores the inverted index, using a port/adapter pattern (ports for Tokenizer and InvertedIndex, implementations for GutenbergTokenizer and file-per-word storage).
• QueryEngine – provides search logic against the Hazelcast-distributed index, implements SearchEngineInterface and returns ranked results.
• API – Spring Boot app exposing HTTP endpoints (SearchEngineController) that wrap calls to the query engine.
• compose.yaml (and individual Dockerfiles) orchestrate all modules plus an NGINX container.
• There’s a ‘gutenberg_books’ folder and precomputed ‘trie_index_by_prefix’ files for prefix querying.
• README hints at hazardcast for index distribution and NGINX for request routing.
README for additional insights
• Hazelcast clusters hold index shards on each node.
• NGINX load-balances queries among query engine containers.
• No single monolithic database; index data lives in-memory (Hazelcast) and on disk via the InvertedIndex module.
• Data Ingestion / Crawler Service
• Inverted Index Service
• Hazelcast Cluster (distributed in-memory grid)
• Query Engine Service (multiple replicas)
• API Gateway / Spring Boot Service
• NGINX Load Balancer
• External Client (Browser / CLI)
• File storage (gutenberg_books, stopwords, msgpack tries)
1. Crawler → Inverted Index Service (raw book contents)
2. Inverted Index Service ↔ Hazelcast Cluster (writes index shards, reads on startup)
3. External Client → NGINX → API Service → QueryEngine instance → Hazelcast Cluster (read) → API Service → Client
4. (Optional) Inverted Index Service → Persistent Storage (msgpack files for trie index)
c. Architectural patterns & design principles:
• Microservices / modular Maven subprojects
• Ports & Adapters / hexagonal architecture in InvertedIndex
• Distributed in-memory data grid (Hazelcast) for scale and resilience
• Separation of concerns: crawler, indexer, search engine, and HTTP API layer
• Containerization & orchestration (Docker Compose)
d. Key technologies & frameworks:
• Java 11+ with Maven
• Spring Boot (API module)
• Hazelcast (clustering / distributed map)
• NGINX (reverse proxy & load balancer)
• Msgpack-based prefix tries (for fast prefix querying)
• Docker / Docker Compose