# sajdoann/inverted_amigos_stage3 architecture

> Architecture diagram of the GitHub repository sajdoann/inverted_amigos_stage3, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-13). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sajdoann/inverted_amigos_stage3
- Repository: https://github.com/sajdoann/inverted_amigos_stage3

## Overview

##### Project type and purpose
- This is a distributed search engine built in Java, packaged as a multi-module Maven project and containerized with Docker.
- It ingests a corpus of Gutenberg books, builds a distributed inverted index (via Hazelcast), and exposes a REST API to serve search queries. NGINX routes incoming queries to the appropriate query node.

##### File structure
- Top-level aggregator (root pom.xml) with four submodules:
- Crawler – reads raw books, tokenizes them, and feeds data into the indexing subsystem.
- InvertedIndex – produces and stores the inverted index, using a port/adapter pattern (ports for Tokenizer and InvertedIndex, implementations for GutenbergTokenizer and file-per-word storage).
- QueryEngine – provides search logic against the Hazelcast-distributed index, implements SearchEngineInterface and returns ranked results.
- API – Spring Boot app exposing HTTP endpoints (SearchEngineController) that wrap calls to the query engine.
- compose.yaml (and individual Dockerfiles) orchestrate all modules plus an NGINX container.
- There’s a ‘gutenberg_books’ folder and precomputed ‘trie_index_by_prefix’ files for prefix querying.
- README hints at hazardcast for index distribution and NGINX for request routing.

##### README for additional insights
- Hazelcast clusters hold index shards on each node.
- NGINX load-balances queries among query engine containers.
- No single monolithic database; index data lives in-memory (Hazelcast) and on disk via the InvertedIndex module.

- Data Ingestion / Crawler Service
- Inverted Index Service
- Hazelcast Cluster (distributed in-memory grid)
- Query Engine Service (multiple replicas)
- API Gateway / Spring Boot Service
- NGINX Load Balancer
- External Client (Browser / CLI)
- File storage (gutenberg_books, stopwords, msgpack tries)
1. Crawler → Inverted Index Service (raw book contents)
2. Inverted Index Service ↔ Hazelcast Cluster (writes index shards, reads on startup)
3. External Client → NGINX → API Service → QueryEngine instance → Hazelcast Cluster (read) → API Service → Client
4. (Optional) Inverted Index Service → Persistent Storage (msgpack files for trie index)
c. Architectural patterns & design principles:
- Microservices / modular Maven subprojects
- Ports & Adapters / hexagonal architecture in InvertedIndex
- Distributed in-memory data grid (Hazelcast) for scale and resilience
- Separation of concerns: crawler, indexer, search engine, and HTTP API layer
- Containerization & orchestration (Docker Compose)
d. Key technologies & frameworks:
- Java 11+ with Maven
- Spring Boot (API module)
- Hazelcast (clustering / distributed map)
- NGINX (reverse proxy & load balancer)
- Msgpack-based prefix tries (for fast prefix querying)
- Docker / Docker Compose

## Diagram (Mermaid)

```mermaid
flowchart TD
    Client["External Client"]:::edge
    Client -->|"HTTP"| NGINX["NGINX Load Balancer"]:::edge
    NGINX -->|"HTTP"| API["API Gateway Service"]:::query
    API -->|"HTTP"| QueryEngine1["QueryEngine Instance"]:::query
    QueryEngine1 -->|"Hazelcast IMap read/write"| Hazelcast["Hazelcast Cluster"]:::datastore
    Crawler["Crawler Service"]:::ingestion -->|"raw book contents"| InvertedIndex["Inverted Index Service"]:::ingestion
    InvertedIndex -->|"Hazelcast IMap write/read"| Hazelcast
    InvertedIndex -->|"file I/O"| Storage[(Persistent File Storage)]:::storage

    subgraph "Docker Compose" 
        direction TB
        Crawler
        InvertedIndex
        API
        QueryEngine1
        Hazelcast
    end

    Storage[(Persistent File Storage)]:::storage

    %% Click Events for Crawler Service
    click Crawler "https://github.com/sajdoann/inverted_amigos_stage3/tree/main/Crawler/"
    click MainCrawler "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/Crawler/src/main/java/org/ulpgc/crawler/MainCrawler.java"
    click CrawlerThread "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/Crawler/src/main/java/org/ulpgc/crawler/CrawlerThread.java"
    click HazelCastCrawler "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/Crawler/src/main/java/org/ulpgc/crawler/HazelCastCrawler.java"
    click ICrawler "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/Crawler/src/main/java/org/ulpgc/crawler/ICrawler.java"
    click CrawlerDockerfile "https://github.com/sajdoann/inverted_amigos_stage3/tree/main/Crawler/Dockerfile"

    %% Click Events for Inverted Index Service
    click InvertedIndex "https://github.com/sajdoann/inverted_amigos_stage3/tree/main/InvertedIndex/"
    click FilePerWordInvertedIndexHazelcast "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/InvertedIndex/src/main/java/org/ulpgc/inverted_index/apps/FilePerWordInvertedIndexHazelcast.java"
    click InvertedIndexPort "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/InvertedIndex/src/main/java/org/ulpgc/inverted_index/ports/InvertedIndex.java"
    click TokenizerPort "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/InvertedIndex/src/main/java/org/ulpgc/inverted_index/ports/Tokenizer.java"
    click GutenbergTokenizer "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/InvertedIndex/src/main/java/org/ulpgc/inverted_index/implementations/GutenbergTokenizer.java"
    click Stopwords "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/InvertedIndex/src/main/resources/stopwords.txt"
    click InvertedIndexDockerfile "https://github.com/sajdoann/inverted_amigos_stage3/tree/main/InvertedIndex/Dockerfile"

    %% Click Events for Query Engine Service
    click QueryEngine1 "https://github.com/sajdoann/inverted_amigos_stage3/tree/main/QueryEngine/"
    click MainQueryEngine "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/QueryEngine/src/main/java/org/ulpgc/query_engine/MainQueryEngine.java"
    click HazelQueryEngine "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/QueryEngine/src/main/java/org/ulpgc/query_engine/HazelQueryEngine.java"
    click SearchEngineInterface "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/QueryEngine/src/main/java/org/ulpgc/query_engine/SearchEngineInterface.java"
    click SearchResult "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/QueryEngine/src/main/java/org/ulpgc/query_engine/SearchResult.java"
    click SearchResultsMW "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/QueryEngine/src/main/java/org/ulpgc/query_engine/SearchResultsMW.java"
    click MultipleWordsResponseList "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/QueryEngine/src/main/java/org/ulpgc/query_engine/MultipleWordsResponseList.java"
    click QueryEngineDockerfile "https://github.com/sajdoann/inverted_amigos_stage3/tree/main/QueryEngine/Dockerfile"

    %% Click Events for API Gateway / Spring Boot Service
    click API "https://github.com/sajdoann/inverted_amigos_stage3/tree/main/API/"
    click ApiApplication "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/API/src/main/java/org/ulpgc/api/ApiApplication.java"
    click SearchEngineController "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/API/src/main/java/org/ulpgc/api/controller/SearchEngineController.java"
    click SearchEngineControllerInterface "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/API/src/main/java/org/ulpgc/api/controller/SearchEngineControllerInterface.java"
    click ApplicationProperties "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/API/src/main/resources/application.properties"
    click APIDockerfile "https://github.com/sajdoann/inverted_amigos_stage3/tree/main/API/Dockerfile"

    %% Click Events for NGINX Load Balancer & Compose
    click Compose "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/compose.yaml"

    %% Click Events for Persistent File Storage
    click GutenbergBooks "https://github.com/sajdoann/inverted_amigos_stage3/tree/main/gutenberg_books/"
    click TrieIndex "https://github.com/sajdoann/inverted_amigos_stage3/tree/main/trie_index_by_prefix/"

    %% Click Events for Root Maven Aggregator
    click RootPom "https://github.com/sajdoann/inverted_amigos_stage3/blob/main/pom.xml"

    classDef ingestion fill:#cce5ff,stroke:#333,stroke-width:1px
    classDef datastore fill:#d4edda,stroke:#333,stroke-width:1px
    classDef query fill:#ffe5cc,stroke:#333,stroke-width:1px
    classDef edge fill:#e2d6f9,stroke:#333,stroke-width:1px
    classDef storage fill:#f0f0f0,stroke:#333,stroke-width:1px
```
