# anupanupranjan-gif/search-catalog-indexer architecture

> Architecture diagram of the GitHub repository anupanupranjan-gif/search-catalog-indexer, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2026-03-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/anupanupranjan-gif/search-catalog-indexer
- Repository: https://github.com/anupanupranjan-gif/search-catalog-indexer

## Overview

##### What this project is (type + purpose)

This repo is a **single-purpose Spring Boot + Spring Batch “indexer” service** designed to run as a **one-shot batch job** (locally, Docker, or Kubernetes Job). Its purpose:

##### Transform raw rows into a clean “indexable” product document (plus suggest payload).
4. **Generate semantic embeddings** using **DJL** with model **all-MiniLM-L6-v2** (384-dim vector).

##### Repo structure into architectural components

Use the code layout to drive your component boundaries:

##### Entry point / Application wrapper
- `CatalogIndexerApplication.java`
- Spring Boot app bootstrap; likely triggers Spring Batch job execution.

##### Batch job orchestration & lifecycle
- `config/BatchConfig.java`
- Defines Job, Steps, chunking, reader/processor/writer wiring.
- `config/JobListeners.java`
- Start/end hooks, metrics/logging, failure handling.

##### External system connectivity/config
- `config/ElasticsearchConfig.java`
- ES client setup; index target; auth; host/port.
- `config/EmbeddingConfig.java`
- DJL model loading; translator; caching; maybe thread pools.

##### Pipeline stages (the core “data plane”)
- **Reader**
- `reader/HuggingFaceParquetReader.java`
- **Transformer / Processor**
- `processor/ProductTransformer.java` (data cleaning + suggest payload)
- `processor/EmbeddingProcessor.java` (calls embedding model)
- `embedding/SentenceEmbeddingTranslator.java` (DJL translation layer)
- **Writer**
- `writer/ElasticsearchBulkWriter.java` (bulk API; chunk size)

##### Data contracts / domain objects
- `model/AmazonProduct.java` (raw-ish dataset row)
- `model/ProductDocument.java` (final ES document including vector)

##### Runtime/deploy artifacts (important for “system context”)
- `src/main/resources/application.yml` (runtime config + env overrides)
- `Dockerfile` (container runtime)
- `k8s/indexer-job.yml` (Kubernetes Job execution)
- `Makefile` (operational entrypoints: run, build, push, verify)
- `pom.xml` (tech stack dependencies; Java 25; Spring Batch; DJL; ES client)

##### Control plane (who starts what)
- **Operator / CI / Dev** runs:
- `make run` / `make run-small` (local)
- or deploys **Kubernetes Job** (k8s manifest)
- This launches the container / JVM which starts Spring Boot, which starts **Spring Batch Job**.

##### Data plane (the actual pipeline)
The README already gives the canonical pipeline; mirror it visually:

**Hugging Face Dataset (Parquet)**
→ download/cache to local filesystem (`DATASET_PATH`, `SKIP_DOWNLOAD`)
→ `HuggingFaceParquetReader` (row-by-row)
→ `ProductTransformer` (clean + suggest)
→ `EmbeddingProcessor` (calls DJL model)
→ `ElasticsearchBulkWriter` (bulk index CHUNK_SIZE)
→ **Elasticsearch index** (`ES_INDEX`)

- **Hugging Face** (dataset source over HTTP)
- **DJL model zoo / model download** (over HTTP) + **local model cache**
- **Local ephemeral storage** (Parquet cache + model cache; in k8s likely emptyDir unless mounted)
- **Elasticsearch cluster** (bulk indexing API)

##### Container/Runtime View (C4 Level 2)
- Shows: Kubernetes Job → Pod → container (Spring Boot app) → network calls to HF/ES; volumes/caches.
##### Component / Batch Pipeline View (C4 Level 3)
- Shows the Spring Batch internals: Job → Step → Reader/Processor/Writer and domain objects.

This repo is small enough that Level 3 can be very concrete and map 1:1 to class names.

**“search-catalog-indexer (Spring Boot + Spring Batch)”**
- With:
- “Chunk-oriented step (chunk size = CHUNK_SIZE)”
- “MAX_RECORDS limit”
- “JobListeners (logging/metrics)”

**HuggingFaceParquetReader**
- Inputs: local parquet path (or triggers download)
- Output: `AmazonProduct`

**ProductTransformer**
- Input: `AmazonProduct`
- Output: `ProductDocument` (partial; no vector yet)

**EmbeddingProcessor**
- calls DJL Predictor/Model
- uses `SentenceEmbeddingTranslator`
- Output: `ProductDocument` with `embedding[384]`

**ElasticsearchBulkWriter**
- uses ES Bulk API
- “500 docs/request by default”
- Output: indexed docs in `products` index (or env-configured)

##### External systems
- **Hugging Face datasets**: `milistu/AMAZON-Products-2023`
- **DJL Model Zoo**: `all-MiniLM-L6-v2`
- **Elasticsearch** (host/port/auth)
- **Filesystem / Cache** (Parquet file + model artifacts)

- **Docker image** (from `Dockerfile`)
- **Kubernetes Job** (from `k8s/indexer-job.yml`)

##### Important architectural patterns/principles present (call these out explicitly)

- **Batch ETL pipeline** (Extract from HF → Transform/Enrich → Load into ES).
- **Spring Batch chunk-oriented processing**:
- read/process/write in chunks to control memory and optimize bulk indexing throughput.
- **Separation of concerns**:
- Reader/Processor/Writer classes clearly partitioned.
- **External artifact caching**:
- dataset and model downloads cached locally; in Kubernetes this implies volume strategy.

- Every major package in `com.search.indexer.*` appears as a component or annotation:
- `config/*` shows orchestration/config
- `reader/processor/writer` show pipeline
- `embedding/*` shows ML integration detail
- `model/*` shows data contracts
- Every operational artifact is represented:
- `Dockerfile` → container
- `k8s/indexer-job.yml` → one-shot job execution
- `application.yml` → env-configurable runtime
- External systems are explicit:
- Hugging Face dataset
- DJL model source + local cache
- Elasticsearch
- Controls/limits are labeled:
- `MAX_RECORDS`, `CHUNK_SIZE`, `ES_*`, `SKIP_DOWNLOAD`, `DATASET_PATH`

## Diagram (Mermaid)

```mermaid
flowchart TD
  %% =========================
  %% View 1: System Context (C4 L1)
  %% =========================
  subgraph "System Context (C4 L1)"
    direction TB

    Operator["Operator / CI / Dev"]:::actor
    Make["Local execution (Make targets)"]:::ops
    K8sJob["Kubernetes Job (one-shot)"]:::deploy

    AppBlackBox["search-catalog-indexer (Spring Boot + Spring Batch)\nOne-shot batch indexer job"]:::app

    HF["Hugging Face Datasets\nmilistu/AMAZON-Products-2023 (Parquet)"]:::external
    ES[("Elasticsearch Cluster\nIndex: ES_INDEX")]:::externaldb
    DJLZoo["DJL Model Zoo\nModel: all-MiniLM-L6-v2"]:::external

    Operator -->|"runs"| Make
    Operator -->|"deploys"| K8sJob

    Make -->|"starts"| AppBlackBox
    K8sJob -->|"starts"| AppBlackBox

    AppBlackBox -->|"HTTP(S) download"| HF
    AppBlackBox -->|"HTTP(S) model download"| DJLZoo
    AppBlackBox -->|"Bulk API"| ES
  end

  %% =========================
  %% View 2: Container / Runtime (C4 L2)
  %% =========================
  subgraph "Container / Runtime View (C4 L2)"
    direction TB

    DockerImage["Docker Image"]:::deploy
    Pod["Kubernetes Pod"]:::deploy
    Container["Container: JVM\nSpring Boot app"]:::deploy

    AppYml["Runtime config (application.yml)\nES_*, DATASET_PATH, SKIP_DOWNLOAD,\nCHUNK_SIZE=500, MAX_RECORDS=50K"]:::config

    FSParquet[["Local FS cache\nDataset Parquet (DATASET_PATH)"]]:::storage
    FSModel[["Local FS cache\nDJL model artifacts"]]:::storage
    NoteEph["Note: In Kubernetes, caches are ephemeral unless a volume is mounted"]:::note

    DockerImage -->|"runs as"| Container
    Pod -->|"hosts"| Container
    K8sJob -->|"creates"| Pod

    Container -.->|"loads"| AppYml

    Container -->|"File I/O"| FSParquet
    Container -->|"File I/O"| FSModel
    FSParquet -.-> NoteEph
    FSModel -.-> NoteEph

    HF -->|"HTTP(S) -> cached file"| FSParquet
    DJLZoo -->|"HTTP(S) -> cached artifacts"| FSModel
    Container -->|"Bulk API"| ES
  end

  %% =========================
  %% View 3: Component / Batch Pipeline (C4 L3)
  %% =========================
  subgraph "Spring Batch Component / Pipeline View (C4 L3)"
    direction TB

    Boot["CatalogIndexerApplication\n(Spring Boot bootstrap)"]:::app
    BatchCfg["BatchConfig\nJob/Step/chunk wiring"]:::config
    Listeners["JobListeners\nstart/end hooks\nlogging/metrics/failure handling"]:::config

    Step["Chunk-oriented Step\nCHUNK_SIZE=500\nMAX_RECORDS configurable"]:::batch

    Reader["HuggingFaceParquetReader\nreads cached Parquet\n(optionally downloads)"]:::component
    RawModel["AmazonProduct\n(raw dataset row)"]:::model

    Transformer["ProductTransformer\nclean + suggest payload"]:::component
    DocModel1["ProductDocument\n(partial; no vector yet)"]:::model

    EmbCfg["EmbeddingConfig\nDJL model loading + predictor"]:::config
    EmbedProc["EmbeddingProcessor\nadds embedding[384]"]:::component
    Translator["SentenceEmbeddingTranslator\nDJL translation layer"]:::component
    DocModel2["ProductDocument\n(final; embedding[384])"]:::model

    ESCfg["ElasticsearchConfig\nES client, auth, host/port, index"]:::config
    Writer["ElasticsearchBulkWriter\nBulk API write\n500 docs/request default"]:::component

    Boot -->|"starts"| BatchCfg
    BatchCfg -->|"registers"| Step
    BatchCfg -.->|"attaches"| Listeners

    Step -->|"read()"| Reader
    Reader -->|"emits"| RawModel
    RawModel -->|"process()"| Transformer
    Transformer -->|"outputs"| DocModel1
    DocModel1 -->|"process()"| EmbedProc
    EmbedProc -->|"outputs"| DocModel2
    DocModel2 -->|"write()"| Writer
    Writer -->|"bulk index"| ES

    %% Dependencies / integrations (dashed)
    Reader -.->|"uses cache"| FSParquet
    Reader -.->|"may download"| HF

    EmbedProc -.->|"depends on"| EmbCfg
    EmbedProc -.->|"uses"| Translator
    EmbCfg -.->|"loads model from"| FSModel
    EmbCfg -.->|"may download model"| DJLZoo

    Writer -.->|"depends on"| ESCfg
    ESCfg -.->|"connects to"| ES
  end

  %% =========================
  %% Operational artifacts
  %% =========================
  subgraph "Build / Ops Artifacts"
    direction TB
    Pom["Build config (pom.xml)\nSpring Boot/Batch, DJL, ES client"]:::ops
    MakefileNode["Makefile\nrun/build/push/verify"]:::ops
    DockerfileNode["Dockerfile\ncontainer build"]:::ops
    K8sManifest["k8s/indexer-job.yml\nKubernetes Job manifest"]:::ops

    Pom -.->|"builds"| DockerImage
    DockerfileNode -.->|"builds"| DockerImage
    MakefileNode -.->|"invokes"| DockerfileNode
    MakefileNode -.->|"invokes"| Pom
    K8sManifest -.->|"defines"| K8sJob
  end

  %% =========================
  %% Click Events (component mapping)
  %% =========================
  click Boot "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/CatalogIndexerApplication.java"
  click BatchCfg "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/config/BatchConfig.java"
  click Listeners "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/config/JobListeners.java"
  click ESCfg "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/config/ElasticsearchConfig.java"
  click EmbCfg "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/config/EmbeddingConfig.java"
  click Reader "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/reader/HuggingFaceParquetReader.java"
  click Transformer "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/processor/ProductTransformer.java"
  click EmbedProc "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/processor/EmbeddingProcessor.java"
  click Translator "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/embedding/SentenceEmbeddingTranslator.java"
  click Writer "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/writer/ElasticsearchBulkWriter.java"
  click RawModel "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/model/AmazonProduct.java"
  click DocModel2 "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/java/com/search/indexer/model/ProductDocument.java"
  click AppYml "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/src/main/resources/application.yml"
  click DockerfileNode "https://github.com/anupanupranjan-gif/search-catalog-indexer/tree/main/Dockerfile"
  click K8sManifest "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/k8s/indexer-job.yml"
  click MakefileNode "https://github.com/anupanupranjan-gif/search-catalog-indexer/tree/main/Makefile"
  click Pom "https://github.com/anupanupranjan-gif/search-catalog-indexer/blob/main/pom.xml"

  %% =========================
  %% Styles
  %% =========================
  classDef app fill:#1f77b4,stroke:#0b3d91,color:#ffffff,stroke-width:1px
  classDef batch fill:#2b6cb0,stroke:#1a365d,color:#ffffff,stroke-width:1px
  classDef component fill:#4c9aff,stroke:#1a4b8c,color:#ffffff,stroke-width:1px
  classDef model fill:#a7c7ff,stroke:#2a5aa8,color:#0b1b33,stroke-width:1px
  classDef config fill:#1e88e5,stroke:#0d47a1,color:#ffffff,stroke-width:1px
  classDef external fill:#f59e0b,stroke:#b45309,color:#1f1300,stroke-width:1px
  classDef externaldb fill:#fb923c,stroke:#c2410c,color:#1f1300,stroke-width:1px
  classDef storage fill:#9ca3af,stroke:#374151,color:#111827,stroke-width:1px
  classDef deploy fill:#7c3aed,stroke:#4c1d95,color:#ffffff,stroke-width:1px
  classDef ops fill:#6b7280,stroke:#111827,color:#ffffff,stroke-width:1px
  classDef actor fill:#0ea5e9,stroke:#075985,color:#ffffff,stroke-width:1px
  classDef note fill:#f3f4f6,stroke:#9ca3af,color:#111827,stroke-dasharray: 4 3
```
