How anupanupranjan-gif/search-catalog-indexer works
What this project is (type + purpose)
This repo is a single-purpose Spring Boot + Spring Batch “indexer” service designed to run as a one-shot batch job (locally, Docker, or Kubernetes Job). Its purpose:
Transform raw rows into a clean “indexable” product document (plus suggest payload).
4. Generate semantic embeddings using DJL with model all-MiniLM-L6-v2 (384-dim vector).
Repo structure into architectural components
Use the code layout to drive your component boundaries:
Entry point / Application wrapper
• CatalogIndexerApplication.java
• Spring Boot app bootstrap; likely triggers Spring Batch job execution.
Batch job orchestration & lifecycle
• config/BatchConfig.java
• Defines Job, Steps, chunking, reader/processor/writer wiring.
• config/JobListeners.java
• Start/end hooks, metrics/logging, failure handling.
External system connectivity/config
• config/ElasticsearchConfig.java
• ES client setup; index target; auth; host/port.
• config/EmbeddingConfig.java
• DJL model loading; translator; caching; maybe thread pools.
Pipeline stages (the core “data plane”)
• Reader
• reader/HuggingFaceParquetReader.java
• Transformer / Processor
• processor/ProductTransformer.java (data cleaning + suggest payload)
• processor/EmbeddingProcessor.java (calls embedding model)
• embedding/SentenceEmbeddingTranslator.java (DJL translation layer)
• Writer
• writer/ElasticsearchBulkWriter.java (bulk API; chunk size)
Data contracts / domain objects
• model/AmazonProduct.java (raw-ish dataset row)
• model/ProductDocument.java (final ES document including vector)
Runtime/deploy artifacts (important for “system context”)
• src/main/resources/application.yml (runtime config + env overrides)
• Dockerfile (container runtime)
• k8s/indexer-job.yml (Kubernetes Job execution)
• Makefile (operational entrypoints: run, build, push, verify)
• pom.xml (tech stack dependencies; Java 25; Spring Batch; DJL; ES client)
Control plane (who starts what)
• Operator / CI / Dev runs:
• make run / make run-small (local)
• or deploys Kubernetes Job (k8s manifest)
• This launches the container / JVM which starts Spring Boot, which starts Spring Batch Job.
Data plane (the actual pipeline)
The README already gives the canonical pipeline; mirror it visually:
Hugging Face Dataset (Parquet)
→ download/cache to local filesystem (DATASET_PATH, SKIP_DOWNLOAD)
→ HuggingFaceParquetReader (row-by-row)
→ ProductTransformer (clean + suggest)
→ EmbeddingProcessor (calls DJL model)
→ ElasticsearchBulkWriter (bulk index CHUNK_SIZE)
→ Elasticsearch index (ES_INDEX)
• Hugging Face (dataset source over HTTP)
• DJL model zoo / model download (over HTTP) + local model cache
• Local ephemeral storage (Parquet cache + model cache; in k8s likely emptyDir unless mounted)
• Elasticsearch cluster (bulk indexing API)
Container/Runtime View (C4 Level 2)
• Shows: Kubernetes Job → Pod → container (Spring Boot app) → network calls to HF/ES; volumes/caches.
Component / Batch Pipeline View (C4 Level 3)
• Shows the Spring Batch internals: Job → Step → Reader/Processor/Writer and domain objects.
This repo is small enough that Level 3 can be very concrete and map 1:1 to class names.
“search-catalog-indexer (Spring Boot + Spring Batch)”
• With:
• “Chunk-oriented step (chunk size = CHUNK_SIZE)”
• “MAX_RECORDS limit”
• “JobListeners (logging/metrics)”
HuggingFaceParquetReader
• Inputs: local parquet path (or triggers download)
• Output: AmazonProduct
ProductTransformer
• Input: AmazonProduct
• Output: ProductDocument (partial; no vector yet)
EmbeddingProcessor
• calls DJL Predictor/Model
• uses SentenceEmbeddingTranslator
• Output: ProductDocument with embedding[384]
ElasticsearchBulkWriter
• uses ES Bulk API
• “500 docs/request by default”
• Output: indexed docs in products index (or env-configured)
External systems
• Hugging Face datasets: milistu/AMAZON-Products-2023
• DJL Model Zoo: all-MiniLM-L6-v2
• Elasticsearch (host/port/auth)
• Filesystem / Cache (Parquet file + model artifacts)
• Docker image (from Dockerfile)
• Kubernetes Job (from k8s/indexer-job.yml)
Important architectural patterns/principles present (call these out explicitly)
• Batch ETL pipeline (Extract from HF → Transform/Enrich → Load into ES).
• Spring Batch chunk-oriented processing:
• read/process/write in chunks to control memory and optimize bulk indexing throughput.
• Separation of concerns:
• Reader/Processor/Writer classes clearly partitioned.
• External artifact caching:
• dataset and model downloads cached locally; in Kubernetes this implies volume strategy.
• Every major package in com.search.indexer.* appears as a component or annotation:
• config/* shows orchestration/config
• reader/processor/writer show pipeline
• embedding/* shows ML integration detail
• model/* shows data contracts
• Every operational artifact is represented:
• Dockerfile → container
• k8s/indexer-job.yml → one-shot job execution
• application.yml → env-configurable runtime
• External systems are explicit:
• Hugging Face dataset
• DJL model source + local cache
• Elasticsearch
• Controls/limits are labeled:
• MAX_RECORDS, CHUNK_SIZE, ES_*, SKIP_DOWNLOAD, DATASET_PATH