How qawsed123144/elasticsearch_vector_db works
Project type and purpose
• This is a lightweight open-source demo/application for building a Chinese vector search system using Elasticsearch 8.x.
• It focuses on two primary flows:
a. Data ingestion & indexing (embed.py)
b. Semantic search via CLI (search.py)
File structure
• Top level files:
• docker-compose.yml → orchestrates Elasticsearch container
• embed.py → Python script for embedding raw data and bulk-import into ES
• search.py → CLI tool that embeds a query and performs a k-NN search
• No frontend; all components are backend/CLI.
• Dependencies: Python (Hugging Face model), Elasticsearch REST API, Docker.
Key insights from README
• Embedding model: BAAI/bge-large-zh-v1.5 (via Hugging Face)
• Elasticsearch index with dense_vector & HNSW for k-NN
• Two-step pipeline: embed & index → vector store; embed query → k-NN search
• Uses Docker-Compose to launch ES 8.5.0
Main components & how to represent them
a. Raw Data Source
b. Embedding Module (embed.py & model)
• Sub-components:
1. Python Embed Script
2. Hugging Face Inference (BAAI/bge-large-zh-v1.5)
c. Vector Database
• Elasticsearch cluster (Docker container)
d. CLI Search Tool (search.py)
e. User / Terminal
Relationships & data flows
• Data Ingestion Flow (A → B → C):
1. Raw Data → embed.py → calls Hugging Face API → obtains embeddings
2. embed.py → Elasticsearch REST Bulk API → indexes vectors into ES
• Query Flow (D → B → A):
1. User enters query in CLI → search.py
2. search.py → calls embedding model → query vector
3. search.py → ES k-NN Search API → returns nearest neighbors
4. search.py → displays results in terminal
Architectural patterns & design principles
• Separation of concerns: embedding vs. indexing vs. search
• Pipeline pattern: transformation of data through discrete stages
• Stateless CLI tools interacting over HTTP with ES
• Extensibility: swap out model, support other embedding scripts or UIs
Technologies & frameworks
• Python 3.x (scripts)
• Hugging Face Transformers (inference)
• Elasticsearch 8.x (dense_vector, HNSW)
• Docker-Compose (deployment)
• RESTful APIs for indexing/search