How sagar8080/semantic-search-system works
Project Type and Purpose
• This is a full-stack, data-processing and search application.
• Purpose: Ingest press releases, run NLP and embedding pipelines, build a vector search index and a knowledge graph, and expose interactive UIs for search and taxonomy management.
File Structure
• Top-level scripts:
• ingest/… → fetch and store press-release metadata & raw text into OpenSearch
• nlp/… → perform NLP enrichment (entities, topics, summaries, embeddings)
• utils/… → wrappers for AWS Bedrock, OpenSearch, vector index setup, secrets, search pipelines
• knowledge_graph/… → build and populate a Neo4j graph from topic-mapping JSONs
• human_feedback/… → “human-in-the-loop” Streamlit interface for reviewing topics
• main_app.py → primary Streamlit search/chat UI
• explorer_app.py → Streamlit taxonomy review & graph viewer
• public/ → static assets (JS, CSS, theme) for UI
• Data and config:
• topics/*.json → topic definitions & mappings
• url_paths/press_releases.json → initial seed data format
• .chainlit/ → chat configuration and translations
Key Insights from README
• Two primary databases:
• OpenSearch with three indices:
– PR_META_URL_IDX (to-do URL list)
– PR_META_RAW_IDX (raw cleaned text)
– PR_META_VECTOR_IDX (embeddings + enriched metadata)
• Neo4j knowledge graph storing BroadTopic and Document nodes plus RELATES_TO_DOC edges
• AWS Bedrock services used for embeddings, entity/topic extraction, summarization, and (optionally) reranking or query expansion
• Streamlit UIs for: end-user search, taxonomy review, graph visualization, and human feedback
Components & Interactions
a. Main Components
1. Web Crawler & Metadata Fetcher (pr_meta_fetch.py)
2. Metadata Stager (pr_meta_store_from_local.py)
3. Raw Content Fetcher (pr_meta_store.py)
4. NLP/Embedding Pipeline (pr_aws_nlp.py, pr_local_nlp_pipeline.py, topic_modeling.py)
5. Vector Index Setup & Loader (create_vector_index.py + enrichment scripts)
6. Knowledge Graph Builder (knowledge_graph.py)
7. Search Service (search_service.py + search_pipeline.py)
8. Human-in-the-loop UI (human_feedback.py)
9. Streamlit UIs (main_app.py, explorer_app.py)
10. Data Stores: OpenSearch, Neo4j
11. External AI: AWS Bedrock (LLM models, embedding models)
b. Relationships & Data Flows
• Crawler → PR_META_URL_IDX
• PR_META_URL_IDX → Raw Content Fetcher → PR_META_RAW_IDX
• PR_META_RAW_IDX → NLP/Embedding Pipeline → PR_META_VECTOR_IDX
• PR_META_VECTOR_IDX → Search Service → Streamlit Search UI
• topics/*.json + PR_META_VECTOR_IDX → Knowledge Graph Builder → Neo4j
• Neo4j ↔ Explorer UI (graph viewer & taxonomy review)
• Human Feedback UI → topics/*.json → Knowledge Graph Builder (loop)
• Throughout: utils modules mediate AWS Bedrock calls and OpenSearch queries
c. Architectural Patterns & Principles
• Pipeline pattern: sequential ingestion → enrichment → indexing
• Polyglot persistence: OpenSearch for full-text & vector search, Neo4j for graph data
• Human-in-the-loop for model validation
• Modular scripts as micro-services
• Separation of concerns: data ingestion, NLP, storage, UI kept in distinct layers
d. Key Technologies & Frameworks
• Python scripts & modules
• Streamlit for UI
• AWS Bedrock for embeddings, entity extraction, summarization, reranking
• OpenSearch with k-NN plugin
• Neo4j for knowledge graph
• JSON files for topic mappings
• Docker/Env? (implied by utils/get_secrets)