# sagar8080/semantic-search-system architecture

> Architecture diagram of the GitHub repository sagar8080/semantic-search-system, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-08-28). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/sagar8080/semantic-search-system
- Repository: https://github.com/sagar8080/semantic-search-system

## Overview

##### Project Type and Purpose
- This is a full-stack, data-processing and search application.
- Purpose: Ingest press releases, run NLP and embedding pipelines, build a vector search index and a knowledge graph, and expose interactive UIs for search and taxonomy management.

##### File Structure
- Top-level scripts:
- ingest/…               → fetch and store press-release metadata & raw text into OpenSearch
- nlp/…                  → perform NLP enrichment (entities, topics, summaries, embeddings)
- utils/…                → wrappers for AWS Bedrock, OpenSearch, vector index setup, secrets, search pipelines
- knowledge_graph/…     → build and populate a Neo4j graph from topic-mapping JSONs
- human_feedback/…      → “human-in-the-loop” Streamlit interface for reviewing topics
- main_app.py            → primary Streamlit search/chat UI
- explorer_app.py        → Streamlit taxonomy review & graph viewer
- public/                → static assets (JS, CSS, theme) for UI
- Data and config:
- topics/*.json          → topic definitions & mappings
- url_paths/press_releases.json → initial seed data format
- .chainlit/             → chat configuration and translations

##### Key Insights from README
- Two primary databases:
- OpenSearch with three indices:
– PR_META_URL_IDX (to-do URL list)
– PR_META_RAW_IDX (raw cleaned text)
– PR_META_VECTOR_IDX (embeddings + enriched metadata)
- Neo4j knowledge graph storing BroadTopic and Document nodes plus RELATES_TO_DOC edges
- AWS Bedrock services used for embeddings, entity/topic extraction, summarization, and (optionally) reranking or query expansion
- Streamlit UIs for: end-user search, taxonomy review, graph visualization, and human feedback

##### Components & Interactions
a. Main Components
1. Web Crawler & Metadata Fetcher (pr_meta_fetch.py)
2. Metadata Stager (pr_meta_store_from_local.py)
3. Raw Content Fetcher (pr_meta_store.py)
4. NLP/Embedding Pipeline (pr_aws_nlp.py, pr_local_nlp_pipeline.py, topic_modeling.py)
5. Vector Index Setup & Loader (create_vector_index.py + enrichment scripts)
6. Knowledge Graph Builder (knowledge_graph.py)
7. Search Service (search_service.py + search_pipeline.py)
8. Human-in-the-loop UI (human_feedback.py)
9. Streamlit UIs (main_app.py, explorer_app.py)
10. Data Stores: OpenSearch, Neo4j
11. External AI: AWS Bedrock (LLM models, embedding models)
b. Relationships & Data Flows
- Crawler → PR_META_URL_IDX
- PR_META_URL_IDX → Raw Content Fetcher → PR_META_RAW_IDX
- PR_META_RAW_IDX → NLP/Embedding Pipeline → PR_META_VECTOR_IDX
- PR_META_VECTOR_IDX → Search Service → Streamlit Search UI
- topics/*.json + PR_META_VECTOR_IDX → Knowledge Graph Builder → Neo4j
- Neo4j ↔ Explorer UI (graph viewer & taxonomy review)
- Human Feedback UI → topics/*.json → Knowledge Graph Builder (loop)
- Throughout: utils modules mediate AWS Bedrock calls and OpenSearch queries
c. Architectural Patterns & Principles
- Pipeline pattern: sequential ingestion → enrichment → indexing
- Polyglot persistence: OpenSearch for full-text & vector search, Neo4j for graph data
- Human-in-the-loop for model validation
- Modular scripts as micro-services
- Separation of concerns: data ingestion, NLP, storage, UI kept in distinct layers
d. Key Technologies & Frameworks
- Python scripts & modules
- Streamlit for UI
- AWS Bedrock for embeddings, entity extraction, summarization, reranking
- OpenSearch with k-NN plugin
- Neo4j for knowledge graph
- JSON files for topic mappings
- Docker/Env? (implied by utils/get_secrets)

## Diagram (Mermaid)

```mermaid
graph TB
  %% UI Layer
  subgraph "UI Layer"
    HF["Human-in-the-loop Review UI"]:::ui
    MA["Streamlit Search Chat UI"]:::ui
    EA["Streamlit Taxonomy Review & Graph Viewer"]:::ui
    PA["Static Frontend Assets"]:::ui
    click HF "https://github.com/sagar8080/semantic-search-system/blob/main/human_feedback/human_feedback.py"
    click MA "https://github.com/sagar8080/semantic-search-system/blob/main/main_app.py"
    click EA "https://github.com/sagar8080/semantic-search-system/blob/main/explorer_app.py"
    click PA "https://github.com/sagar8080/semantic-search-system/tree/main/public/"
  end

  %% Backend Layer
  subgraph "Backend Layer"
    subgraph "Ingestion Services"
      CRAWLER["Web Crawler & Metadata Fetcher"]:::service
      STAGER["Metadata Stager"]:::service
      RAWFETCH["Raw Content Fetcher & Initial Storer"]:::service
      click CRAWLER "https://github.com/sagar8080/semantic-search-system/blob/main/ingest/pr_meta_fetch.py"
      click STAGER "https://github.com/sagar8080/semantic-search-system/blob/main/ingest/pr_meta_store_from_local.py"
      click RAWFETCH "https://github.com/sagar8080/semantic-search-system/blob/main/ingest/pr_meta_store.py"
    end

    subgraph "NLP & Embedding Services"
      AWSNLP["AWS-backed NLP & embeddings"]:::service
      LOCALNLP["Local NLP Pipeline"]:::service
      TOPICMOD["Topic Modeling"]:::service
      click AWSNLP "https://github.com/sagar8080/semantic-search-system/blob/main/nlp/pr_aws_nlp.py"
      click LOCALNLP "https://github.com/sagar8080/semantic-search-system/blob/main/nlp/pr_local_nlp_pipeline.py"
      click TOPICMOD "https://github.com/sagar8080/semantic-search-system/blob/main/nlp/topic_modeling.py"
    end

    VECTOR["Vector Index Setup & Loader"]:::service
    SEARCHSVC["Search Service"]:::service
    SEARCHPIPE["Query/Enrichment Pipeline"]:::service

    subgraph "Knowledge Graph Services"
      KGBUILD["Knowledge Graph Builder"]:::service
      KGUTILS["Graph Utilities"]:::service
      click KGBUILD "https://github.com/sagar8080/semantic-search-system/blob/main/knowledge_graph/knowledge_graph.py"
      click KGUTILS "https://github.com/sagar8080/semantic-search-system/blob/main/knowledge_graph/utils.py"
    end

    AWSINT["AWS Bedrock Integration"]:::external
    OPENSEARCHWRAP["OpenSearch Client & Constants"]:::external
    CONSTS["Index & Constants"]:::external
    SECRETS["Secrets & Config Loader"]:::external
    click AWSINT "https://github.com/sagar8080/semantic-search-system/blob/main/utils/bedrock.py"
    click OPENSEARCHWRAP "https://github.com/sagar8080/semantic-search-system/blob/main/utils/opensearch.py"
    click CONSTS "https://github.com/sagar8080/semantic-search-system/blob/main/utils/constants.py"
    click SECRETS "https://github.com/sagar8080/semantic-search-system/blob/main/utils/get_secrets.py"
  end

  %% Data Stores
  subgraph "Data Stores"
    OS_URL["PR_META_URL_IDX"]:::datastore
    OS_RAW["PR_META_RAW_IDX"]:::datastore
    OS_VEC["PR_META_VECTOR_IDX"]:::datastore
    NEO4J["Neo4j Knowledge Graph"]:::datastore
  end

  %% External Services
  subgraph "External Services"
    AWS["AWS Bedrock"]:::external
  end

  %% Static Data
  TOPICS["Topic Definitions & Mappings"]:::datastore
  URLSEED["Initial URL Seed Data"]:::datastore
  click TOPICS "https://github.com/sagar8080/semantic-search-system/tree/main/topics/"
  click URLSEED "https://github.com/sagar8080/semantic-search-system/blob/main/url_paths/press_releases.json"

  %% Data Flows
  CRAWLER --> OS_URL
  OS_URL --> RAWFETCH --> OS_RAW
  OS_RAW --> AWSNLP --> OS_VEC
  OS_RAW --> LOCALNLP --> OS_VEC
  AWSNLP --> VECTOR --> OS_VEC
  LOCALNLP --> VECTOR
  TOPICMOD --> VECTOR

  OS_VEC --> SEARCHSVC --> MA
  SEARCHPIPE --> SEARCHSVC
  SEARCHSVC --> MA

  OS_VEC --> KGBUILD --> NEO4J
  TOPICS --> KGBUILD
  NEO4J --> EA
  HF --> TOPICS --> KGBUILD

  %% Utils interactions
  AWSINT -.-> AWSNLP
  AWSINT -.-> SEARCHPIPE
  OPENSEARCHWRAP -.-> CRAWLER
  OPENSEARCHWRAP -.-> RAWFETCH
  OPENSEARCHWRAP -.-> SEARCHSVC
  CONSTS -.-> CRAWLER
  CONSTS -.-> RAWFETCH
  CONSTS -.-> AWSNLP
  SECRETS -.-> CRAWLER

  %% Styles
  classDef ui fill:#FFA500,stroke:#333,stroke-width:1px
  classDef service fill:#90EE90,stroke:#333,stroke-width:1px
  classDef datastore fill:#87CEFA,stroke:#333,stroke-width:1px,shape:cylinder
  classDef external fill:#DDA0DD,stroke:#333,stroke-width:1px,stroke-dasharray: 5 5
```
