How microsoft/dstoolkit-kg-search works
Project Type and Purpose
• This is a full-stack, domain-aware search accelerator. It consists of two Flask applications—a front-end UI and a KG Search API—that work together to enrich user queries using a knowledge graph and surface results from a general-purpose search engine.
• Purpose: take a user’s search query, identify domain entities, expand/ refine the query via a KG (hosted in Azure Cosmos DB), and then retrieve documents from Azure Cognitive Search.
High-Level Components
a. Front-End UI (Flask + HTML/CSS/JS)
b. KG Search API (Flask)
c. Graph DB (Azure Cosmos DB with Gremlin API)
d. Search Index (Azure Cognitive Search)
e. Data Lake Storage (Azure Data Lake Gen2)
f. Supporting Scripts (data preparation & graph initialization)
g. Identity & Configuration (Azure AD for API auth, .env, App Settings)
Relationships & Data Flows
• User → Front-End UI: submits query
• UI → KG Search API (authenticated HTTP): sends query + KG-Enabled flag
• API pipeline (inside):
1. Preprocessing (remove stop words, normalize)
2. NER (rule-based or ML)
3. Graph Query (Gremlin calls to Cosmos DB)
4. Query Rewriting (combine original + related entities)
5. Index Query (Azure Cognitive Search SDK/REST)
6. Postprocessing (filtering/ranking)
• API → UI: returns ranked documents
• Offline scripts:
• prepare_data.py → writes JSON docs to Data Lake
• create_acs_index (Postman) → points ACS at Data Lake to index
• initialize_graph.py → populates KG in Cosmos DB
Architectural Patterns & Principles
• Layered/Modular pipeline in API (clear separation: preprocess → NER → KG → rewrite → search → postprocess)
• Separation of concerns: UI vs API vs Data & KG stores
• Extensibility: each pipeline stage implements a base class, swap in custom logic
• Cloud-native: externalize config in environment variables or App Settings, leverage PaaS services
Detailed Elements to Capture
• Authentication: UI obtains token from Azure AD, passes to API
• Environment configuration: .env / App Settings for ACS_ENDPOINT, COSMOS_DB_SERVER, etc.
• Deployment details: Gunicorn startup commands (—chdir api app:app and —chdir ui app:app)