# microsoft/dstoolkit-kg-search architecture

> Architecture diagram of the GitHub repository microsoft/dstoolkit-kg-search, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-31). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/dstoolkit-kg-search
- Repository: https://github.com/microsoft/dstoolkit-kg-search

## Overview

##### Project Type and Purpose
- This is a full-stack, domain-aware search accelerator. It consists of two Flask applications—a front-end UI and a KG Search API—that work together to enrich user queries using a knowledge graph and surface results from a general-purpose search engine.
- Purpose: take a user’s search query, identify domain entities, expand/ refine the query via a KG (hosted in Azure Cosmos DB), and then retrieve documents from Azure Cognitive Search.

##### High-Level Components
a. Front-End UI (Flask + HTML/CSS/JS)
b. KG Search API (Flask)
c. Graph DB (Azure Cosmos DB with Gremlin API)
d. Search Index (Azure Cognitive Search)
e. Data Lake Storage (Azure Data Lake Gen2)
f. Supporting Scripts (data preparation & graph initialization)
g. Identity & Configuration (Azure AD for API auth, .env, App Settings)

##### Relationships & Data Flows
- User → Front-End UI: submits query
- UI → KG Search API (authenticated HTTP): sends query + KG-Enabled flag
- API pipeline (inside):
1. Preprocessing (remove stop words, normalize)
2. NER (rule-based or ML)
3. Graph Query (Gremlin calls to Cosmos DB)
4. Query Rewriting (combine original + related entities)
5. Index Query (Azure Cognitive Search SDK/REST)
6. Postprocessing (filtering/ranking)
- API → UI: returns ranked documents
- Offline scripts:
- prepare_data.py → writes JSON docs to Data Lake
- create_acs_index (Postman) → points ACS at Data Lake to index
- initialize_graph.py → populates KG in Cosmos DB

##### Architectural Patterns & Principles
- Layered/Modular pipeline in API (clear separation: preprocess → NER → KG → rewrite → search → postprocess)
- Separation of concerns: UI vs API vs Data & KG stores
- Extensibility: each pipeline stage implements a base class, swap in custom logic
- Cloud-native: externalize config in environment variables or App Settings, leverage PaaS services

##### Detailed Elements to Capture
- Authentication: UI obtains token from Azure AD, passes to API
- Environment configuration: .env / App Settings for ACS_ENDPOINT, COSMOS_DB_SERVER, etc.
- Deployment details: Gunicorn startup commands (—chdir api app:app and —chdir ui app:app)

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Services
    subgraph "Front-End UI Service"
        UI["Front-End UI Service\n(Flask + Gunicorn)"]:::service
    end
    subgraph "KG Search API Service"
        API["KG Search API Service\n(Flask + Gunicorn)"]:::service
        subgraph "Search Expander Orchestrator"
            SE["Search Expander Orchestrator"]:::module
            PP["Preprocessor"]:::module
            NP["NER Processor"]:::module
            GC["Graph DB Client"]:::module
            KE["Knowledge Extractor"]:::module
            QR["Query Rewriter"]:::module
            AS["ACS SDK Client"]:::module
            PO["Postprocessor"]:::module
        end
        UF["Utility Functions"]:::module
    end

    %% PaaS Services
    Cosmos["Cosmos DB (Gremlin API)"]:::paas
    ACS["Azure Cognitive Search"]:::paas
    DL["Azure Data Lake Gen2"]:::paas

    %% Scripts
    PD["prepare_data.py"]:::script
    CI["create_acs_index"]:::script
    IG["initialize_graph.py"]:::script

    %% Configuration
    CFG["Azure AD & Config\n(.env / App Settings)"]:::config

    %% Flows
    User["User"]:::external -->|Submits Query| UI
    UI -->|Obtain Token| CFG
    CFG -.->|Provides Token| API
    UI -->|HTTPS/JSON| API

    API --> SE
    SE --> PP
    PP --> NP
    NP --> GC
    GC --> Cosmos
    GC --> KE
    KE --> QR
    QR --> AS
    AS --> ACS
    ACS --> AS
    AS --> PO
    PO --> API
    API -->|HTTPS/JSON| UI

    PD -->|Writes JSON| DL
    CI -->|Indexer Setup| ACS
    IG -->|Populates KG| Cosmos

    %% Click Events
    click UI "https://github.com/microsoft/dstoolkit-kg-search/blob/main/ui/app.py"
    click SE "https://github.com/microsoft/dstoolkit-kg-search/blob/main/api/search_expansion/search_expander.py"
    click PP "https://github.com/microsoft/dstoolkit-kg-search/blob/main/api/search_expansion/preprocessing/preprocessor.py"
    click NP "https://github.com/microsoft/dstoolkit-kg-search/blob/main/api/search_expansion/nerprocessing/ner.py"
    click GC "https://github.com/microsoft/dstoolkit-kg-search/blob/main/api/search_expansion/kg/graph_db.py"
    click KE "https://github.com/microsoft/dstoolkit-kg-search/blob/main/api/search_expansion/kg/knowledge_extractor.py"
    click QR "https://github.com/microsoft/dstoolkit-kg-search/blob/main/api/search_expansion/rewriting/query_rewriter.py"
    click AS "https://github.com/microsoft/dstoolkit-kg-search/blob/main/api/search_expansion/search_sdk.py"
    click PO "https://github.com/microsoft/dstoolkit-kg-search/blob/main/api/search_expansion/postprocessing/postprocessor.py"
    click UF "https://github.com/microsoft/dstoolkit-kg-search/blob/main/api/search_expansion/util.py"
    click API "https://github.com/microsoft/dstoolkit-kg-search/blob/main/api/app.py"
    click PD "https://github.com/microsoft/dstoolkit-kg-search/blob/main/scripts/prepare_data.py"
    click CI "https://github.com/microsoft/dstoolkit-kg-search/blob/main/scripts/create_acs_index.postman_collection.json"
    click IG "https://github.com/microsoft/dstoolkit-kg-search/blob/main/scripts/initialize_graph.py"
    click CFG "https://github.com/microsoft/dstoolkit-kg-search/blob/main/config/logging.config"

    %% Styles
    classDef service fill:#D6EAF8,stroke:#2980B9,color:#154360;
    classDef module fill:#AED6F1,stroke:#2874A6,color:#1B4F72;
    classDef paas fill:#A9DFBF,stroke:#27AE60,color:#1D8348;
    classDef script fill:#E5E7E9,stroke:#A6ACAF,color:#566573;
    classDef config fill:#FADBD8,stroke:#C0392B,color:#641E16;
    classDef external fill:#F9E79F,stroke:#B7950B,color:#7D6608;
```
