# microsoft/typeagent-py architecture

> Architecture diagram of the GitHub repository microsoft/typeagent-py, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-10-27). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/microsoft/typeagent-py
- Repository: https://github.com/microsoft/typeagent-py

## Overview

##### Project type and purpose
- Project: “typeagent” – a Python library/agent for Structured RAG (Retrieval-Augmented Generation).
- Purpose: ingest diverse data sources (emails, transcripts, podcasts), build indexes, serve queries via a conversational agent that calls out to an LLM and returns structured answers.

##### File structure
Top-level directories:
· demo/ and examples/ – sample ingestion/query scripts.
· docs/ and spec/ – design docs, API guides, architecture overviews.
· typeagent/ – core library, subdivided into:
- aitools – LLM auth, embeddings, vector store abstraction
- emails, podcasts, transcripts – ingestion pipelines & domain-specific preprocessors
- knowpro – core conversational logic, schemas, search/query/answer modules
- storage – memory & sqlite providers for indexes (inverted, semantic, timestamp, etc.)
- mcp – lightweight server component (RPC or micro-service)
· tools/ – CLI wrappers around demo or ingestion/query workflows
· test/ and testdata/ – unit tests & sample data

##### README for insights
- LLM calls to third-party APIs
- MVP oriented around TypeAgent KnowPro translation
- Warnings about sending confidential data

A. Data Sources (left side)
– Gmail dump (gmail/), podcast feed files, transcript VTT or text.
– Demo/testdata as example sources.
B. Ingestion Pipelines
– modules in typeagent/emails, typeagent/podcasts, typeagent/transcripts
– demo/ingest.py, tools/ingest_vtt.py
C. Indexing & Storage
– Index modules in typeagent/knowpro (fuzzyindex, secindex, textlocindex, timestampindex, etc.)
– Storage providers:
- In-memory (typeagent/storage/memory)
- SQLite (typeagent/storage/sqlite)
D. Search & Retrieval
– knowpro.search, searchlib, query, search_query_schema
– UDFs for ranking, related terms, property filters
E. Conversational Agent / Orchestrator
– knowpro.factory, conversation_base, convknowledge, convsettings
– Takes user query → retrieves context → calls aitools.embeddings + LLM → uses answer_response_schema
F. External Services
– Third-party LLM API (auth via aitools/auth)
– Optional micro-service interface via mcp/server
G. Client Interfaces
– demo/query.py, tools/query.py, examples/simple_query_demo.py
– Potential CLI (Makefile targets, tools/)

##### Relationships & interactions
- Data Sources → Ingestion Pipeline → Raw documents
- Raw documents → Indexing modules → Written to Storage Provider
- Client Query → Agent Orchestrator → Search Module → Storage → Retrieved context
- Orchestrator → aitools.embeddings & LLM API → Answer generation
- Answer → Returned to Client Interface

##### Architectural patterns & principles
- Modular plugin-style ingestion (one module per data type)
- Storage-agnostic design: swappable providers (memory vs SQLite)
- Layered separation: ingestion → indexing → retrieval → LLM orchestration
- Schema-driven conversation & answer flows (pydantic-style schemas in knowpro)
- Thin HTTP/RPC layer for optional “mcp” micro server

## Diagram (Mermaid)

```mermaid
flowchart TB
    subgraph "Data Sources"
        DS1["Gmail Dump"]:::green
        DS2["Podcast Feed Files"]:::green
        DS3["Transcript Files"]:::green
    end

    subgraph "Ingestion Pipelines"
        EI["Email Ingestion"]:::green
        PI["Podcast Ingestion"]:::green
        TI["Transcript Ingestion"]:::green
        DI["Demo Script"]:::green
        CI["CLI VTT Ingestion"]:::green
    end

    subgraph "Indexing & Storage"
        subgraph "Indexing Modules"
            IQ1["Fuzzy Index"]:::blue
            IQ2["Section Index"]:::blue
            IQ3["Text Location Index"]:::blue
            IQ4["Timestamp Index"]:::blue
        end
        subgraph "Storage Providers"
            SP1["In-Memory Provider"]:::blue
            SP2["SQLite Provider"]:::blue
        end
    end

    subgraph "Search & Retrieval"
        SE["Search Engine"]:::blue
        SL["Search Library"]:::blue
        QM["Query Module"]:::blue
        SS["Search Schema"]:::blue
    end

    subgraph "Conversational Orchestrator"
        FO["Factory"]:::orange
        CB["Conversation Base"]:::orange
        CK["Conversation Knowledge"]:::orange
        CS["Conversation Settings"]:::orange
    end

    subgraph "External Services"
        Auth["LLM Authentication"]:::purple
        Emb["Embeddings & VectorBase"]:::purple
        Micro["Microservice Server"]:::purple
    end

    subgraph "Client Interfaces"
        CQ["Demo Query"]:::green
        CLIQ["CLI Query Tool"]:::green
        EXQ["Example Query Demo"]:::green
    end

    DS1 --> EI
    DS2 --> PI
    DS3 --> TI
    DI --> IQ4
    CI --> TI

    EI --> IQ1
    PI --> IQ2
    TI --> IQ3

    IQ1 --> SP1
    IQ1 --> SP2
    IQ2 --> SP1
    IQ2 --> SP2
    IQ3 --> SP1
    IQ3 --> SP2
    IQ4 --> SP1
    IQ4 --> SP2

    SP1 --> SE
    SP2 --> SE
    SE --> SL
    SL --> QM
    QM --> SS

    SS --> FO
    FO --> CB
    CB --> CK
    CK --> CS
    CS --> FO

    FO --> Auth
    FO --> Emb
    Micro --> FO

    CQ --> FO
    CLIQ --> FO
    EXQ --> FO

    click DS1 "https://github.com/microsoft/typeagent-py/blob/main/gmail/gmail_dump.py"
    click DS2 "https://github.com/microsoft/typeagent-py/blob/main/testdata/Confuse-A-Cat.vtt"
    click DS3 "https://github.com/microsoft/typeagent-py/blob/main/testdata/Confuse-A-Cat.vtt"
    click EI "https://github.com/microsoft/typeagent-py/blob/main/typeagent/emails/email_import.py"
    click PI "https://github.com/microsoft/typeagent-py/blob/main/typeagent/podcasts/podcast_ingest.py"
    click TI "https://github.com/microsoft/typeagent-py/blob/main/typeagent/transcripts/transcript_ingest.py"
    click DI "https://github.com/microsoft/typeagent-py/blob/main/demo/ingest.py"
    click CI "https://github.com/microsoft/typeagent-py/blob/main/tools/ingest_vtt.py"
    click IQ1 "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/fuzzyindex.py"
    click IQ2 "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/secindex.py"
    click IQ3 "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/textlocindex.py"
    click IQ4 "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/timestampindex.py"
    click SP1 "https://github.com/microsoft/typeagent-py/blob/main/typeagent/storage/memory/provider.py"
    click SP2 "https://github.com/microsoft/typeagent-py/blob/main/typeagent/storage/sqlite/provider.py"
    click SE "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/search.py"
    click SL "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/searchlib.py"
    click QM "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/query.py"
    click SS "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/search_query_schema.py"
    click FO "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/factory.py"
    click CB "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/conversation_base.py"
    click CK "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/convknowledge.py"
    click CS "https://github.com/microsoft/typeagent-py/blob/main/typeagent/knowpro/convsettings.py"
    click Auth "https://github.com/microsoft/typeagent-py/blob/main/typeagent/aitools/auth.py"
    click Emb "https://github.com/microsoft/typeagent-py/blob/main/typeagent/aitools/embeddings.py"
    click Micro "https://github.com/microsoft/typeagent-py/blob/main/typeagent/mcp/server.py"
    click CQ "https://github.com/microsoft/typeagent-py/blob/main/demo/query.py"
    click CLIQ "https://github.com/microsoft/typeagent-py/blob/main/tools/query.py"
    click EXQ "https://github.com/microsoft/typeagent-py/blob/main/examples/simple_query_demo.py"

    classDef green fill:#e0f7fa,stroke:#000
    classDef blue fill:#bbdefb,stroke:#000
    classDef orange fill:#ffe0b2,stroke:#000
    classDef purple fill:#e1bee7,stroke:#000
```
