How microsoft/typeagent-py works
Project type and purpose
• Project: “typeagent” – a Python library/agent for Structured RAG (Retrieval-Augmented Generation).
• Purpose: ingest diverse data sources (emails, transcripts, podcasts), build indexes, serve queries via a conversational agent that calls out to an LLM and returns structured answers.
File structure
Top-level directories:
· demo/ and examples/ – sample ingestion/query scripts.
· docs/ and spec/ – design docs, API guides, architecture overviews.
· typeagent/ – core library, subdivided into:
• aitools – LLM auth, embeddings, vector store abstraction
• emails, podcasts, transcripts – ingestion pipelines & domain-specific preprocessors
• knowpro – core conversational logic, schemas, search/query/answer modules
• storage – memory & sqlite providers for indexes (inverted, semantic, timestamp, etc.)
• mcp – lightweight server component (RPC or micro-service)
· tools/ – CLI wrappers around demo or ingestion/query workflows
· test/ and testdata/ – unit tests & sample data
README for insights
• LLM calls to third-party APIs
• MVP oriented around TypeAgent KnowPro translation
• Warnings about sending confidential data
A. Data Sources (left side)
– Gmail dump (gmail/), podcast feed files, transcript VTT or text.
– Demo/testdata as example sources.
B. Ingestion Pipelines
– modules in typeagent/emails, typeagent/podcasts, typeagent/transcripts
– demo/ingest.py, tools/ingest_vtt.py
C. Indexing & Storage
– Index modules in typeagent/knowpro (fuzzyindex, secindex, textlocindex, timestampindex, etc.)
– Storage providers:
• In-memory (typeagent/storage/memory)
• SQLite (typeagent/storage/sqlite)
D. Search & Retrieval
– knowpro.search, searchlib, query, search_query_schema
– UDFs for ranking, related terms, property filters
E. Conversational Agent / Orchestrator
– knowpro.factory, conversation_base, convknowledge, convsettings
– Takes user query → retrieves context → calls aitools.embeddings + LLM → uses answer_response_schema
F. External Services
– Third-party LLM API (auth via aitools/auth)
– Optional micro-service interface via mcp/server
G. Client Interfaces
– demo/query.py, tools/query.py, examples/simple_query_demo.py
– Potential CLI (Makefile targets, tools/)
Relationships & interactions
• Data Sources → Ingestion Pipeline → Raw documents
• Raw documents → Indexing modules → Written to Storage Provider
• Client Query → Agent Orchestrator → Search Module → Storage → Retrieved context
• Orchestrator → aitools.embeddings & LLM API → Answer generation
• Answer → Returned to Client Interface
Architectural patterns & principles
• Modular plugin-style ingestion (one module per data type)
• Storage-agnostic design: swappable providers (memory vs SQLite)
• Layered separation: ingestion → indexing → retrieval → LLM orchestration
• Schema-driven conversation & answer flows (pydantic-style schemas in knowpro)
• Thin HTTP/RPC layer for optional “mcp” micro server