How mastra-ai/rag-workshop-code works
Main Components
• Document Sources: The project contains two kinds of source documents – one in the regular documents folder (src/documents) and another in the bonus section (src/bonus/documents). These files include markdown, JSON, and even TypeScript documents to be upserted into a vector store.
• Chunking Modules: Several examples demonstrate different chunking methods:
• Character Chunking, Recursive Chunking, JSON Chunking, and Markdown Chunking (found under src/examples/01-04).
• Embedding & Vector Store Operations: Examples showing embedding (src/examples/05-embedding.ts), vector upserting (src/examples/06-vector-upserting.ts), similarity search (src/examples/07-vector-search.ts), and reranking (src/examples/08-vector-reranking.ts).
• Agents: Modules related to the search functionalities are implemented within the agents folder (src/mastra/agents) along with corresponding examples (src/examples/09-basic-search-usage.ts and src/examples/10-query-vector-usage.ts). They are responsible for both basic keyword and semantic (vector-based) searches.
• Bonus Examples: An additional section (src/bonus/) provides advanced, code-aware search functionalities including examples targeting code usage (like 01-find-code-usage.ts) and documents (authentication, logging, etc).
Relationships/Interactions
• Flow from Document Ingestion → Chunking → Embedding → Upserting into the Vector Store.
• Retrieval path: When a user query is submitted, the relevant agent (from the agents module) leverages vector search operations (by querying PgVector via PostgreSQL) and may also perform re-ranking to produce the end result.
• The bonus examples demonstrate similar flows but are tailored for code searches, indicating specialized handling and processing logic.
Important Architectural Patterns and Design Principles
• Separation of Concerns: The directory structure is segmented (documents, examples, mastra, bonus) where each module is clearly responsible for a specific function: ingesting documents, processing/chunking, vector operations, and search functionalities.
• Modular Design: Tools and agents (found in src/mastra) are separate modules that can be recombined to form different instructions.
• Data Flow Pipelines: The overall process from document ingestion to vector search represents a clear pipeline pattern—each step passes data to the next.
• From “Documents” to “Chunking Strategies”.
• From “Chunking” to “Embedding Service”.
• From “Embedding” to “Vector Store (PgVector/Database)”.
• From “Vector Store” to “Agents” (basic and semantic).
• Optionally, highlight a secondary flow for the bonus examples (code search path).