How nelaturuharsha/ayamcooking works
Project type and purpose
• AyaMCooking is a prototype/demo voice‑to‑voice, multi‑lingual RAG (Retrieval‑Augmented Generation) agent implemented as a Jupyter notebook.
• Its purpose is to let users speak a query in one of ten languages, retrieve and generate relevant cooking recipes via a multi‑lingual RAG pipeline, then synthesize a spoken reply.
File structure
• Top level only contains a single notebook (AyaMCooking.ipynb), two license files, and README.md.
• No separate frontend folder or backend code—everything is orchestrated inside the notebook.
• External dependencies and services (ASR, embedding, RAG, TTS) are invoked via API calls from within the notebook.
README for additional insights
• Core pipeline stages:
1. Whisper ASR to transcribe user audio.
2. Cohere multilingual embeddings + index build + re‑ranking.
3. Aya Expanse LLM to generate step‑by‑step recipes.
4. Coqui TTS for voice synthesis.
• Data: 100‑recipe multilingual dataset hosted on Hugging Face.
• Interaction pattern: Voice in → text in → RAG lookup → LLM generation → text out → voice out.
Main components and interactions
a. Components
• User Device (microphone & speaker)
• ASR Service (Whisper)
• Embedding & Vector Store (Cohere‑multilingual‑v3, custom index)
• Reranker (Cohere ReRank3)
• LLM Service (Aya Expanse / Cohere API)
• TTS Service (Coqui)
• Recipe Dataset (Hugging Face)
• Orchestration Notebook (AyaMCooking.ipynb)
b. Data flow
1. User records audio → Notebook streams audio to Whisper → transcribed text.
2. Notebook calls Cohere Embed API → generate embeddings for query.
3. Query embedding → vector store search (recipe dataset) → top‑k docs.
4. Retrieved docs → Cohere ReRank3 → refine top result(s).
5. Combine refined context + prompt template + language code → call Aya Expanse → generate recipe text.
6. Generated text → Coqui TTS API → voice response → played on user device.
c. Architectural patterns & design principles
• RAG pattern: decouples retrieval (search + ranking) from generation (LLM).
• Language‑agnostic pipeline: each component supports the same set of languages or is wrapped to handle language codes.
d. Key technologies
• ASR: Whisper (OpenAI)
• Embedding & Indexing: Cohere multilingual embeddings + custom vector store
• Re‑ranking: Cohere ReRank3
• Generation: Aya Expanse (Cohere LLM API)
• TTS: Coqui TTS
• Dataset host: Hugging Face Datasets
• Notebook: Python / Jupyter