# nelaturuharsha/ayamcooking architecture

> Architecture diagram of the GitHub repository nelaturuharsha/ayamcooking, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-22). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/nelaturuharsha/ayamcooking
- Repository: https://github.com/nelaturuharsha/ayamcooking

## Overview

##### Project type and purpose
- AyaMCooking is a prototype/demo voice‑to‑voice, multi‑lingual RAG (Retrieval‑Augmented Generation) agent implemented as a Jupyter notebook.
- Its purpose is to let users speak a query in one of ten languages, retrieve and generate relevant cooking recipes via a multi‑lingual RAG pipeline, then synthesize a spoken reply.

##### File structure
- Top level only contains a single notebook (`AyaMCooking.ipynb`), two license files, and `README.md`.
- No separate frontend folder or backend code—everything is orchestrated inside the notebook.
- External dependencies and services (ASR, embedding, RAG, TTS) are invoked via API calls from within the notebook.

##### README for additional insights
- Core pipeline stages:
1. Whisper ASR to transcribe user audio.
2. Cohere multilingual embeddings + index build + re‑ranking.
3. Aya Expanse LLM to generate step‑by‑step recipes.
4. Coqui TTS for voice synthesis.
- Data: 100‑recipe multilingual dataset hosted on Hugging Face.
- Interaction pattern: Voice in → text in → RAG lookup → LLM generation → text out → voice out.

##### Main components and interactions
a. Components
- User Device (microphone & speaker)
- ASR Service (Whisper)
- Embedding & Vector Store (Cohere‑multilingual‑v3, custom index)
- Reranker (Cohere ReRank3)
- LLM Service (Aya Expanse / Cohere API)
- TTS Service (Coqui)
- Recipe Dataset (Hugging Face)
- Orchestration Notebook (AyaMCooking.ipynb)

b. Data flow
1. User records audio → Notebook streams audio to Whisper → transcribed text.
2. Notebook calls Cohere Embed API → generate embeddings for query.
3. Query embedding → vector store search (recipe dataset) → top‑k docs.
4. Retrieved docs → Cohere ReRank3 → refine top result(s).
5. Combine refined context + prompt template + language code → call Aya Expanse → generate recipe text.
6. Generated text → Coqui TTS API → voice response → played on user device.

c. Architectural patterns & design principles
- RAG pattern: decouples retrieval (search + ranking) from generation (LLM).
- Language‑agnostic pipeline: each component supports the same set of languages or is wrapped to handle language codes.

d. Key technologies
- ASR: Whisper (OpenAI)
- Embedding & Indexing: Cohere multilingual embeddings + custom vector store
- Re‑ranking: Cohere ReRank3
- Generation: Aya Expanse (Cohere LLM API)
- TTS: Coqui TTS
- Dataset host: Hugging Face Datasets
- Notebook: Python / Jupyter

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User Device
    subgraph "User Device"
        AudioIn([Microphone]):::user
        Speaker([Speaker]):::user
    end

    %% Orchestration
    subgraph "Orchestration Layer"
        Notebook["AyaMCooking.ipynb"]:::orchestrator
    end

    %% External Services & Data
    subgraph "External Services & Data"
        ASR["Whisper ASR"]:::service
        Embed["Cohere Embed & Vector Store"]:::service
        Rerank["Cohere ReRank3"]:::service
        LLM["Aya Expanse LLM"]:::service
        TTS["Coqui TTS"]:::service
        Dataset["Recipe Dataset (Hugging Face)"]:::data
    end

    %% Project Documentation
    subgraph "Project Documentation"
        README["README.md"]:::doc
        LICENSE["LICENSE"]:::doc
        LICENSE2["LICENSE.txt"]:::doc
    end

    %% Data Flow
    AudioIn -->|".wav audio"| Notebook
    Notebook -->|".wav audio"| ASR
    ASR -->|"UTF-8 text\n(lang code)"| Notebook
    Notebook -->|"UTF-8 text\n(lang code)"| Embed
    Dataset -->|"Recipes data"| Embed
    Embed -->|"Docs (top-k)"| Rerank
    Rerank -->|"Refined docs"| Notebook
    Notebook -->|"Prompt+Context\n(lang code)"| LLM
    LLM -->|"Recipe text"| Notebook
    Notebook -->|"Recipe text"| TTS
    TTS -->|".wav audio"| Notebook
    Notebook -->|"Play audio"| Speaker

    %% Click Events
    click Notebook "https://github.com/nelaturuharsha/ayamcooking/blob/main/AyaMCooking.ipynb"
    click README "https://github.com/nelaturuharsha/ayamcooking/blob/main/README.md"
    click LICENSE "https://github.com/nelaturuharsha/ayamcooking/tree/main/LICENSE"
    click LICENSE2 "https://github.com/nelaturuharsha/ayamcooking/blob/main/LICENSE.txt"

    %% Styles
    classDef user fill:#D6EFFF,stroke:#1E90FF,color:#000;
    classDef orchestrator fill:#FFF2CC,stroke:#FFD700,color:#000;
    classDef service fill:#E2F0D9,stroke:#228B22,color:#000;
    classDef data fill:#FCE4D6,stroke:#FF8C00,color:#000;
    classDef doc fill:#F0F0F0,stroke:#808080,color:#000;
```
