# rksm/org-ai architecture

> Architecture diagram of the GitHub repository rksm/org-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-03). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/rksm/org-ai
- Repository: https://github.com/rksm/org-ai

## Overview

Based on our analysis of the file tree and README, the “org-ai” project is an Emacs minor mode that extends org-mode to integrate generative AI capabilities. It provides access to various AI models (ChatGPT, DALL-E, Stable Diffusion, local LLMs via oobabooga, etc.), speech input/output support, noweb support, and even global commands for interacting with code and project files. The project is organized as a single-package Emacs Lisp distribution with several modular files (each handling a specific area of functionality) and some support snippets.

##### Main Components
- Core Integration Module – The basic org-ai functionality that hooks into org-mode (activation via org-ai-mode and org-ai-global-mode).
- Block Parsing/Handling – Responsible for recognizing and processing special org-mode blocks (those starting with “#+begin_ai”) and handling noweb expansion.
- AI Request Manager – Manages assembling requests (chat prompts, image prompts, etc.) and parameters. This includes handling different APIs (OpenAI, Azure, perplexity.ai, Anthropics) as well as local LLM integrations.
- External API Integration – Separate subcomponents for:
– Text generation for ChatGPT-style requests
– Image generation handling through DALL-E and Stable Diffusion, with dedicated Emacs Lisp support for configuring defaults (e.g., image size, count)
– Local models handled via oobabooga (accessible with the :local option)
- Speech Input/Output Module – Integrates with whisper.el (or Siri dictation on macOS) for speech recognition, and uses greader/espeak for synthesizing speech. This module handles capturing, processing, and returning voice input/output.
- Global Commands & Project Integration – Implements additional commands for selecting regions, entire files, or entire projects (org-ai-on-project) for prompting and even generating diffs.
- Configuration & Customization – Handles user settings (API keys, default model names, image settings, etc.) and uses integration with auth-source for secure token management.
- Snippets Support – Provides optional snippets for org-mode integration to improve developer productivity.

##### Relationships and Data Flow
- Users interact with the system primarily from within an org-mode buffer by writing special “#+begin_ai … #+end_ai” blocks.
- The block parser analyzes the region and then routes the contents (possibly through noweb expansion) to the AI Request Manager.
- The AI Request Manager formats the request into an API payload. Depending on block options (e.g., “:image”, “:local”, “:completion”), it directs the request to an appropriate external API integration module.
- The external API integration component manages the communication with external services (or a local server). It sends requests and gets streamed responses back.
- Responses (text or images) are then injected back into the org buffer, and any file- or project-based commands trigger diff generation or modifications through the global commands module.
- Optionally, if speech input is enabled, a separate pipeline handles audio capture via whisper.el and inserts the transcribed text into the process flow; similarly, speech output is managed from the AI response.
- Throughout these operations, configuration settings (stored in user customizations or loaded from auth-info, etc.) influence the behavior of the request formatting, API selection, and even parameter defaults.

##### Important Architectural Patterns & Principles
- Modularity: The design is decomposed into discrete Emacs Lisp files that each handle a separate concern (e.g., org-ai-block.el for block processing, org-ai-openai.el for API requests, etc.).
- Extensibility: Multiple API integrations exist (OpenAI, Azure, perplexity.ai, local LLM, etc.), and the system is flexible enough to support additional services.
- Loose Coupling: Communication between the UI (org-mode interface) and the back-end processing (API integration) is mediated by clear, well-defined interfaces – the user’s prompt is parsed and then transformed into JSON payloads or HTTP calls.
- Customizability: With numerous customizable variables (for models, image sizes, tokens, speech settings), the architecture embraces configuration through Emacs’ built-in systems.

## Diagram (Mermaid)

```mermaid
flowchart TD
    A["Org-Mode Interface"]:::ui
    B["Core Integration Module"]:::internal
    C["Block Parsing & Noweb Expansion"]:::internal
    D["AI Request Manager (Payload Formation & Routing)"]:::internal
    E["Response Processing & Insertion"]:::internal

    subgraph "External API Integration"
        D1["OpenAI/ChatGPT"]:::external
        D2["Image Generation - DALL-E"]:::external
        D3["Image Generation - Stable Diffusion"]:::external
        D4["Local LLM (oobabooga)"]:::external
        D5["Other Services (Azure,Perplexity.ai,Anthropic)"]:::external
    end

    subgraph "Speech Module"
        S1["Speech Input (whisper.el/Siri)"]:::speech
        S2["Speech Output (greader/espeak)"]:::speech
    end

    G["Global Commands & Project Handler"]:::internal
    H["Configuration & Auth Management"]:::config
    I["Snippets Support"]:::snippets

    %% Flow Connections
    A -->|"Trigger"| B
    B -->|"Hook org-ai-mode"| C
    C -->|"ExtractAIBlocks"| D
    D -->|"RouteRequest"| D1
    D -->|"RouteRequest"| D2
    D -->|"RouteRequest"| D3
    D -->|"RouteRequest"| D4
    D -->|"RouteRequest"| D5
    D1 -->|"ReturnResponse"| E
    D2 -->|"ReturnResponse"| E
    D3 -->|"ReturnResponse"| E
    D4 -->|"ReturnResponse"| E
    D5 -->|"ReturnResponse"| E
    E -->|"InsertResponse"| A

    A -->|"GlobalCommands"| G
    G -->|"ApplyDiff"| E

    S1 -->|"VoiceInput"| D
    D -->|"VoiceResponse"| S2
    S2 -->|"SpeakOutput"| A

    H -.-> D
    H -.-> D1
    H -.-> D2
    H -.-> D3
    H -.-> D4
    H -.-> D5
    H -.-> G

    I -->|"EnhanceOrgMode"| C

    %% Click Events
    click B "https://github.com/rksm/org-ai/blob/master/org-ai.el"
    click C "https://github.com/rksm/org-ai/blob/master/org-ai-block.el"
    click D "https://github.com/rksm/org-ai/blob/master/org-ai-openai.el"
    click D1 "https://github.com/rksm/org-ai/blob/master/org-ai-openai.el"
    click D2 "https://github.com/rksm/org-ai/blob/master/org-ai-openai-image.el"
    click D3 "https://github.com/rksm/org-ai/blob/master/org-ai-sd.el"
    click D4 "https://github.com/rksm/org-ai/blob/master/org-ai-oobabooga.el"
    click S1 "https://github.com/rksm/org-ai/blob/master/org-ai-talk.el"
    click S2 "https://github.com/rksm/org-ai/blob/master/org-ai-talk.el"
    click G "https://github.com/rksm/org-ai/blob/master/org-ai-on-project.el"
    click H "https://github.com/rksm/org-ai/blob/master/org-ai-useful.el"
    click I "https://github.com/rksm/org-ai/tree/master/snippets/org-mode"

    %% Class Styles
    classDef ui fill:#fef2f2,stroke:#b91c1c,stroke-width:2px,stroke-dasharray: 5 5;
    classDef internal fill:#dbeafe,stroke:#1e3a8a,stroke-width:2px;
    classDef external fill:#dcfce7,stroke:#16a34a,stroke-width:2px;
    classDef speech fill:#fef3c7,stroke:#92400e,stroke-width:2px;
    classDef config fill:#e0f2fe,stroke:#0369a1,stroke-width:2px;
    classDef snippets fill:#ede9fe,stroke:#6b21a8,stroke-width:2px;
```
