# principia-ai/writehere architecture

> Architecture diagram of the GitHub repository principia-ai/writehere, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-04-24). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/principia-ai/writehere
- Repository: https://github.com/principia-ai/writehere

## Overview

##### Project Type and Purpose
- This is a full-stack application with two operating modes:
a) **CLI-only engine** for batch processing of stories/reports (via `recursive/engine.py`).
b) **Web visualization interface** (React frontend + Flask backend).
- Core purpose: orchestrate recursive task decomposition, heterogeneous integration (retrieval, reasoning, composition), and dynamic adaptation via LLMs (OpenAI, Anthropic, SerpAPI).

##### Top-Level Components
A. **Frontend (React)**
- Runs on Node.js (port 3000 by default).
- Key folders: `frontend/src/pages` (Home, About, ReportGeneration, StoryGeneration, Results), `frontend/src/components` (Header, Footer, LiveTaskList, TaskGraph, HistoryPanel).
- Communicates with Flask API over HTTP.

B. **Backend API Server (Flask)**
- Implements REST endpoints in `backend/server.py`.
- Dependencies in `backend/requirements.txt`.
- Exposes endpoints for:
- starting a new run
- streaming task updates/status/results
- serving logs/debug info

C. **Core Engine (Python “recursive” package)**
- **engine.py**: entry point for CLI and backend invocation.
- **Agent Layer** (`recursive/agent`):
- `agent_base.py`: base agent interface
- `agents/regular.py`: default agent implementation
- `prompts/…`: organized per use-case (report vs story).
- **Executor Layer** (`recursive/executor`):
- `actions/`: action executors (bing_browser, builtin, parser, etc.)
- `agents/`: wrap LLM calls (claude_fc_react, base_agent).
- `schema.py`: defines task schema.
- **LLM Integrations** (`recursive/llm/llm.py`): abstracts OpenAI/Anthropic API calls via `recursive/proxy.py`.
- **Supporting Modules**:
- `memory.py`: memory management
- `cache.py`: reusable results cache
- `graph.py`: in-memory task graph representation
- `utils/`: display, file I/O, indexing, registration
- **Scripts**: `test_run_report.sh`, `test_run_story.sh` for sample runs.

D. **Data Sources & Config**
- `test_data/`: example JSONL and PDF inputs.
- `recursive/api_key.env`: holds LLM API keys.
- Config & setup scripts: `start.sh`, `setup_env.sh`, `run_with_anaconda.sh`.

E. **External Services**
- OpenAI (GPT family)
- Anthropic (Claude)
- SerpAPI (web search)

##### Relationships & Data Flows
- **User → Frontend**: via browser UI → submits story/report request.
- **Frontend → Backend**: HTTP calls to Flask endpoints.
- **Backend → Core Engine**: invokes `engine.py` (in-process or subprocess), passing file paths, mode flags, port/socket for updates.
- **Engine**
- Loads input data → builds initial task graph → stores in memory/cache.
- Agent decomposes tasks (recursive planning) → submits subtasks to Executor.
- Executor selects action (retrieval, reasoning, composition) → calls LLM or built-in action.
- LLM calls go through proxy → external API → returns content.
- Engine updates graph state → emits events/status.
- **Engine → Backend**: streams task-level events (ready, in-progress, completed) and results.
- **Backend → Frontend**: server-sent events or WebSocket pushes updates to React.
- **Frontend**: updates UI components (LiveTaskList, TaskGraph) in real-time; final results shown on ResultsPage.

##### Architectural Patterns & Principles
- **Modular layering**: clear separation between agent, executor, LLM integrations, memory, and I/O.
- **Event-driven task graph**: central `graph.py` manages state and event propagation.
- **Pluggable prompts & actions**: easy to add new prompt templates or actions via `recursive/agent/prompts` and `recursive/executor/actions`.
- **Dual-mode operation**: same engine core supports CLI batch runs and live web UI.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User and UI Layer
    User(("User")):::external
    Frontend["React App"]:::ui
    Pages["pages/"]:::ui
    Components["components/"]:::ui
    APIUtil["utils/api.js"]:::ui

    User -->|"interacts"| Frontend
    Frontend -->|"renders pages/components"| Pages
    Frontend -->|"renders pages/components"| Components
    Frontend -->|"calls API"| APIUtil

    %% Backend Layer
    Backend["Flask API Server"]:::backend
    ReqTxt["requirements.txt"]:::backend
    TestAPI["test_api.py"]:::backend

    Frontend -->|"HTTP/JSON"| Backend
    Backend -->|"loads deps"| ReqTxt
    Backend -->|"tests endpoints"| TestAPI

    %% Core Engine
    Backend -->|"invoke engine.py"| Engine
    Engine["engine.py"]:::core

    subgraph "Core Engine" 
        direction TB
        subgraph "Agent Layer"
            direction TB
            AgentBase["agent_base.py"]:::core
            RegularAgent["regular.py"]:::core
            Prompts["prompts/"]:::core
            Proxy["proxy.py"]:::core
        end
        subgraph "Executor Layer"
            direction TB
            Actions["actions/"]:::core
            ExecAgents["agents/"]:::core
            Schema["schema.py"]:::core
        end
        subgraph "LLM Integrations"
            direction TB
            LLM["llm.py"]:::core
        end
        subgraph "Task Graph & State"
            direction TB
            Graph["graph.py"]:::core
            Memory["memory.py"]:::core
            Cache["cache.py"]:::core
        end
        subgraph "Utilities"
            direction TB
            Display["display.py"]:::core
            FileIO["file_io.py"]:::core
            GetIndex["get_index.py"]:::core
            Register["register.py"]:::core
        end

        AgentBase -->|decomposes tasks| ExecAgents
        RegularAgent -->|uses prompts| Prompts
        ExecAgents -->|selects actions| Actions
        Actions -->|calls| LLM
        LLM -->|API calls| Proxy
        Proxy -->|forwards to| ExternalOpenAI
        Proxy -->|forwards to| ExternalAnthropic
        Proxy -->|forwards to| ExternalSerpAPI
        ExecAgents -->|updates| Graph
        Graph -->|stores state| Memory
        Graph -->|caches results| Cache
        Utilities -->|support| Engine
    end

    Engine -->|"streams events"| Backend
    Backend -->|"SSE/WebSocket"| Frontend

    %% Data & Config
    TestData["test_data/"]:::data
    APIEnv["api_key.env"]:::data
    Scripts["start.sh<br/>setup_env.sh<br/>run_with_anaconda.sh"]:::data

    TestData --> Engine
    APIEnv --> Engine
    Scripts --> |"setup/run"| Engine

    %% External Services
    ExternalOpenAI[(OpenAI API)]:::external
    ExternalAnthropic[(Anthropic API)]:::external
    ExternalSerpAPI[(SerpAPI)]:::external

    %% Click Events
    click Frontend "https://github.com/principia-ai/writehere/blob/main/frontend/src/App.js"
    click Pages "https://github.com/principia-ai/writehere/tree/main/frontend/src/pages/"
    click Components "https://github.com/principia-ai/writehere/tree/main/frontend/src/components/"
    click APIUtil "https://github.com/principia-ai/writehere/blob/main/frontend/src/utils/api.js"
    click Backend "https://github.com/principia-ai/writehere/blob/main/backend/server.py"
    click ReqTxt "https://github.com/principia-ai/writehere/blob/main/backend/requirements.txt"
    click TestAPI "https://github.com/principia-ai/writehere/blob/main/backend/test_api.py"
    click Engine "https://github.com/principia-ai/writehere/blob/main/recursive/engine.py"
    click AgentBase "https://github.com/principia-ai/writehere/blob/main/recursive/agent/agent_base.py"
    click RegularAgent "https://github.com/principia-ai/writehere/blob/main/recursive/agent/agents/regular.py"
    click Prompts "https://github.com/principia-ai/writehere/tree/main/recursive/agent/prompts/"
    click Proxy "https://github.com/principia-ai/writehere/blob/main/recursive/agent/proxy.py"
    click Actions "https://github.com/principia-ai/writehere/tree/main/recursive/executor/actions/"
    click ExecAgents "https://github.com/principia-ai/writehere/tree/main/recursive/executor/agents/"
    click Schema "https://github.com/principia-ai/writehere/blob/main/recursive/executor/schema.py"
    click LLM "https://github.com/principia-ai/writehere/blob/main/recursive/llm/llm.py"
    click Graph "https://github.com/principia-ai/writehere/blob/main/recursive/graph.py"
    click Memory "https://github.com/principia-ai/writehere/blob/main/recursive/memory.py"
    click Cache "https://github.com/principia-ai/writehere/blob/main/recursive/cache.py"
    click Display "https://github.com/principia-ai/writehere/blob/main/recursive/utils/display.py"
    click FileIO "https://github.com/principia-ai/writehere/blob/main/recursive/utils/file_io.py"
    click GetIndex "https://github.com/principia-ai/writehere/blob/main/recursive/utils/get_index.py"
    click Register "https://github.com/principia-ai/writehere/blob/main/recursive/utils/register.py"
    click TestData "https://github.com/principia-ai/writehere/tree/main/test_data/"
    click APIEnv "https://github.com/principia-ai/writehere/blob/main/recursive/api_key.env.example"
    click Scripts "https://github.com/principia-ai/writehere/blob/main/start.sh"

    %% Styles
    classDef ui fill:#add8e6,stroke:#000,color:#000;
    classDef backend fill:#90ee90,stroke:#000,color:#000;
    classDef core fill:#ffa500,stroke:#000,color:#000;
    classDef external fill:#d3d3d3,stroke:#000,color:#000;
    classDef data fill:#f0f0f0,stroke:#000,color:#000;
```
