How principia-ai/writehere works
Project Type and Purpose
• This is a full-stack application with two operating modes:
a) CLI-only engine for batch processing of stories/reports (via recursive/engine.py).
b) Web visualization interface (React frontend + Flask backend).
• Core purpose: orchestrate recursive task decomposition, heterogeneous integration (retrieval, reasoning, composition), and dynamic adaptation via LLMs (OpenAI, Anthropic, SerpAPI).
Top-Level Components
A. Frontend (React)
• Runs on Node.js (port 3000 by default).
• Key folders: frontend/src/pages (Home, About, ReportGeneration, StoryGeneration, Results), frontend/src/components (Header, Footer, LiveTaskList, TaskGraph, HistoryPanel).
• Communicates with Flask API over HTTP.
B. Backend API Server (Flask)
• Implements REST endpoints in backend/server.py.
• Dependencies in backend/requirements.txt.
• Exposes endpoints for:
• starting a new run
• streaming task updates/status/results
• serving logs/debug info
C. Core Engine (Python “recursive” package)
• engine.py: entry point for CLI and backend invocation.
• Agent Layer (recursive/agent):
• agent_base.py: base agent interface
• agents/regular.py: default agent implementation
• prompts/…: organized per use-case (report vs story).
• Executor Layer (recursive/executor):
• actions/: action executors (bing_browser, builtin, parser, etc.)
• agents/: wrap LLM calls (claude_fc_react, base_agent).
• schema.py: defines task schema.
• LLM Integrations (recursive/llm/llm.py): abstracts OpenAI/Anthropic API calls via recursive/proxy.py.
• Supporting Modules:
• memory.py: memory management
• cache.py: reusable results cache
• graph.py: in-memory task graph representation
• utils/: display, file I/O, indexing, registration
• Scripts: test_run_report.sh, test_run_story.sh for sample runs.
D. Data Sources & Config
• test_data/: example JSONL and PDF inputs.
• recursive/api_key.env: holds LLM API keys.
• Config & setup scripts: start.sh, setup_env.sh, run_with_anaconda.sh.
E. External Services
• OpenAI (GPT family)
• Anthropic (Claude)
• SerpAPI (web search)
Relationships & Data Flows
• User → Frontend: via browser UI → submits story/report request.
• Frontend → Backend: HTTP calls to Flask endpoints.
• Backend → Core Engine: invokes engine.py (in-process or subprocess), passing file paths, mode flags, port/socket for updates.
• Engine
• Loads input data → builds initial task graph → stores in memory/cache.
• Agent decomposes tasks (recursive planning) → submits subtasks to Executor.
• Executor selects action (retrieval, reasoning, composition) → calls LLM or built-in action.
• LLM calls go through proxy → external API → returns content.
• Engine updates graph state → emits events/status.
• Engine → Backend: streams task-level events (ready, in-progress, completed) and results.
• Backend → Frontend: server-sent events or WebSocket pushes updates to React.
• Frontend: updates UI components (LiveTaskList, TaskGraph) in real-time; final results shown on ResultsPage.
Architectural Patterns & Principles
• Modular layering: clear separation between agent, executor, LLM integrations, memory, and I/O.
• Event-driven task graph: central graph.py manages state and event propagation.
• Pluggable prompts & actions: easy to add new prompt templates or actions via recursive/agent/prompts and recursive/executor/actions.
• Dual-mode operation: same engine core supports CLI batch runs and live web UI.