How ucbepic/docetl works
Main System Components
The system should be divided into three major sections:
a) Core Engine (docetl package):
• Operations Layer (docetl/operations/*)
• Optimization Layer (docetl/optimizers/*)
• Pipeline Execution Layer (docetl/runner.py, docetl/builder.py)
• Configuration Management (docetl/config_wrapper.py)
b) Web Interface (website directory):
• Next.js Frontend (website/src/app)
• API Routes (website/src/app/api)
• WebSocket Communication (website/src/contexts/WebSocketContext.tsx)
c) Backend Server (server directory):
• FastAPI Server (server/app)
• Route Handlers (server/app/routes)
Component Relationships
a) Data Flow:
• Document Input → Core Engine → Processed Output
• UI Components → API Routes → Backend Server → Core Engine
• WebSocket connections between Frontend and Backend for real-time updates
b) Integration Points:
• Configuration flow between UI and Core Engine
• Pipeline execution results from Core Engine to UI
• File system interactions for document storage
Specific Implementation Details
a) Core Processing Components (Hexagons):
• Operation Nodes (Map, Reduce, Filter, etc.)
• Optimization Nodes
• Pipeline Execution Engine
b) Interface Components (Rectangles):
• UI Components
• API Endpoints
• Configuration Management
• Document Storage
• Pipeline Configurations
• Execution State
Architecture Patterns to Highlight
a) Pipeline Pattern:
• Flow of operations in the core engine
• Optimization layer's interaction points
b) Client-Server Architecture:
• Clear separation between frontend, backend, and core engine
• API boundaries and communication protocols
Technical Stack Indicators
• Python (Core Engine)
• Next.js (Frontend)
• FastAPI (Backend)
• WebSocket (Real-time Communication)
• OpenAI/LLM Integration Points
Optimization Flow
• Optimizer
• Join Optimizer
• Reduce Optimizer
• Their interaction with the main pipeline
[UI Layer]
• ↕
[API Layer]
• ↕
[Backend Server Layer]
• ↕
[Core Engine Layer]
• ↕
[Storage Layer]
Important Considerations:
• Operation execution flow
• Optimization process
• Document processing pipeline
• UI interaction flow
This system design should capture both the high-level architecture and the detailed component interactions while maintaining clarity and readability.