How ishaan1013/shadow works
Project type and purpose
• Shadow is a full-stack, open-source “background coding agent” platform.
• It consists of:
• A Next.js frontend (real-time chat, file explorer, terminal emulator)
• A Node.js orchestrator backend (task management, LLM integration, WebSocket, API)
• A Sidecar service (isolated file/command execution in containers)
• A PostgreSQL database (via Prisma)
• Auxiliary packages (shared types, DB client, command-security, ESLint/TS configs)
a. Client Layer
• Developer’s browser (UI)
• Next.js Frontend (apps/frontend)
– HTTP-REST endpoints (/api/...)
– WebSocket connection for real-time task updates
b. Orchestrator Layer (Server)
• API Gateway & Middleware
• WebSocket Server (task socket)
• Task Manager & Workflow Engine
• Agent Layer (LLM integration, “braintrust,” PR generation)
• Execution Abstraction
– LocalMode: Local-FS Git service & tool executor
– RemoteMode: Kubernetes orchestration → Kata containers
• Indexing Service
– Code chunker, semantic embedding, Pinecone vector store
– Shadow Wiki generator
• GitHub Integration (OAuth, repo/branch/issues API)
• Database Services (Prisma client, task records, user settings, memories)
• Auxiliary Services (memory cleanup, checkpoint, pr-manager, summarization)
c. Sidecar Layer (apps/sidecar)
• REST APIs for file ops, git ops, terminal, workspace manager, filesystem watcher
• Runs inside VM/container, invoked by Server’s remote execution
d. Data & External Services
• PostgreSQL (packages/db)
• Vector DB (Pinecone) for semantic search
• LLM Providers (OpenAI, Anthropic, OpenRouter)
• GitHub API
Relationships & data flows
• Browser ⇄ Frontend via HTTPS & WebSocket
• Frontend → Server HTTP API (task creation, GitHub queries, user settings)
• Frontend ←→ Server WebSocket (streaming chat, task status)
• Server ⇄ PostgreSQL (Prisma ORM)
• Server → Pinecone (semantic embeddings)
• Server → GitHub (REST API calls, webhooks)
• Server ↔ Sidecar (HTTPS REST for file/git/exec when in remote mode)
• Server → Kubernetes API (create/teardown Kata containers)
• Server → LLM Providers (chat completions, streaming responses)
Architectural patterns & principles
• Layered architecture: UI, API/Gateway, Business logic, Data layer
• Abstraction of execution modes (Strategy pattern for local vs remote)
• Micro-service-style Sidecar for isolation
• Event-driven streaming via WebSockets
• Modular packages for shared types, security, and DB client
• Kubernetes + Kata containers for hardware-level isolation
Detailed elements
• Frontend: Next.js, React components, API routes, Socket client
• Server: Express/Koa (or custom) HTTP server, WebSocket server, Prisma client, agent modules, indexing modules, execution modules
• Sidecar: Express REST API, file/git/terminal services, workspace manager
• Data stores: PostgreSQL, Pinecone
• External: GitHub API, LLM endpoints
• Deployment: Dockerfiles for apps/server and apps/sidecar, Kubernetes YAML for remote agents