How saihaj/doge-ai works
Project Type and Purpose
• DOGEai is a microservices-based, full-stack monorepo that:
• Crawls U.S. Congress bill data (“crawler” service).
• Enriches and stores that data in a Postgres database (“database”).
• Runs an autonomous AI “agent” to analyze that data, tweet findings, and respond on X.
• Exposes multiple front-ends:
– A public Next.js “website” (dogexbt.ai)
– A “chat” interface for conversational access
– An internal “dashboard” for managing agents, prompts, and manual knowledge bases.
• Uses Inngest for queue orchestration, Envoy as an edge proxy, and a log-shipper for observability.
File Structure
• Top-level directories map 1:1 to services/components:
• crawler/ – bill scraping & enrichment
• agent/ – Inngest workflows + Twitter ingestion/reply
• database/ – Drizzle ORM migrations + client
• website/ – Next.js public website
• chat/ – Next.js chat UI (wrangler/cloudflare worker)
• dashboard/ – Next.js internal admin UI
• envoy/ – Envoy proxy config & Docker
• log-shipper/ – collects and ships logs
• Shared config: pnpm workspace, Dockerfiles, Fly.io manifests.
README for Additional Insights
• External dependencies: Congress API, Twitter/X API, Inngest service, PostHog (analytics).
• Cron jobs and scheduled scrapes.
• Drizzle ORM snapshots for migrations.
Main Components & Their Interactions
a. Components
– External Services:
• U.S. Congress API
• Twitter/X API
• Inngest Queue Service
• PostHog Analytics (optional)
– Crawler Service (Node.js on Fly.io)
– Agent Service (Node.js on Fly.io with Inngest)
– Database (Postgres with Drizzle ORM)
– Envoy Edge Proxy (routes HTTP/TCP traffic)
– Log Shipper (aggregates logs to external system)
– Front-ends:
• Website (Next.js)
• Chat UI (Next.js / Wrangler)
• Dashboard (Next.js)
b. Data Flows and Relationships
1. Crawler
• Cron trigger → Congress API (list bills) → Inngest queue → bill enrichment tasks
• Enriched data → writes to Database
2. Agent
• Cron / Streaming ingest from X → Inngest “decision” queue
• ProcessInteraction → calls context/prompts in Database → generate reply → POST to X
• In parallel, manual-KB and prompt registry lookups in Database
3. Front-ends
• Client (browser) → Envoy → respective Next.js service
• Next.js → tRPC / REST endpoints on Agent (or separate API) → Database calls
4. Envoy
• Routes incoming HTTP to website, chat, dashboard, and agent API endpoints
5. Log Shipper
• Tails service logs (crawler, agent, front-ends) → external logging/monitoring
Architectural Patterns & Design Principles
• Microservices: each directory is an independently deployable service
• Event-driven queues (Inngest) for fault-tolerant, retriable workflows
• Shared database for canonical state (Drizzle-managed migrations)
• Edge routing via Envoy (service mesh / reverse proxy)
• Containerization (Docker + Fly.io)
• Modular front-ends (Next.js) using tRPC to decouple UI from backend logic