# saihaj/doge-ai architecture

> Architecture diagram of the GitHub repository saihaj/doge-ai, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-05-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/saihaj/doge-ai
- Repository: https://github.com/saihaj/doge-ai

## Overview

##### Project Type and Purpose
- DOGEai is a microservices-based, full-stack monorepo that:
- Crawls U.S. Congress bill data (“crawler” service).
- Enriches and stores that data in a Postgres database (“database”).
- Runs an autonomous AI “agent” to analyze that data, tweet findings, and respond on X.
- Exposes multiple front-ends:
– A public Next.js “website” (dogexbt.ai)
– A “chat” interface for conversational access
– An internal “dashboard” for managing agents, prompts, and manual knowledge bases.
- Uses Inngest for queue orchestration, Envoy as an edge proxy, and a log-shipper for observability.

##### File Structure
- Top-level directories map 1:1 to services/components:
- crawler/       – bill scraping & enrichment
- agent/         – Inngest workflows + Twitter ingestion/reply
- database/      – Drizzle ORM migrations + client
- website/       – Next.js public website
- chat/          – Next.js chat UI (wrangler/cloudflare worker)
- dashboard/     – Next.js internal admin UI
- envoy/         – Envoy proxy config & Docker
- log-shipper/   – collects and ships logs
- Shared config: pnpm workspace, Dockerfiles, Fly.io manifests.

##### README for Additional Insights
- External dependencies: Congress API, Twitter/X API, Inngest service, PostHog (analytics).
- Cron jobs and scheduled scrapes.
- Drizzle ORM snapshots for migrations.

##### Main Components & Their Interactions
a. Components
– External Services:
- U.S. Congress API
- Twitter/X API
- Inngest Queue Service
- PostHog Analytics (optional)
– Crawler Service (Node.js on Fly.io)
– Agent Service (Node.js on Fly.io with Inngest)
– Database (Postgres with Drizzle ORM)
– Envoy Edge Proxy (routes HTTP/TCP traffic)
– Log Shipper (aggregates logs to external system)
– Front-ends:
- Website (Next.js)
- Chat UI (Next.js / Wrangler)
- Dashboard (Next.js)
b. Data Flows and Relationships
1. Crawler
- Cron trigger → Congress API (list bills) → Inngest queue → bill enrichment tasks
- Enriched data → writes to Database
2. Agent
- Cron / Streaming ingest from X → Inngest “decision” queue
- ProcessInteraction → calls context/prompts in Database → generate reply → POST to X
- In parallel, manual-KB and prompt registry lookups in Database
3. Front-ends
- Client (browser) → Envoy → respective Next.js service
- Next.js → tRPC / REST endpoints on Agent (or separate API) → Database calls
4. Envoy
- Routes incoming HTTP to website, chat, dashboard, and agent API endpoints
5. Log Shipper
- Tails service logs (crawler, agent, front-ends) → external logging/monitoring

##### Architectural Patterns & Design Principles
- Microservices: each directory is an independently deployable service
- Event-driven queues (Inngest) for fault-tolerant, retriable workflows
- Shared database for canonical state (Drizzle-managed migrations)
- Edge routing via Envoy (service mesh / reverse proxy)
- Containerization (Docker + Fly.io)
- Modular front-ends (Next.js) using tRPC to decouple UI from backend logic

## Diagram (Mermaid)

```mermaid
flowchart TB
    %% External services
    subgraph "External Services"
        Congress["Congress API"]:::external
        XAPI["Twitter/X API"]:::external
        Inngest["Inngest Queue Service"]:::external
        PostHog["PostHog Analytics"]:::external
    end

    %% Core infrastructure
    subgraph "Core Infrastructure"
        DB["Postgres + Drizzle ORM"]:::database
        Envoy["Envoy Edge Proxy"]:::infra
        LogShipper["Log Shipper"]:::infra
    end

    %% Microservices layer
    subgraph "Microservices"
        Crawler["Crawler Service"]:::service
        Agent["Agent Service"]:::service
    end

    %% Front-end UIs
    subgraph "Front-ends"
        Website["Public Website"]:::frontend
        Chat["Chat Interface"]:::frontend
        Dashboard["Internal Dashboard"]:::frontend
    end

    %% Shared monorepo config
    subgraph "Shared Monorepo Config"
        Workspace["pnpm-workspace.yaml"]:::config
        RootPkg["package.json"]:::config
        Patches["patches/"]:::config
        CIWorkflow[".github/workflows/ci.yaml"]:::config
    end

    %% Cron triggers
    CronCrawler(["Cron"]):::infra
    CronAgent(["Cron"]):::infra

    %% Data flows - Crawler
    CronCrawler -->|sched| Crawler
    Crawler -->|"list bills"| Congress
    Congress -->|"bill data"| Crawler
    Crawler -->|"enqueue bills"| Inngest
    Inngest -->|"tasks"| Crawler
    Crawler -->|logs| LogShipper

    %% Data flows - Agent
    CronAgent -->|sched/stream| Agent
    XAPI -->|streaming| Agent
    Agent -->|"decision queue"| Inngest
    Inngest -->|"trigger workflows"| Agent
    Agent -->|reply tweets| XAPI
    Agent -->|analytics| PostHog
    Agent <-->|"read/write"| DB
    Agent -->|logs| LogShipper

    %% Front-end traffic via Envoy
    Website -->|HTTP| Envoy
    Chat -->|HTTP| Envoy
    Dashboard -->|HTTP| Envoy
    Envoy -->|route to UI| Website
    Envoy -->|route to UI| Chat
    Envoy -->|route to UI| Dashboard
    Envoy -->|route to API| Agent
    Website -->|tRPC/REST| Agent
    Chat -->|tRPC/REST| Agent
    Dashboard -->|tRPC/REST| Agent
    Dashboard -->|DB access| DB
    Website -->|logs| LogShipper
    Chat -->|logs| LogShipper
    Dashboard -->|logs| LogShipper
    Envoy -->|logs| LogShipper

    %% Click Events
    click Crawler "https://github.com/saihaj/doge-ai/tree/main/crawler/"
    click Agent "https://github.com/saihaj/doge-ai/tree/main/agent/"
    click DB "https://github.com/saihaj/doge-ai/tree/main/database/"
    click Website "https://github.com/saihaj/doge-ai/tree/main/website/"
    click Chat "https://github.com/saihaj/doge-ai/tree/main/chat/"
    click Dashboard "https://github.com/saihaj/doge-ai/tree/main/dashboard/"
    click Envoy "https://github.com/saihaj/doge-ai/tree/main/envoy/"
    click LogShipper "https://github.com/saihaj/doge-ai/tree/main/log-shipper/"
    click Workspace "https://github.com/saihaj/doge-ai/blob/main/pnpm-workspace.yaml"
    click RootPkg "https://github.com/saihaj/doge-ai/blob/main/package.json"
    click Patches "https://github.com/saihaj/doge-ai/tree/main/patches/"
    click CIWorkflow "https://github.com/saihaj/doge-ai/blob/main/.github/workflows/ci.yaml"

    %% Styles
    classDef external fill:#DDDDDD,stroke:#888888,color:#333333
    classDef service fill:#FFDD99,stroke:#CC9966,color:#663300
    classDef database fill:#99FF99,stroke:#66CC66,color:#006600
    classDef infra fill:#EEEEEE,stroke:#AAAAAA,color:#333333
    classDef frontend fill:#99CCFF,stroke:#6699CC,color:#003366
    classDef config fill:#F2F2F2,stroke:#BBBBBB,color:#333333
```
