How firebird-technologies/auto-analyst works
Project Type & Purpose
• This is a full-stack, modular AI data-science platform.
• The backend (auto-analyst-backend) is a Python/FastAPI (or Flask) service that orchestrates LLM‐based “agents” using DSPy, stores sessions, code, feedback in a SQL database, and runs scheduled reports.
• The frontend (auto-analyst-frontend) is a Next.js 13 app with:
• A React UI for chat, code editing, analytics dashboards
• Serverless API routes (Next.js API) handling auth (NextAuth), billing (Stripe), credits & trials, webhooks, and proxying chat requests.
• Infrastructure is managed via Terraform, CI pipelines (.github/workflows), and Docker.
Top‐Level Containers
A. Frontend Container (Next.js App)
• Client Browser (React UI)
• Next.js Server (Node.js runtime)
• API Routes for auth, subscription, credits, webhooks, analytics hub
• Proxy Module (app/api/chat? calls the Python backend)
• State Stores (Zustand in client / Redis on server for session & credit data)
B. Backend Container (Python Service)
• HTTP Layer (FastAPI or Flask in app.py behind Uvicorn/Gunicorn)
• Agent Orchestrator (DSPy planner, multi-agent workflows)
• Agents Module (preprocessing, statistical, ML, viz)
• Managers & Routes (chat_manager, session_manager, REST endpoints)
• Database Layer (SQLAlchemy models → PostgreSQL or similar)
• Utils (logger, model_registry, report generator)
• Scripts & Schedulers (daily reports, DB init, agent template populators)
C. Data Stores & External Services
• SQL Database (users, sessions, feedback, templates)
• Redis Cache (sessions, credit counts, pub/sub for real-time)
• Object Storage (for uploaded CSVs/XLSX files)
• LLM Providers (OpenAI, Anthropic, Deepseek)
• Stripe API (payments, subscriptions, webhooks)
• OAuth Providers (Google, GitHub via NextAuth)
• Terraform-provisioned Infrastructure (AWS RDS, ElastiCache, S3, Vercel, Hugging Face Spaces)
Relationships & Data Flow
• Client → Next.js UI: user interactions (chat, uploads, settings)
• Next.js UI ↔ Next.js Server: REST/Server Components for auth, billing, analytics
• Next.js Server → Stripe/OAuth/Redis: subscription checks, credit updates
• Next.js Server → Python Backend API: chat queries, dataset uploads, agent routing
• Python Backend → LLM Providers: send prompt & code, receive results
• Python Backend → SQL DB: persist sessions, feedback, user data
• Python Backend → Object Storage: store & retrieve datasets
• Python Backend → Redis (optional): shared dataframes, intermediate caches
• Python Backend → Next.js (via API response): returns chat messages, code, charts
• Terraform & CI/CD → Provisioning & Deployment: Vercel for frontend, Docker Image → HF Spaces or Kubernetes for backend
Architectural Patterns & Principles
• Containerized micro-service style: clear separation of frontend and backend containers
• Serverless API layer in frontend: isolates auth/billing logic from UI
• Modular agent architecture: each agent is a DSPy Signature class, pluggable
• Shared Planner Pattern: central planner orchestrates multi-agent workflows
• Dataflow pipelines: user uploads → metadata extraction → agent processing → result