How kanaries/rath works
Project type and purpose
• This is a monorepo for RATH, an open-source, full-stack, automated EDA & BI platform. It includes:
• A web client (React/TypeScript, MobX, Web Workers, Vega)
• A desktop client (Electron/Node.js)
• Multiple backend microservices (Python Flask, AWS Lambda functions)
• Data connectors to SQL/NoSQL sources and various analytics engines
File structure
• Root/
– Docker orchestration: docker-compose.yml, client.dockerfile, Nginx config
– CI/CD: .github/workflows/*.yml, buildspec.yml
• apps/
– test-datasets/: sample SQL files
• packages/ (Lerna/workspaces)
– rath-client/: full React SPA with extensive modules, pages, stores, services, and Web Workers
– olap-connector/: backend–for–frontend proxy layer to various OLAP engines
• services/
– connector/: Python Flask + AWS Lambda handlers for database connectivity
– causal-service/: Python microservice for causal discovery (multiple algorithms)
– narrative-service/: Jupyter/Flask service for narrative insight generation
– prediction/: Python classification & regression service
– text-pattern-service/: Python service for pattern extraction
README for insights
• Purpose: automate EDA workflow (AutoPilot, Copilot, Natural-Language Interface)
• Features: data profiling, data wrangling, one-click data insight, data painting, causal analysis, dashboard generation
• Supported Sources: Athena, Redshift, Spark, Doris, ClickHouse, MySQL, PostgreSQL, etc.
Main components & interactions
a. Components
• Desktop Client (Electron)
• Web Client (React SPA + Web Workers)
• API Gateway / Reverse Proxy (Nginx)
• OLAP Connector Service (proxies queries to external DBs)
• Microservices:
– Causal-Service
– Narrative-Service
– Prediction-Service
– Text-Pattern-Service
• External Data Sources (various SQL/NoSQL DBs)
b. Relationships & data flow
1. Client (desktop or browser) issues HTTP/WS requests → Nginx
2. Nginx routes to:
– OLAP Connector for CRUD and query operations on external DBs
– Causal/Narrative/Prediction/Text services for analysis tasks
3. OLAP Connector translates requests → executes on target DB → returns results
4. Analysis services compute models/insights and return structured JSON
5. Client receives data → employs Web Workers & Vega libs to render visualizations
c. Architectural patterns & design principles
• Monorepo & workspaces for shared code and consistent versioning
• Microservices: separation of analytical engines by concern (causal, narrative, prediction…)
• Backend-for-Frontend (OLAP connector) abstracts DB connectivity
• Containerization (Docker Compose) for local integration testing
• Web Workers to offload heavy data processing in the browser
• MVVM/MVC-like separation in the React client (pages, components, stores, services)
d. Key technologies/frameworks
• Frontend: React, TypeScript, MobX, Webpack, Babel, CSS modules, Web Workers
• Visualization: Vega-Lite scenegraph, WebGL renderer
• Desktop: Electron, NSIS installer
• Backend: Python (Flask), AWS Lambda handlers, Jupyter notebooks (narrative service)
• Containerization: Docker, Nginx, Docker-Compose
• CI/CD: GitHub Actions, AWS CodeBuild (buildspec.yml)
Detailed capture of essential elements
• monorepo structure at a high level (packages vs apps vs services)
• Web Worker threads offloading data parsing/analysis
• Docker Compose orchestration layering (Nginx + services)
• AWS Lambda functions under Connector as a sub-component