How dashaisoftware/dashai works
Project Type and Purpose
• DashAI is a full-stack web application combining a React-based Single-Page Application (SPA) frontend and a Python backend that exposes RESTful APIs. Its purpose is to provide a graphical toolbox for training, evaluating, explaining, and deploying AI/ML models.
High-Level Architecture Overview
• Frontend: DashAI/front (React + TypeScript, Yarn)
• Backend: DashAI/back (Python 3.8+, FastAPI-style / Pydantic schemas, SQLAlchemy, Alembic)
• Database: SQLite by default (configurable), managed via SQLAlchemy and Alembic migrations
• Job Queue: simple_job_queue or pluggable queues for async tasks (training, conversion, explanation)
• File Storage / Local Path: for datasets, experiments, runs, plugin artifacts
System Components
a. Client Layer (Browser)
• React SPA served by yarn start or static build
• UI Components: pages, modals, forms for datasets, experiments, explainers, explorers, predictions, results, plugins
• API Client Modules: src/api/*.ts (datasets.ts, experiment.ts, predict.ts, etc.)
b. API Gateway Layer
• DashAI/back/app.py or __main__.py launching FastAPI
• front_api.py (if any unified gateway)
• api_v0 and api_v1 routers mounted under /api/v0 and /api/v1
c. Business Logic Layer (Backend)
• Dataloaders: DashAI/back/dataloaders (CSV, JSON, Excel, audio, etc.)
• Converters: standardized ML preprocessing converters (scikit-learn, SMOTE, HuggingFace wrappers)
• Models: factory pattern producing BaseModel or wrappers (scikit-learn, transformers)
• Tasks & Jobs: BaseTask, BaseJob classes; converter_job, dataset_job, explainer_job, explorer_job, model_job, predict_job
• Exploration & Explainability: explorers (statistics, plots), explainers (SHAP, PDP, permutation importance)
• Optimizers: OptunaOptimizer, HyperoptOptimizer
• Plugins & Registry: dynamic component registration, tag-based discovery
d. Persistence and Async
• Database: SQLAlchemy models in dependencies/database/models.py
• Alembic migrations for schema evolution
• Job Queue: enqueued background tasks, can be in-process or external like Celery via abstraction
• Local Filesystem: storing raw datasets, intermediate artifacts, results
Relationships & Data Flows
1. User interacts with React SPA → invokes REST calls (HTTP/JSON)
2. API Gateway dispatches to appropriate router → Pydantic schema validation
3. Endpoints call into service layer modules (dataloaders, converters, jobs)
4. For synchronous endpoints (list datasets, retrieve results) → direct DB queries (SQLAlchemy)
5. For asynchronous operations (train model, convert dataset) → request enqueued in Job Queue → Worker processes job invoking Job classes → update database and file storage → status polled via endpoint
6. Explainability & Exploration jobs follow same async pattern
7. Frontend polls or subscribes to job status, then fetches results (graphs, tables)
Architectural Patterns & Design Principles
• Layered Architecture: Presentation (UI) → API Gateway → Service/Domain → Persistence
• Dependency Injection / Container Pattern: container.py builds config, registry, DB sessions
• Plug-in & Registry Pattern: components/plugins discovered and registered at runtime
• Versioned API: api_v0 vs api_v1 for backward compatibility
• Job Abstraction: uniform Job interface, pluggable job queue implementations
Technologies & Frameworks
• Frontend: React, Yarn, TypeScript, ESLint, Prettier, json-server for mock API
• Backend: Python, FastAPI (or Starlette), Pydantic, SQLAlchemy, Alembic, pytest, ruff, black
• Machine Learning: scikit-learn, imbalanced-learn, HuggingFace transformers, Optuna/Hyperopt
• Data Versioning: DVC integration at project root