# darshan-stack/cursor-for-ml architecture

> Architecture diagram of the GitHub repository darshan-stack/cursor-for-ml, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-09-25). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/darshan-stack/cursor-for-ml
- Repository: https://github.com/darshan-stack/cursor-for-ml

## Overview

##### Project Type and Purpose
- This is a full-stack, microservices-based Machine Learning Platform.
- It provides an AI-powered React IDE, end-to-end workflow orchestration, multi-model coordination, continual learning, governance, and extensible plugins.
- It is production-grade, supports Docker and native deployment, and includes CI/CD pipelines.

##### File Structure
- Top-level folders:
- interfaces/react – React AI IDE (Monaco Editor, Prompt Bar, Logs Panel, Project Explorer)
- interfaces/web – Server-rendered Web UI for data management, workflow creation, monitoring (Python + templates + static assets)
- services – Individual microservices (data-management, workflow-orchestration, prompt-orchestrator, continual-learning, governance, multi-model orchestrator)
- models – Core model orchestration and entry point
- workflows – Workflow engine definitions and YAML samples
- plugins – Base classes for custom extensions
- config – Global settings (config/settings.py)
- scripts – Deployment and native run scripts
- docker-compose.yml – Multi-container orchestrator for local Docker deployment
- pids – PID files for process management
- tests – Unit/integration tests per service

##### Extract Architectural Insights from README
- Microservices pattern with REST APIs and optional message queues.
- Core services: Data Management, Workflow Orchestration, Multi-Model Orchestration, Governance, Web Interface, Prompt Orchestrator, Continual Learning.
- Reactive frontend (React) talking to backend services via REST or WebSocket.
- Persistent layers: MongoDB, PostgreSQL, Redis.
- External integrations: OpenAI API, HuggingFace, plugin connectors.

##### Main Components
a. Frontend Layer
– React AI IDE (interfaces/react)
– Web UI (interfaces/web)
b. Backend Microservices
– Data Management Service
– Workflow Orchestration Service
– Prompt Orchestrator Service
– Multi-Model Orchestration Service
– Continual Learning Service
– Governance Service
c. Workflow Engine
– workflows/main.py & orchestrator
– Sample workflows (.yaml)
d. Model Orchestration
– models/orchestration/multi_model_orchestrator.py
– models/main.py entry
e. Persistence & Cache
– MongoDB (data store & versioning)
– PostgreSQL (relational data, metadata)
– Redis (caching, queues)
f. External Services
– OpenAI / HuggingFace APIs
– Plugin endpoints (custom DataConnectorPlugin, etc.)
g. DevOps & CI/CD
– Docker Compose, GitHub Actions, Kubernetes (k8s/)
– scripts/* for deploy, run, stop

##### Relationships & Interactions
- User interacts with React IDE → sends prompt or code edits → HTTP/WebSocket to Prompt Orchestrator / Workflow Orchestration
- Workflow Orchestration calls Data Management, Model Training (via Multi-Model Orchestration), Continual Learning, Governance
- Services communicate via REST endpoints (and optionally message broker/Redis)
- Data Management persists datasets to MongoDB/PostgreSQL; caches jobs/status in Redis
- Multi-Model Orchestrator coordinates calls to multiple AI models (OpenAI/HuggingFace)
- Governance service intercepts calls for RBAC, audit logging
- Web UI allows manual data management, monitoring, and triggers workflows via REST API

##### Architectural Patterns & Design Principles
- Microservices: Each service is independently deployable (own Dockerfile or Python module)
- Layered Architecture: UI → API Gateway (if used) / Orchestrator → Services → Datastores
- Plugin-based Extensibility: services/plugins provide extension points
- Workflow-Driven: YAML definitions drive the sequence of calls across services
- Event-Driven / Orchestration: central orchestrator triggers service chains
- Security & Compliance: governance sits as a cross-cutting concern

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% Frontend Layer
    subgraph "Frontend Layer"
        ReactAI["React AI IDE"]:::frontend
        WebUI["Web UI"]:::frontend
    end

    %% Backend Services
    subgraph "Backend Services"
        DataMgmt["Data Management Service (port 8001)"]:::service
        WorkflowService["Workflow Orchestration Service (port 8002)"]:::orchestrator
        PromptOrch["Prompt Orchestrator Service (port 8003)"]:::service
        MultiModelSvc["Multi-Model Orchestration Service (port 8004)"]:::service
        ContinualLearn["Continual Learning Service (port 8005)"]:::service
        GovernanceSvc["Governance Service (port 8006)"]:::service
    end

    %% Workflow Engine Definitions
    subgraph "Workflow Engine"
        WE_Main["main.py"]:::orchestrator
        WE_Orch["orchestrator.py"]:::orchestrator
        WE_Sample["sample_workflow.yaml"]:::orchestrator
    end

    %% Model Orchestration
    subgraph "Model Orchestration"
        ModelMain["models/main.py"]:::service
        MultiModelMod["multi_model_orchestrator.py"]:::service
    end

    %% Configuration & Plugins
    subgraph "Configuration & Plugins"
        Config["Global Configuration"]:::service
        Plugins["Plugin Base Classes"]:::service
    end

    %% Persistence & Cache
    subgraph "Persistence & Cache"
        Mongo["MongoDB"]:::db
        Postgres["PostgreSQL"]:::db
        Redis["Redis"]:::db
    end

    %% External Services
    subgraph "External Services"
        OpenAI["OpenAI API"]:::external
        HF["HuggingFace API"]:::external
    end

    %% Deployment & CI/CD
    subgraph "Deployment & CI/CD"
        DockerCompose["Docker Compose"]:::cicd
        Kubernetes["Kubernetes"]:::cicd
        Scripts["Native Scripts"]:::cicd
        CICD["GitHub Actions"]:::cicd
    end

    %% Data Flows
    ReactAI -->|HTTP/WebSocket| PromptOrch
    ReactAI -->|HTTP/WebSocket| WorkflowService
    WebUI -->|REST| DataMgmt
    WebUI -->|REST| WorkflowService

    WorkflowService -->|invoke definitions| WE_Main
    WE_Main -->|call orchestrator| WE_Orch
    WE_Orch -->|uses sample| WE_Sample

    WorkflowService -->|invoke| DataMgmt
    WorkflowService -->|orchestrate| MultiModelSvc
    WorkflowService -->|orchestrate| ContinualLearn
    WorkflowService -->|audit| GovernanceSvc

    PromptOrch -->|calls| MultiModelSvc

    DataMgmt -->|persist| Mongo
    DataMgmt -->|persist| Postgres
    DataMgmt -->|cache| Redis

    MultiModelSvc -->|calls| OpenAI
    MultiModelSvc -->|calls| HF

    ModelMain -->|entry for| MultiModelMod

    %% Config & Plugins links
    Config --> DataMgmt
    Config --> WorkflowService
    Config --> PromptOrch
    Config --> MultiModelSvc
    Config --> ContinualLearn
    Config --> GovernanceSvc

    Plugins --> DataMgmt
    Plugins --> WorkflowService

    %% CI/CD & Deployment
    CICD --> DockerCompose
    CICD --> Kubernetes
    Scripts --> DockerCompose
    Scripts --> Kubernetes

    %% Click Events
    click ReactAI "https://github.com/darshan-stack/cursor-for-ml/tree/main/interfaces/react"
    click WebUI "https://github.com/darshan-stack/cursor-for-ml/tree/main/interfaces/web"
    click DataMgmt "https://github.com/darshan-stack/cursor-for-ml/blob/main/services/data-management/main.py"
    click WE_Main "https://github.com/darshan-stack/cursor-for-ml/blob/main/workflows/main.py"
    click WE_Orch "https://github.com/darshan-stack/cursor-for-ml/blob/main/workflows/orchestrator.py"
    click WE_Sample "https://github.com/darshan-stack/cursor-for-ml/blob/main/workflows/sample_workflow.yaml"
    click PromptOrch "https://github.com/darshan-stack/cursor-for-ml/blob/main/services/prompt-orchestrator/main.py"
    click ContinualLearn "https://github.com/darshan-stack/cursor-for-ml/blob/main/services/continual-learning/main.py"
    click GovernanceSvc "https://github.com/darshan-stack/cursor-for-ml/blob/main/services/governance/main.py"
    click MultiModelMod "https://github.com/darshan-stack/cursor-for-ml/blob/main/models/orchestration/multi_model_orchestrator.py"
    click ModelMain "https://github.com/darshan-stack/cursor-for-ml/blob/main/models/main.py"
    click Config "https://github.com/darshan-stack/cursor-for-ml/blob/main/config/settings.py"
    click Plugins "https://github.com/darshan-stack/cursor-for-ml/blob/main/plugins/__init__.py"
    click DockerCompose "https://github.com/darshan-stack/cursor-for-ml/blob/main/docker-compose.yml"
    click Scripts "https://github.com/darshan-stack/cursor-for-ml/blob/main/scripts/deploy.sh"
    click CICD "https://github.com/darshan-stack/cursor-for-ml/blob/main/.github/workflows/ci.yml"

    %% Styles
    classDef frontend fill:#ccf,stroke:#333,stroke-width:1px;
    classDef service fill:#cfc,stroke:#333,stroke-width:1px;
    classDef db fill:#fcc,stroke:#333,stroke-width:1px,shape:cylinder;
    classDef external fill:#ddd,stroke:#333,stroke-width:1px,shape:cloud;
    classDef orchestrator fill:#ffc,stroke:#333,stroke-width:1px;
    classDef cicd fill:#fbd,stroke:#333,stroke-width:1px;
```
