# tensorplex-labs/dojo architecture

> Architecture diagram of the GitHub repository tensorplex-labs/dojo, generated by GitDiagram from its file tree, README and a sample of source files (last updated 2025-02-14). It is an AI-made overview: check details against the source.

- Interactive diagram: https://gitdiagram.com/tensorplex-labs/dojo
- Repository: https://github.com/tensorplex-labs/dojo

## Overview

##### Project Type and Purpose
- The README and file structure indicate that Dojo Subnet is an open platform for crowdsourcing high-quality, human-generated datasets. It serves two major roles – as a decentralized “miner” for task processing (data labeling) and a “validator” for quality control.
- The project also provides a CLI tool (dojo_cli.py) for managing API and subscription keys and has multiple Docker and make targets showing its containerized deployment methods.
- It integrates with external protocols (Bittensor subtensor nodes, external APIs, AWS S3 for dataset storage, etc.) and uses modern tools (conda, pm2, Docker Compose) to ensure smooth operations.

##### File Structure
- Top-Level Directories & Files:
- “commons”: Contains shared modules for APIs, caching, dataset generation, exceptions, and utility functions.
- “dojo”: Contains core business logic with modules for miner and validator configurations and protocols.
- “neurons”: Likely contains implementations for mining and validating functionalities.
- “simulator”: Used for simulation-based testing or alternative operational modes.
- “database”: Houses database connectors and migration scripts (SQL files). The presence of “migrations” and “schema.prisma” indicates ORMs and schema management.
- “docker” and multiple “docker-compose” YAML files: Provide various containers for different deployment methods (miner, validator, subtensor node, dataset service, etc.).
- “scripts”: Contains setup and validation scripts (for Docker, pm2, and checking compatibility).
- “tests”: Contains unit tests that validate core functionalities.
- Configuration and CI/CD: Directories like “.circleci” and “.github/workflows”, along with Makefile and pre-commit configurations, reveal a robust DevOps process.

##### Components and Their Relationships

A. Core Application & Business Logic:
- Miner Service: Contains components for both centralized and decentralized mining. Refer to files like main_miner.py, miners in the “neurons” and “dojo” folders.
- Validator Service: Built around similar principles to the miner. Main entry is clearly flagged by main_validator.py and .env.validator.
- Dojo CLI: Command-line interface for managing API keys and subscriptions, essential for setup and operations.

B. Shared Libraries and Common Utilities:
- Commons: Include API middleware, caching utilities, dataset generation modules, and scoring functions.
- Obfuscation and human-feedback: Modules to secure operations and ensure data authenticity.

C. Data Layer:
- Database: A Postgres-backed relational database (as indicated by “database” and prisma schema files) that stores miner/validator state, task data, and dataset collections.
- Migrations: SQL scripts to set up or update the database.

D. Containerization, Automation, and Deployment:
- Docker: Multiple Dockerfiles in “docker” and compose configurations (docker-compose.*.yaml) for separate services like miner, validator, dataset service, and the subtensor node.
- CI/CD: Directories for GitHub workflows and CircleCI pipelines ensure testing, linting, and deployment automation.
- Auto-Updater: Python script (auto_update.py) used with pm2 to monitor and update service code.

E. External Services and Dependencies:
- Bittensor Subtensor Integration: External blockchain endpoints for registration, wallet management, and network interaction.
- Redis, AWS S3, and Ethereum Node: For session management, dataset storage, and blockchain interactions respectively.
- External API Providers: For Openrouter, Langfuse, and other third-party integrations.
- UI & Subscription: Although not a full frontend app, the assets folder indicates UI assets (e.g., images, subscription components) used by the Dojo UI interface.

##### Relationships and Data Flow
- The CLI sets up and manages keys and config, feeding into the Miner and Validator services.
- The Miner service communicates with external APIs (via the commons libraries) and updates results in the database.
- The Validator service pulls tasks from the database, applies synthetic task generation and obfuscation routines, then writes back validated results.
- Docker containers encapsulate individual services and communicate on a virtual network, with API layers interfacing with the Dojo CLI and potentially a web-based UI.

##### Additional Inclusions
- Incorporate notes related to configuration (for example, multiple .env files reveal environment-specific configurations).
- Given the multi-faceted nature (CLI, automated update services, mining, validating), consider grouping components that have similar roles under “services”, “utilities”, or “infrastructure”.

## Diagram (Mermaid)

```mermaid
flowchart TD
    %% User & Configuration Layer
    subgraph "User & Configuration"
        DojoCLI("Dojo CLI"):::cli
    end

    %% Core Processing Layer
    subgraph "Core Processing"
        %% Miner Services
        subgraph "Miner Services"
            MinerMain("Main Miner"):::core
            MinerNeurons("Miner - Neurons"):::core
            MinerDojo("Miner - Dojo Base"):::core
        end
        %% Validator Services
        subgraph "Validator Services"
            ValidatorMain("Main Validator"):::core
            ValidatorNeurons("Validator - Neurons"):::core
            ValidatorDojo("Validator - Dojo Base"):::core
        end
        %% Simulation Mode
        subgraph "Simulation Mode"
            SimMiner("Simulator Miner"):::core
            SimValidator("Simulator Validator"):::core
        end
    end

    %% Shared & Data Layer
    subgraph "Shared & Data Layer"
        Commons("Commons & Utilities"):::shared
        Database("Database & Migrations"):::database
    end

    %% Infrastructure & DevOps Layer
    subgraph "Infrastructure & DevOps"
        Docker("Docker & Deployment"):::infra
        CICD("CI/CD Pipeline"):::infra
        AutoUpdater("Auto-Updater Service"):::infra
    end

    %% External Services Layer
    subgraph "External Services"
        Bittensor("Bittensor Subtensor"):::external
        AWSS3("AWS S3"):::external
        Ethereum("Ethereum Node"):::external
        Redis("Redis"):::external
        ExtAPIs("External APIs"):::external
    end

    %% Connections: User & Configuration to Core Processing
    DojoCLI -->|"configures"| MinerMain
    DojoCLI -->|"configures"| ValidatorMain

    %% Internal connections within Miner Services
    MinerMain -->|"orchestrates"| MinerNeurons
    MinerMain -->|"orchestrates"| MinerDojo

    %% Internal connections within Validator Services
    ValidatorMain -->|"orchestrates"| ValidatorNeurons
    ValidatorMain -->|"orchestrates"| ValidatorDojo

    %% Simulation Mode influences core processing
    SimMiner -->|"simulates"| MinerMain
    SimValidator -->|"simulates"| ValidatorMain

    %% Core Processing interactions with Commons & Utilities
    MinerMain -->|"uses"| Commons
    MinerNeurons -->|"uses"| Commons
    MinerDojo -->|"uses"| Commons
    ValidatorMain -->|"uses"| Commons
    ValidatorNeurons -->|"uses"| Commons
    ValidatorDojo -->|"uses"| Commons

    %% Core Processing interactions with Database
    MinerMain -->|"updates"| Database
    ValidatorMain -->|"updates"| Database
    Commons -->|"manages data"| Database

    %% Infrastructure & DevOps influencing operations
    Docker -->|"hosts"| MinerMain
    Docker -->|"hosts"| ValidatorMain
    Docker -->|"hosts"| AutoUpdater
    CICD -->|"deploys"| Docker
    AutoUpdater -->|"monitors"| MinerMain
    AutoUpdater -->|"monitors"| ValidatorMain

    %% Commons & Core External Integrations
    MinerMain -->|"integrates"| Bittensor
    ValidatorMain -->|"integrates"| Ethereum
    Commons -->|"calls"| ExtAPIs
    Commons -->|"stores"| AWSS3
    Commons -->|"caches"| Redis

    %% Styling Classes
    classDef cli fill:#ADD8E6,stroke:#000,stroke-width:2px;
    classDef core fill:#90EE90,stroke:#000,stroke-width:2px;
    classDef shared fill:#FFA07A,stroke:#000,stroke-width:2px;
    classDef database fill:#EEE8AA,stroke:#000,stroke-width:2px,stroke-dasharray: 5,5;
    classDef infra fill:#FFD700,stroke:#000,stroke-width:2px;
    classDef external fill:#FFB6C1,stroke:#000,stroke-width:2px;

    %% Click Events
    click DojoCLI "https://github.com/tensorplex-labs/dojo/blob/main/dojo_cli.py"
    click MinerMain "https://github.com/tensorplex-labs/dojo/blob/main/main_miner.py"
    click MinerNeurons "https://github.com/tensorplex-labs/dojo/blob/main/neurons/miner.py"
    click MinerDojo "https://github.com/tensorplex-labs/dojo/blob/main/dojo/base/miner.py"
    click ValidatorMain "https://github.com/tensorplex-labs/dojo/blob/main/main_validator.py"
    click ValidatorNeurons "https://github.com/tensorplex-labs/dojo/blob/main/neurons/validator.py"
    click SimMiner "https://github.com/tensorplex-labs/dojo/blob/main/simulator/miner.py"
    click SimValidator "https://github.com/tensorplex-labs/dojo/blob/main/simulator/validator.py"
    click Commons "https://github.com/tensorplex-labs/dojo/tree/main/commons/"
    click Database "https://github.com/tensorplex-labs/dojo/tree/main/database/"
    click Docker "https://github.com/tensorplex-labs/dojo/tree/main/docker/"
    click CICD "https://github.com/tensorplex-labs/dojo/tree/main/.circleci/"
    click AutoUpdater "https://github.com/tensorplex-labs/dojo/blob/main/auto_update.py"
```
