How tensorplex-labs/dojo works
Project Type and Purpose
• The README and file structure indicate that Dojo Subnet is an open platform for crowdsourcing high-quality, human-generated datasets. It serves two major roles – as a decentralized “miner” for task processing (data labeling) and a “validator” for quality control.
• The project also provides a CLI tool (dojo_cli.py) for managing API and subscription keys and has multiple Docker and make targets showing its containerized deployment methods.
• It integrates with external protocols (Bittensor subtensor nodes, external APIs, AWS S3 for dataset storage, etc.) and uses modern tools (conda, pm2, Docker Compose) to ensure smooth operations.
File Structure
• Top-Level Directories & Files:
• “commons”: Contains shared modules for APIs, caching, dataset generation, exceptions, and utility functions.
• “dojo”: Contains core business logic with modules for miner and validator configurations and protocols.
• “neurons”: Likely contains implementations for mining and validating functionalities.
• “simulator”: Used for simulation-based testing or alternative operational modes.
• “database”: Houses database connectors and migration scripts (SQL files). The presence of “migrations” and “schema.prisma” indicates ORMs and schema management.
• “docker” and multiple “docker-compose” YAML files: Provide various containers for different deployment methods (miner, validator, subtensor node, dataset service, etc.).
• “scripts”: Contains setup and validation scripts (for Docker, pm2, and checking compatibility).
• “tests”: Contains unit tests that validate core functionalities.
• Configuration and CI/CD: Directories like “.circleci” and “.github/workflows”, along with Makefile and pre-commit configurations, reveal a robust DevOps process.
Components and Their Relationships
A. Core Application & Business Logic:
• Miner Service: Contains components for both centralized and decentralized mining. Refer to files like main_miner.py, miners in the “neurons” and “dojo” folders.
• Validator Service: Built around similar principles to the miner. Main entry is clearly flagged by main_validator.py and .env.validator.
• Dojo CLI: Command-line interface for managing API keys and subscriptions, essential for setup and operations.
B. Shared Libraries and Common Utilities:
• Commons: Include API middleware, caching utilities, dataset generation modules, and scoring functions.
• Obfuscation and human-feedback: Modules to secure operations and ensure data authenticity.
C. Data Layer:
• Database: A Postgres-backed relational database (as indicated by “database” and prisma schema files) that stores miner/validator state, task data, and dataset collections.
• Migrations: SQL scripts to set up or update the database.
D. Containerization, Automation, and Deployment:
• Docker: Multiple Dockerfiles in “docker” and compose configurations (docker-compose.*.yaml) for separate services like miner, validator, dataset service, and the subtensor node.
• CI/CD: Directories for GitHub workflows and CircleCI pipelines ensure testing, linting, and deployment automation.
• Auto-Updater: Python script (auto_update.py) used with pm2 to monitor and update service code.
E. External Services and Dependencies:
• Bittensor Subtensor Integration: External blockchain endpoints for registration, wallet management, and network interaction.
• Redis, AWS S3, and Ethereum Node: For session management, dataset storage, and blockchain interactions respectively.
• External API Providers: For Openrouter, Langfuse, and other third-party integrations.
• UI & Subscription: Although not a full frontend app, the assets folder indicates UI assets (e.g., images, subscription components) used by the Dojo UI interface.
Relationships and Data Flow
• The CLI sets up and manages keys and config, feeding into the Miner and Validator services.
• The Miner service communicates with external APIs (via the commons libraries) and updates results in the database.
• The Validator service pulls tasks from the database, applies synthetic task generation and obfuscation routines, then writes back validated results.
• Docker containers encapsulate individual services and communicate on a virtual network, with API layers interfacing with the Dojo CLI and potentially a web-based UI.
Additional Inclusions
• Incorporate notes related to configuration (for example, multiple .env files reveal environment-specific configurations).
• Given the multi-faceted nature (CLI, automated update services, mining, validating), consider grouping components that have similar roles under “services”, “utilities”, or “infrastructure”.