How macrocosm-os/folding works
Project Overview
• This codebase represents a decentralized, scientific computing subnet within the larger Bittensor ecosystem.
• It is built primarily for protein folding simulations, where validators create protein simulation challenges and miners execute the simulations using OpenMM.
• The objective is to support academic research by simulating protein structures and evaluating them using a physics-based energy minimization (reward) process.
• Key roles include Validators and Miners. Validation and mining logic are cleanly separated and orchestrated via a Bittensor network.
Analysis of the File Structure
• The source code is organized into several top-level directories:
• "folding": Contains central modules—base logic (evaluation, simulation, reward calculations, registration of miners, validators, etc.), utilities, and experiment scripts.
• "folding_api": Implements helper routines and endpoints for authentication, chain interactions, and configuration. This supports communicating with the Bittensor network.
• "neurons": Contains scripts for miners and validators. These will be the executable agents responsible for simulation or validation functionalities.
• "scripts": Provides helper shell and python scripts to run tasks such as starting miners/validators, job queries, and other utilities.
• "tests": Includes unit tests and fixtures ensuring integrity of core components.
• Additional directories like "assets", "documentation", and "contrib" help with images, research documents, and contribution guidelines respectively.
• Configurations such as ".env.example", "pyproject.toml", and "install.sh" indicate environment setup using Conda, Poetry, and shell scripts.
Key Components & Their Interactions
a. Core Simulation Engine and Business Logic:
• The "folding/base" modules outline core simulation routines (e.g., simulation.py), energy/reward computation (reward.py, linear_reward.py, md_rewards.py), and evaluation (evaluation.py) using OpenMM.
• Modules for miners and validators are also in "folding/miners" and "folding/validators".
• “folding/registries” allow dynamic registration of evaluation and miner implementations—an architectural pattern to enable extensibility.
b. API & External Interaction Layer:
• The "folding_api" directory encapsulates RESTful or RPC style endpoints to handle authentication, queries, protein data interactions, and external configuration.
• This layer serves as a bridge between on-chain registration processes (using btcli commands as detailed in the README) and the local computing tasks.
c. Neuron (Validator and Miner) Components:
• “neurons/validator.py” and “neurons/miner.py” are entry points for the two main roles.
• Validators are responsible for enqueuing protein simulation tasks, dispatching these to multiple miners and logging results.
• Miners execute parallel simulations, answer protein folding tasks, and may interact with a local instance of a job pool (GJP) via scripts like “query_rqlite.py”.
d. External Dependencies and Integrations:
• OpenMM: Used as the molecular dynamics (MD) simulation engine.
• Wandb: For logging and experimentation tracking (especially for validator operations).
• S3-compatible storage: Seen from the S3 configuration references for checkpoint and log storage.
• RQLite: Used for local job queue synchronization in miners, as detailed with pm2/node configuration scripts.
• Bittensor network: The blockchain type network that registers the subnet and facilitates the economic reward system.
Architectural Patterns and Design Principles
• Modular Design: Separation of business logic ("folding"), API interfaces ("folding_api"), and agent executables ("neurons") promotes decoupled development and testing.
• Registry Pattern: The registries’ folder and dynamic registrations help in plugging in new evaluators or miners without altering core code.
• Distributed Processing: The design leverages parallel/minimum worker set-ups for miners (using multiple threads or processes) to handle protein simulation in parallel.
• Config-driven Behavior: Extensive use of configuration files and environment variables (for OpenMM, S3, network details) ensures flexible deployment.