How liuruoze/mini-alphastar works
Major Components
• Core Architecture (alphastarmini/core/arch): Contains modules for the deep neural network architecture. Files such as agent.py, arch_model.py, and various “head” files (e.g., action_type_head.py, delay_head.py) indicate that separate neural network components or layers have been modularized.
• Supervised Learning (alphastarmini/core/sl): Implements the supervised learning pipeline. Modules include dataset.py, transform_replay_data.py, and multiple variants for training (by tensor, by pickle, single GPU vs. multi-GPU).
• Reinforcement Learning (alphastarmini/core/rl): Contains the reinforcement learning modules. Look at actor.py, learner.py, rl_train_with_replay.py, and utilities for RL – these are critical for the RL training and evaluation phases.
• Multi-agent (alphastarmini/core/ma): Contains components for managing multi-agent training (coordinator, league, etc.) that are used to set up the environment and manage training for multiple agents.
• Utility and Support Libraries (alphastarmini/lib, alphastarmini/third): Provide supporting functionalities and external integrations. For instance, transformer components and additional helper functions are encapsulated here.
• Scripts and Documentation: Top-level scripts (run.py, test.py) and files in the “doc” and “res” directories supply instructions, pseudocode, and historical context.
Relationships and Interactions
• Data Flow: Begin with “replay” data that is downloaded (instructions in the README and scripts in the “scripts” folder) and then transformed by the transform_replay_data module into training data.
• Supervised Learning Pipeline: Training data is passed through the supervised learning modules in core/sl where an initial model is trained. This model is evaluated in the SC2 environment via the evaluation module in core/rl.
• Reinforcement Learning Pipeline: After validating the SL model via evaluation, the RL modules take over. Reinforcement learning is driven by multi-processing (see files like rl_vs_inner_bot_mp.py) and involves actors, learners, and pseudo-reward modules to further train the agent.
• Multi-agent Coordination: Components from core/ma work in tandem to organize the training across multiple agents or processes.
• Neural Architecture: The deep learning parts (architectural “heads” and encoders) interact with both training pipelines to produce the action decisions in gameplay.
Architectural Patterns and Design Principles
• Modular Design: The project adheres to separation of concerns by distinguishing modules for network architecture, supervised learning, reinforcement learning, and multi-agent coordination.
• Pipeline Flow: The setup clearly defines a sequential pipeline starting from data transformation, through initial SL training, then evaluation, and finally to RL correction.
• Extensibility: With packages segmented into core and library functions, the design encourages extensibility and potential replacement of components (for instance, swapping SL training strategies).
Relevant Technologies
• PyTorch is the central dependency (as specified in the README and requirements.txt) and is used throughout for network construction and training.
• Python’s built-in modules handle multi-processing and threading (particularly for the RL experiments where multiple processes and threads are spawned).
Detailed Elements to Be Included
• Clear Labels: Mark major modules (for instance, “Supervised Learning (core/sl)”, “Reinforcement Learning (core/rl)”, “Multi-agent (core/ma)” and “Deep Neural Architecture (core/arch)”).