How kohakublueleaf/hakuriver works
Main Components
• Host Server (hakuriver.host): This module acts as the cluster “brain”. It coordinates task submissions from clients, allocates tasks to available compute nodes based on free CPU cores, maintains a persistent SQLite database (located via the configuration in the db folder and default_config.toml), and handles status updates and heartbeats from runners.
• Runner Agent (hakuriver.runner): Running on each compute node, this component registers itself with the host, reports available CPU core counts, monitors task executions, and uses system tools (like numactl) for task core isolation.
• Client (hakuriver.client): This provides the command-line interface for users. It submits tasks to the host server, queries task statuses, and can issue control commands (like kill or list nodes).
• Storage: Although not a coded component, the architecture relies on two storage types:
– Shared Storage (shared_dir): Used to store scripts and task outputs, accessible with the same path by all nodes.
– Local Temporary Storage (local_temp_dir): Each runner’s local scratch space.
• Hakurun Utility (demo_hakurun.py and the hakurun CLI in the file tree): A local tool for expanding command-line arguments and running combinations of tasks in parallel. It is integral for pre-submission testing and batch processing of similar tasks.
File Structure for Implementation Details
• The source code is located under src/hakuriver and src/hakurun. Under “hakuriver”, you have well-organized subdirectories for cli, core, database (db), and utilities, indicating clear module separation.
• The “cli” directories in both locations (src/hakuriver/cli and src/hakurun) are dedicated to user interactions.
• Key modules (client.py, host.py, runner.py) are duplicated in both cli and core directories, suggesting standard interfaces for external commands and core functionality.
• The “utils” directory shows support functionality such as configuration loading, logging, and even a snowflake generator for unique identifiers (likely for task IDs).
Architectural Patterns and Design Principles
• Service Separation: The clear separation between Host, Runner, and Client simplifies management and scalability.
• Modularity: The file/directory structure indicates that each component (CLI, core, DB, utils) is independently maintained.
• Lightweight Resource Management: Emphasize the architecture’s focus on CPU-bound tasks and minimalistic multi-node coordination (as opposed to full-fledged HPC schedulers).