How neuroinformatics-unit/datashuttle works
Main Components
• Core Engine: At the heart of the project you have the datashuttle package which contains:
• The primary business logic (datashuttle_class.py and datashuttle_functions.py).
• Configuration management modules (all files in datashuttle/configs).
• Utilities and helper functions (in datashuttle/utils covering data_transfer, ssh, rclone, logging, validation, etc.).
• Text User Interface (TUI) Layer: This is a separate module (under datashuttle/tui) that provides:
• The entry point for the user interface (tui_launcher.py, app.py).
• UI components such as screens, tabs, custom widgets, CSS styling, and utility decorators/validators for interface logic.
• External Services / Integrations: Elements that facilitate operations such as SSH file transfer and rclone (found within datashuttle/utils/ssh.py and datashuttle/utils/rclone.py).
• Testing & Continuous Integration: The tests directory (with both unit and integration tests) ensures that business logic, configurations, and TUI functions are validated. GitHub workflows under .github/workflows define the CI/CD processes.
• Documentation and DevOps: The docs folder contains build scripts and documentation source files while configuration files (like pyproject.toml, tox.ini, .pre-commit-config.yaml) guide development standards and deployment.
Relationships and Interactions
• The TUI Layer serves as the primary user interaction surface. Its screens and widgets (found under datashuttle/tui/screens, tabs, and custom_widgets.py) interact with the Core Engine to perform tasks such as creating folders, transferring data, or validating configurations.
• The Core Engine uses the configuration management modules to load canonical configs, folder structures, and tags. It then relies on the utilities for actual processing tasks like data transfer or SSH interactions.
• The utility modules are responsible for lower-level tasks, error handling and logging, which are called upon by both the Core Engine and the TUI components.
• Testing components simulate both UI and backend flows to ensure that from user input (via the TUI) to folder creation and data transfer, every component functions as expected.
• The GitHub workflows (and associated CI configuration files) tie the development cycles together by automating code testing, documentation build, and deployment processes.
Architectural Patterns and Design Principles
• Separation of Concerns: The project separates the presentation (TUI) from the core business logic and configuration/utility layers.
• Modular Design: Notice how distinct functionalities (configuration, UI, core functions, utilities) are broken into separate modules/directories, facilitating easier testing and future extension.
• Reusability and Extensibility: By isolating configuration parsing, data transfer functions, and TUI components, the design allows for each part to be modified or extended independently.
• Clear Testing Infrastructure: The tests directory is split into unit, integration, and TUI-specific tests, which is a best practice ensuring each layer is verified.
Technology, Frameworks, and External Dependencies
• Python is the core language, with project management handled via pyproject.toml, tox.ini, etc.
• The TUI components likely use a text-based interface framework – the structure of the tui directory (screens, tabs, custom widgets) indicates a custom-built or extended framework for terminal-based UIs.
• External libraries for SSH and file transfers (e.g., through rclone) are integrated into the utils.
• The CI/CD pipelines in the .github folder ensure standardized testing and documentation deployment.