How idsia/sacred works
Main Components
• Experiment Engine: The core part of Sacred that runs the experiments. Key files include sacred/experiment.py, sacred/run.py, and modules for configuration management (in sacred/config/…) and command-line parsing (sacred/arg_parser.py and sacred/commandline_options.py).
• Configuration Management: Handles the use of config scopes, captured functions, configuration injection, and custom containers. All of these components reside in the sacred/config directory (and related modules such as sacred/config/config_dict.py, sacred/config/config_files.py, etc.).
• Observers: Modular logging components that save experiment metadata and results into various storage systems. They can log to file storage, MongoDB, Google Cloud Storage, S3, SQL databases, or third-party tools via integrations. Look into sacred/observers/ for the various observer implementations.
• Command-Line Interface (CLI): Provides a powerful way to launch experiments and adjust their parameters. This functionality is spread between sacred/arg_parser.py, sacred/commandline_options.py, and integrated within sacred/run.py.
• Utilities and Support: Additional modules—such as randomness control (sacred/randomness.py), logging (sacred/metrics_logger.py), and utilities to capture standard output—support the overall functionality.
Relationships and Interactions
• The User (or developer) initiates an experiment via the CLI.
• The CLI parses parameters and passes them to the Experiment Engine.
• The Experiment Engine loads the experiment configuration using the config management modules (leveraging config scopes and the injection mechanism).
• During execution, the Experiment Engine interacts with one or more Observer components to record metadata, configurations, metrics, and results. These observers can be thought of as pluggable “loggers” that forward data to external systems and dashboards.
• When necessary, utilities like randomness controls ensure that experiments are reproducible.
Architectural Patterns and Design Principles
• Modularity & Extensibility: Sacred’s architecture is designed with modular components (the core experiment engine, configuration management, and multiple observers) that can be extended or replaced based on user needs.
• Separation of Concerns: The configuration system, experiment runner, and logging/observation modules are clearly separated, which aids in maintainability and testability.
• Injection & Dynamic Configuration: The system extensively uses dependency injection (particularly with configuration parameters) to keep experiment functions decoupled from the infrastructure needed to run them.
• Plugin (Observer) Architecture: Allows for flexible logging; new observers for different data stores or front-ends (like Omniboard, Incense, Sacredboard) can be added independently.
Relevant Technologies, Frameworks, or Libraries
• Python (core language)
• Optional dependencies: numpy and pymongo (for randomness control and MongoDB logging)
• External Frontends/Dashboards: Omniboard, Incense, Sacredboard, Neptune, etc.
• Testing Frameworks: pytest and tox (for automated testing across multiple Python versions)
Specific Elements to Be Included
• Clear Labels for Each Component:
• “CLI/Command-Line Parser”
• “Experiment Engine”
• “Configuration Manager”
• “Observers” (with sub-labels for File, MongoDB, etc.)
• “Utilities (Randomness, Metrics, etc.)”
• From CLI to Experiment Engine (input parameters)
• From Experiment Engine to Configuration Manager (request for config injection)
• From Experiment Engine to Observers (sending log data, metadata, and results)
• Feedback loops (if any) between Observers and Experiment Engine (e.g., error handling or dynamic adjustments)
• Visual Distinctions:
• Annotations:
• Brief notes beside each component explaining its primary function.
• Design patterns (like dependency injection for configuration and plugin architecture for observers).
Final Assembly
• Flow downward into the core experiment engine which incorporates configuration management.
• Branch off to a collection of observer components that “listen” to the experiment’s execution and record details.
• Optionally, annotate external systems (like MongoDB, file systems, or external dashboards) that connect through observers.
Happy diagramming!