How superb-nightwolf/googleagentaicapability works
Purpose and entry points
This is a config-driven Python capability for running, logging, comparing, and selecting Google Vertex AI Experiment runs. Its primary training entry point is mlops/experiments/experiment_run.py (shown in the README tree); comparison without retraining uses mlops/experiments/compare_runs.py. Training behavior is selected dynamically from YAML rather than imported as a fixed model implementation.
Configuration and runtime dispatch
mlops/experiments/config.py loads and validates the experiment configuration, including GCP project/location, experiment identity, training callable, logging strategy, and champion-selection policy. mlops/experiments/config.yaml is the default package configuration. The configured training.module and training.function are dynamically loaded and invoked through mlops/experiments/utils/training_loader.py.
Experiment and run lifecycle
mlops/experiments/vertex_experiment.py owns reusable parent-experiment access via aiplatform.Experiment.get_or_create(). mlops/experiments/vertex_experiment_run.py creates a persistent ExperimentRun, attaches it as the active SDK run with aiplatform.start_run(..., resume=True), and ends or fails it through the package-level Vertex SDK API. This separates durable run creation from process-local active-session control.
Training and logging pipeline
mlops/experiments/experiment_run.py orchestrates the primary flow: load config, create/start run, optionally enable autologging, execute configured training, explicitly log the returned result, close the run, and optionally select a champion. Training returns a plain dictionary containing model, curated parameters, final metrics, optional classification/time-series data, model metadata, and input examples. Explicit logging remains the authoritative path for custom values and artifacts.
Autologging boundary
mlops/experiments/experiment_autolog.py enables/disables process-global Vertex SDK autologging around training only. It is activated after the target run starts, ensuring automatic framework telemetry attaches to that run. Because autologging is global mutable process state, it is disabled after training or run completion to prevent leakage into later workloads.
Model artifact and lineage handling
mlops/experiments/utils/model_logging.py implements configurable model logging. Native mode uses ExperimentRun.log_model() for SDK-supported flavors such as scikit-learn, XGBoost, and TensorFlow. Artifact mode, or auto fallback for unsupported models, serializes the model to Cloud Storage and creates a Vertex ML Metadata system.Model artifact plus execution association. This is the principal compatibility boundary between framework-specific model objects and Vertex lineage.
Run comparison and champion promotion
mlops/experiments/experiment_comparison.py retrieves experiment-run DataFrames, ranks completed runs using metric.<name>, marks champion/challenger state through run parameters, and writes/reads a champion manifest. mlops/experiments/compare_runs.py exposes this selection flow independently of training. The manifest is stored in the staging GCS bucket and supplies the winning model artifact URI to downstream Model Registry workflows.
Infrastructure and external services
The runtime is Python with google-cloud-aiplatform, including the autologging extra when enabled. It requires Google Cloud Vertex AI Experiments, Vertex ML Metadata, and Cloud Storage; authentication uses Application Default Credentials and IAM must permit Vertex/metadata writes plus staging-bucket access. Example executable integrations are under mlops/experiments/examples/, including supported scikit-learn and fallback LightGBM paths.
Components 17
Entry points
- Training entry point · Runs the configured experiment lifecycle.
- Comparison entry point · Selects a champion without retraining.
Experiment runtime
- Config loader · Loads experiment, training, logging, and selection settings.
- Default configuration
- Dynamic training loader · Imports and invokes the configured training callable.
- Configured training · Returns model, parameters, metrics, metadata, and examples.
- Parent experiment access · Gets or creates the reusable parent experiment.
- Run lifecycle · Creates persistent runs and controls the active SDK session.
Logging and lineage
- Autologging boundary · Enables telemetry only while the target run is active.
- Explicit result logging · Logs returned custom values and artifacts.
- Model logging · Uses native SDK logging or artifact lineage fallback.
Comparison and promotion
- Run comparison · Ranks completed runs and maintains champion state.
- Champion manifest · Publishes the winning model artifact URI.
Google Cloud services
- Vertex AI Experiments
- Vertex ML Metadata
- Cloud Storage staging bucket
- Downstream Model Registry · Consumes the promoted model artifact URI.
Connections 22
- Training entry point → Config loader: loads
- Config loader → Default configuration: defaults
- Training entry point → Parent experiment access: gets or creates
- Parent experiment access → Vertex AI Experiments: uses
- Training entry point → Run lifecycle: creates and starts
- Run lifecycle → Vertex AI Experiments: persists run
- Training entry point → Autologging boundary: scopes around training
- Training entry point → Dynamic training loader: dispatches
- Dynamic training loader → Configured training: dynamically invokes
- Configured training → Explicit result logging: returns result
- Explicit result logging → Run lifecycle: logs to active run
- Explicit result logging → Model logging: logs model
- Model logging → Vertex AI Experiments: native model logging
- Model logging → Cloud Storage staging bucket: artifact fallback
- Model logging → Vertex ML Metadata: creates model lineage
- Training entry point → Run comparison: optional selection
- Comparison entry point → Config loader: loads
- Comparison entry point → Run comparison: runs selection
- Run comparison → Vertex AI Experiments: retrieves and marks runs
- Run comparison → Champion manifest: writes and reads
- Champion manifest → Cloud Storage staging bucket: stored in
- Champion manifest → Downstream Model Registry: supplies artifact URI