Cloudml-hypertune is a lightweight Python library providing helper functions to report hyperparameter tuning metrics to Google Cloud's Vertex AI (formerly Cloud ML Engine). It enables the hyperparameter tuning service to track and optimize model training trials by collecting objective metrics. Despite its `0.1.0.dev6` version being quite old (last released December 2019), it remains the standard way to report custom metrics for hyperparameter tuning on Google Cloud.
pip install cloudml-hypertuneVerified import paths — ran on the pinned version, not inferred.
This quickstart demonstrates how to use `cloudml-hypertune` within a training script to report a metric. The script accepts hyperparameters as command-line arguments, which is a critical requirement for Google Cloud's hyperparameter tuning service to inject trial-specific values. The `HyperTune` instance then reports the objective metric and the current training step. You would run this script within a container on Vertex AI/Cloud ML Engine.
Ensure your training job is submitted to Vertex AI (or deprecated Cloud ML Engine) with a hyperparameter tuning configuration.
Modify your training script to parse hyperparameters as `argparse` arguments, ensuring they are named consistently with your tuning job configuration.
Carefully verify that the metric tag string in your code is identical to the one in your Vertex AI hyperparameter tuning job configuration.
No direct fix needed, but be aware of its static nature. If encountering unexpected behavior with bleeding-edge Python or ML frameworks, consider if the issue is external to `cloudml-hypertune`.
Implement robust error handling and numeric stability checks (e.g., clipping gradients, checking for `NaN`s) in your training code. Ensure `report_hyperparameter_tuning_metric` is called even if training terminates early due to an error, or within a `finally` block if a fallback metric can be determined.
Verify that the `hyperparameter_metric_tag` used in `hpt.report_hyperparameter_tuning_metric()` exactly matches the `metric_id` specified in your Vertex AI hyperparameter tuning job configuration. Also ensure the `report_hyperparameter_tuning_metric` call is actually reached and executed during training.
Ensure `hpt.report_hyperparameter_tuning_metric` is called with a valid `metric_value` and `hyperparameter_metric_tag` at the end of each evaluation step or epoch. For non-TensorFlow models or custom evaluation loops, `cloudml-hypertune` is the definitive way to report.
Debug your training script to ensure `hpt.report_hyperparameter_tuning_metric` is called. Add logging around this call to confirm its execution and the values being passed. Double-check the `hyperparameter_metric_tag` against your Vertex AI job configuration.
No dependency data recorded yet.