API Reference: Optimization

masterful.optimization.OptimizationParams

class masterful.optimization.OptimizationParams(batch_size=None, drop_remainder=False, epochs=None, learning_rate=None, learning_rate_schedule=None, learning_rate_callback=None, warmup_learning_rate=None, warmup_epochs=None, optimizer=None, loss=None, loss_weights=None, early_stopping_callback=None, metrics=None, readonly_callbacks=None)

Parameters controlling the optimization of the model during training.

Optimization means finding the best weights for a model and training data. Optimization is different from regularization because optimization does not consider generalization to unseen data. The central challenge of optimization is speed - find the best weights faster.

Parameters

  • batch_size (int) – Batch size of the features and labels to use during training.

  • drop_remainder (bool) – True if any examples that do not fit evenly into a full batch should be dropped, False otherwise. If False, the last batch during training may or may not equal batch_size.

  • epochs (int) – The number of epochs to train for. An epoch is defined as one full sweep through the training data.

  • learning_rate (float) – The initial learning rate to use for training. This will also be the starting learning rate post-warmup, if warmup is specified.

  • learning_rate_schedule (Optional[tensorflow.python.keras.optimizer_v2.learning_rate_schedule.LearningRateSchedule]) – An instance if tf.keras.optimizers.schedules.LearningRateSchedule which modulate how the learning rate of your optimizer changes over time. Only one of learning_rate_schedule or learning_rate_callback must be set.

  • learning_rate_callback (Optional[tensorflow.python.keras.callbacks.Callback]) – An instance of tf.keras.callbacks.Callback which modulate how the learning rate of your optimizer changes over time. Only one of learning_rate_schedule or learning_rate_callback must be set.

  • warmup_learning_rate (float) – The initial learning rate to use during warmup. If warmup_epochs is greater than zero, the learning rate will linearly grow from warmup_learning_rate to learning_rate over warmup_epochs training epochs, incrementing at each step in each epoch.

  • warmup_epochs (int) – The number of epochs to perform model warmup during training. An epoch is defined as one full sweep through the training data.

  • optimizer (tensorflow.python.keras.optimizer_v2.optimizer_v2.OptimizerV2) – An instance of tf.keras.optimizers.Optimizer, to use as the optimizer during training.

  • loss (Union[tensorflow.python.keras.losses.Loss, Dict[str, tensorflow.python.keras.losses.Loss]]) – An instance of tf.keras.losses.Loss, which is the function that will be optmized by the optimizer. If the model support multiple outputs, this can be a dictionary of loss functions.

  • loss_weights (Union[Sequence, Dict]) – Optional list or dictionary specifying scalar coefficients (Python floats) to weight the loss contributions of different model outputs. The loss value that will be minimized by the model will then be the weighted sum of all individual losses, weighted by the loss_weights coefficients. If a list, it is expected to have a 1:1 mapping to the model’s outputs. If a dict, it is expected to map output names (strings) to scalar coefficients.

  • early_stopping_callback (Optional[tensorflow.python.keras.callbacks.Callback]) – An instance of tf.keras.callbacks.Callback which can stop training before total_steps have been reached depending on certain conditions.

  • metrics (Optional[Sequence[tensorflow.python.keras.metrics.Metric]]) – An optional list of additional metrics that will be measured and reported during training.

  • readonly_callbacks (Optional[Sequence[tensorflow.python.keras.callbacks.Callback]]) – An optional list of callbacks to pass to the training loop. These callbacks must be readonly. They must not attempt to modify any parameters of the training loop, other than entries in the logs.

Return type

None

masterful.optimization.learn_optimization_params

masterful.optimization.learn_optimization_params(model, model_params, dataset, dataset_params, **kwargs)

Learns the optimal set of optimization parameters for training.

Example:

model: tf.keras.Model = ...
model_params: masterful.architecture.params.ArchitectureParams = ...
training_dataset: tf.data.Dataset = ...
training_dataset_params: masterful.data.params.DataParams = ...
optimization_params = masterful.optimization.learn_optimization_params(
    model, model_params, training_dataset, training_dataset_params)
Parameters

  • model (tensorflow.python.keras.engine.training.Model) – The model to learn the optimal set of parameters for.

  • model_params (masterful.architecture.params.ArchitectureParams) – The parameters of the given model.

  • dataset (Union[tensorflow.python.data.ops.dataset_ops.DatasetV2, numpy.ndarray, Tuple[numpy.ndarray, numpy.ndarray], tensorflow.python.keras.utils.data_utils.Sequence, Tuple[Callable[[], Iterator], Tuple[tensorflow.python.framework.tensor_spec.TensorSpec, tensorflow.python.framework.tensor_spec.TensorSpec]]]) – The dataset to learn the optimization parameters for.

  • dataset_params (masterful.data.params.DataParams) – Parameters of the dataset used.

Returns

An instance of OptimizationParams that can be used during training.

Return type

masterful.optimization.params.OptimizationParams