API Reference: Evaluation

CocoEvaluationMetrics

class masterful.evaluation.detection.coco.CocoEvaluationMetrics(categories, include_metrics_per_category=False, all_metrics_per_category=False, skip_predictions_for_unlabeled_class=False, super_categories=None)

Class for evaluating the MSCOCO evaluation metrics against the outputs from a Keras based object detection model.

This relies on the pycocotools package from the MSCOCO official release to perform the evaluation.

__init__(categories, include_metrics_per_category=False, all_metrics_per_category=False, skip_predictions_for_unlabeled_class=False, super_categories=None)

Initializes a new instance of CocoEvaluationMetrics.

Parameters

  • categories – A list of dicts, each of which has the following keys - ‘id’: (required) an integer id uniquely identifying this category. ‘name’: (required) string representing category name e.g., ‘cat’, ‘dog’.

  • include_metrics_per_category – If True, include metrics for each category.

  • all_metrics_per_category – Whether to include all the summary metrics for each category in per_category_ap. Be careful with setting it to true if you have more than handful of categories, because it will pollute your mldash.

  • skip_predictions_for_unlabeled_class – Skip predictions that do not match with the labeled classes for the image.

  • super_categories – None or a python dict mapping super-category names (strings) to lists of categories (corresponding to category names in the label_map). Metrics are aggregated along these super-categories and added to the per_category_ap and are associated with the name PerformanceBySuperCategory/<super-category-name>.

clear()

Clears the state to prepare for a fresh evaluation.

evaluate_model(model, predictions_to_labels, test_dataset, num_classes, max_examples=9223372036854775807)

Evaluates a Keras based detection model and returns the COCO statistics.

Parameters

  • model (tensorflow.python.keras.engine.training.Model) – The model to use for predictions.

  • predictions_to_labels (Callable) – A Callbable function which converts model predictions into labels that can be evaluated using the COCO metrics.

  • test_dataset (tensorflow.python.data.ops.dataset_ops.DatasetV2) – The dataset to measure over.

  • num_classes (int) – The number of classes in the dataset and model predictions.

  • max_examples (int) – Maximum number of examples to evaluate. Defaults to all examples.

Returns

A dictionary of COCO evaluation metrics.

Return type

Dict

write_detections_to_json_file(json_output_path)

Saves the detections into json_output_path in the format used by MS COCO.

Parameters

json_output_path – String containing the output file’s path. It can be also None. In that case nothing will be written to the output file.