Classification of Land Use with the Masterful CLI Trainer

Open In Colab        DownloadDownload this Notebook

In this guide, you will take a deeper look at Classification with the Masterful CLI Trainer to train a state of the art model.

This guide will use a satellite imagery dataset from UCMerced to classify different plots of land according to their usage.

Define the Application

In this guide, your challenge is to build a computer vision model that can classify different plots of land according to their usage. In this instance, the images have already been extracted from larger satellite imagery and each image can be classified into a single category.

Since each image can be categorized into a single class, and there are more than one possible classes, the computer vision task for this problem is multi-class classification.

TL;DR

Don’t want to read the rest of this guide, and want to start training immediately? The following command shows you how to start training with Masterful, using a configuration file and dataset on S3.

masterful-train --config https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/training.yaml

Prepare the Data

For this guide, you will use the UCMerced dataset, which is a land use classification dataset used for image classification. In this section, you will take the raw dataset and convert it into a format that Masterful can understand. Along the way, you will visualize a few examples of the data to see both the input to the model, as well as the predictions after training the model.

The UCMerced dataset contains 100 images for each of 21 different classes. Each image measures 256x256 pixels. The images were manually extracted from large images from the USGS National Map Urban Area Imagery collection for various urban areas around the country. The pixel resolution of this public domain imagery is 1 foot.

The CSV format for Image Classification can be found at Masterful Image Classification. Each line contains the relative path to the image, and the 0-indexed integer of the groundtruth class. For example, the following are the first few lines from the trainval.csv file:

images/golfcourse17.png, 9
images/intersection98.png, 11
images/agricultural59.png, 0
images/overpass28.png, 14
images/agricultural81.png, 0
images/freeway01.png, 8
images/river77.png, 16
.
.
.

The label map file helps convert the integer class indices into human readable forms. The label map for UCMerced looks like:

0,agricultural
1,airplane
2,baseballdiamond
3,beach
4,buildings
5,chaparral
6,denseresidential
7,forest
8,freeway
9,golfcourse
10,harbor
11,intersection
12,mediumresidential
13,mobilehomepark
14,overpass
15,parkinglot
16,river
17,runway
18,sparseresidential
19,storagetanks
20,tenniscourt

Converting each dataset is typically a one-off operation that is different for every dataset you want to train with. For brevity, you can use the already converted dataset located at the public S3 bucket s3://masterful-public/datasets/ucmerced/. In this bucket, you will see the following files:

ucmerced\
  test.csv
  trainval.csv
  training.yaml
  label_map.csv
  images\

Explore the Data

You should always visually inspect your dataset to get a sense of what the model will see, and to roughly verify that your dataset conversion routine worked properly and you are not training with corrupted or incorrect data.

[2]:
# Install dependencies necessary to run the following
# code.
!pip install opencv-python-headless --quiet
!pip install masterful --quiet

# Import the packages used below.
import matplotlib.pyplot as plt
import os
import requests
import tarfile
import tempfile
import tensorflow as tf
import urllib.request

# Import and activate the Masterful package.
import masterful
masterful = masterful.activate()

# Helper function to display a progress when downloading
# a file using HTTP.
from masterful.utils.downloader import progress_bar_factory

# This is necessary for running inside of Colab/Jupyter,
# since the CLI trainer is run outside of the kernel
# (as a script command).
gpus = tf.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
  tf.config.experimental.set_memory_growth(gpu, True)
MASTERFUL: Your account has been successfully activated. Masterful v0.5.2 is loaded.
[11]:
DATASET_ROOT = "https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/"
TRAINING_CSV_URL = os.path.join(DATASET_ROOT, "trainval.csv")

with tempfile.TemporaryDirectory() as temp_directory:
  training_csv_name = os.path.join(temp_directory, "trainval.csv")
  _ =  urllib.request.urlretrieve(TRAINING_CSV_URL, training_csv_name)

  examples = []
  with open(training_csv_name) as training_csv:
    examples = training_csv.readlines()

image1_path, label1 = examples[0].split(',')
image2_path, label2 = examples[1].split(',')
image3_path, label3 = examples[2].split(',')
image4_path, label4 = examples[3].split(',')

label_map = {
  "0": "agricultural",
  "1": "airplane",
  "2": "baseballdiamond",
  "3": "beach",
  "4": "buildings",
  "5": "chaparral",
  "6": "denseresidential",
  "7": "forest",
  "8": "freeway",
  "9": "golfcourse",
  "10": "harbor",
  "11": "intersection",
  "12": "mediumresidential",
  "13": "mobilehomepark",
  "14": "overpass",
  "15": "parkinglot",
  "16": "river",
  "17": "runway",
  "18": "sparseresidential",
  "19": "storagetanks",
  "20": "tenniscourt",
}
image1 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image1_path))).read(), channels=3, dtype=tf.float32)
image2 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image2_path))).read(), channels=3,  dtype=tf.float32)
image3 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image3_path))).read(), channels=3,  dtype=tf.float32)
image4 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image4_path))).read(), channels=3,  dtype=tf.float32)

f, axarr = plt.subplots(2, 2, figsize=(15,15))
_ = axarr[0, 0].imshow(image1)
_ = axarr[0, 0].title.set_text(label_map[label1.strip()])
_ = axarr[0, 1].imshow(image2)
_ = axarr[0, 1].title.set_text(label_map[label2.strip()])
_ = axarr[1, 0].imshow(image3)
_ = axarr[1, 0].title.set_text(label_map[label3.strip()])
_ = axarr[1, 1].imshow(image4)
_ = axarr[1, 1].title.set_text(label_map[label4.strip()])

../_images/notebooks_guide_cli_classification_land_use_6_0.png

Configure the CLI Trainer

The Masterful CLI Trainer is a command line tool that trains a production quality model with no code required. The Masterful CLI Trainer take a YAML configuration file as input. The configuration file fully specifies everything necessary for training, including the dataset, model, export formats, and evaluation metrics.

Choosing a Model

Masterful provides many different state of the art image classification models that you can choose from. In general, choosing a model can have different constraints than training the model (are you deploying on server or edge? Runtime or latency constraints? Memory constraints?). You generally want to choose a model that is large enough to fit your data, but not so large as to overfit the training data and “memorize” the results, which can lead to poor generalization performance.

The Masterful Image Classification Model Zoo supports the following model architectures.

Model Name

Year

Description

resnet50v1

2015

ResNet-50 architecture from the paper Deep Residual Learning for Image Recognition

resnet101v1

2015

ResNet-101 architecture from the paper Deep Residual Learning for Image Recognition

resnet152v1

2015

ResNet-152 architecture from the paper Deep Residual Learning for Image Recognition

resnet50v2

2016

ResNet-50 architecture from the paper Identity Mappings in Deep Residual Networks

resnet101v2

2016

ResNet-101 architecture from the paper Identity Mappings in Deep Residual Networks

resnet152v2

2016

ResNet-152 architecture from the paper Identity Mappings in Deep Residual Networks

efficientnetb0_v1

2019

B0 architecture from the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

efficientnetb0_v1_small

2019

B0 architecture from the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks. This architecture is tuned for small (32x32) input sizes.

efficientnetb1_v1

2019

B1 architecture from the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

efficientnetb2_v1

2019

B2 architecture from the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

efficientnetb3_v1

2019

B3 architecture from the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

efficientnetb4_v1

2019

B4 architecture from the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

efficientnetb5_v1

2019

B5 architecture from the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

efficientnetb6_v1

2019

B6 architecture from the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

efficientnetb7_v1

2019

B7 architecture from the paper EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks

mobilenetv3_small

2019

The small architecture (alpha = 1.0) from the paper Searching for MobileNetV3

mobilenetv3_large

2019

The large architecture (alpha = 1.0) from the paper Searching for MobileNetV3

vgg16

2015

The VGG16 architecture from the paper Very Deep Convolutional Networks for Large-Scale Image Recognition

vgg19

2015

The VGG19 architecture from the paper Very Deep Convolutional Networks for Large-Scale Image Recognition

xception

2017

The Xception v1 architecture from the paper Xception: Deep Learning with Depthwise Separable Convolutions

inception_v3

2016

The Inception V3 architecture from the paper Rethinking the Inception Architecture for Computer Vision

In the example below, you will select a resnet50v1 model. This is a great model to start with, as it generalizes well across most domains.

The Configuration File

The following section is a condensed YAML configuration file for training against this dataset with NO unlabeled data. It points to the dataset created above, and selects a resnet50v1 model, which is a good model for small datasets and edge inference use cases.

Note all of the model artifacts will be saved to the directory ~/model_output, which you can configure in the output section below.

The original YAML file can be found here.

dataset:
  root_path: https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced
  splits: [trainval, test]
  label_map: label_map
  optimize: True
model:
  architecture: resnet50v1
  num_classes: 21
  input_shape: [256,256,3]
training:
  task: classification
  training_split: trainval
output:
  formats: [saved_model, onnx]
  path: ~/model_output
evaluation:
  split: test

Train the Model

Training the model involves simply running the Masterful CLI Trainer (masterful-train) against the YAML file on S3.

Below, you will setup the CLI and configuration file to run in this notebook. However, if you want to train this model outside of the notebook, against the dataset on S3, you can run:

masterful-train --config https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/training.yaml
[1]:
# Use the Masterful CLI to train the model without unlabeled data
!masterful-train --config https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/training.yaml
MASTERFUL: Your account has been successfully activated. Masterful v0.5.2 is loaded.
MASTERFUL [14:04:16]: Training with configuration 'https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/training.yaml':
----------  --------------------------------------------------------------------------------
dataset     root_path  https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced
            splits     ['trainval', 'test']
            label_map  label_map
            optimize   True
model       architecture  resnet50v1
            num_classes   21
            input_shape   [256, 256, 3]
training    task            classification
            training_split  trainval
output      formats  ['saved_model', 'onnx']
            path     ~/model_output
evaluation  split  test
----------  --------------------------------------------------------------------------------
MASTERFUL [14:04:20]: Building model 'resnet50v1'...
MASTERFUL [14:04:22]: Using model resnet50v1 with:
MASTERFUL [14:04:22]:     23630741 total parameters
MASTERFUL [14:04:22]:     23577621 trainable parameters
MASTERFUL [14:04:22]:     53120 untrainable parameters
MASTERFUL [14:04:22]: Caching I/O optimized dataset split 'trainval' to /home/sam/.masterful/datasets/2723296d09d4f0d7f7fb923db3237bc6ca4f7668...
100%|███████████████████████████████████████| 1890/1890 [00:46<00:00, 40.83it/s]
MASTERFUL [14:05:10]: Caching I/O optimized dataset split 'test' to /home/sam/.masterful/datasets/538fb0c1083cebe30f260bcc37f79adfb8bdde92...
100%|█████████████████████████████████████████| 210/210 [00:05<00:00, 40.95it/s]
MASTERFUL [14:05:17]: Dataset Summary:
MASTERFUL [14:05:17]:   Training Dataset: 1890 examples.
MASTERFUL [14:05:17]:   Validation Dataset: 0 examples.
MASTERFUL [14:05:17]:   Test Dataset: 210 examples.
MASTERFUL [14:05:17]:   Unlabeled Dataset: 0 examples.
MASTERFUL [14:05:17]: Training Dataset Analysis:
100%|██████████████████████████████████████| 1890/1890 [00:07<00:00, 258.13it/s]

MASTERFUL [14:05:24]: Training dataset analysis finished at 14:05:24 in 7 seconds (7s), returned:
------------------  ----------------------------------------
Total Examples      1890
Label Counts        agricultural       88
                    airplane           89
                    baseballdiamond    92
                    beach              87
                    buildings          83
                    chaparral          90
                    denseresidential   90
                    forest             91
                    freeway            83
                    golfcourse         94
                    harbor             90
                    intersection       85
                    mediumresidential  88
                    mobilehomepark     91
                    overpass           93
                    parkinglot         92
                    river              92
                    runway             92
                    sparseresidential  91
                    storagetanks       95
                    tenniscourt        94
Label Distribution  agricultural       0.0465608
                    airplane           0.0470899
                    baseballdiamond    0.0486772
                    beach              0.0460317
                    buildings          0.0439153
                    chaparral          0.047619
                    denseresidential   0.047619
                    forest             0.0481481
                    freeway            0.0439153
                    golfcourse         0.0497354
                    harbor             0.047619
                    intersection       0.0449735
                    mediumresidential  0.0465608
                    mobilehomepark     0.0481481
                    overpass           0.0492063
                    parkinglot         0.0486772
                    river              0.0486772
                    runway             0.0486772
                    sparseresidential  0.0481481
                    storagetanks       0.0502646
                    tenniscourt        0.0497354
Balanced            Yes
Per Channel Mean    [123.41418882 124.86677271 114.86978589]
Per Channel StdDev  [44.09977763 41.55092148 39.49503547]
Min Height          247
Min Width           242
Max Height          257
Max Width           257
Average Height      256
Average Width       256
Largest Image       (257, 257, 3)
Smallest Image      (247, 247, 3)
Duplicates          0
------------------  ----------------------------------------
MASTERFUL [14:05:24]: Test Dataset Analysis:
100%|████████████████████████████████████████| 210/210 [00:00<00:00, 251.64it/s]

MASTERFUL [14:05:25]: Test dataset analysis finished at 14:05:25 in 1 seconds (1s), returned:
------------------  ----------------------------------------
Total Examples      210
Label Counts        agricultural       12
                    airplane           11
                    baseballdiamond     8
                    beach              13
                    buildings          17
                    chaparral          10
                    denseresidential   10
                    forest              9
                    freeway            17
                    golfcourse          6
                    harbor             10
                    intersection       15
                    mediumresidential  12
                    mobilehomepark      9
                    overpass            7
                    parkinglot          8
                    river               8
                    runway              8
                    sparseresidential   9
                    storagetanks        5
                    tenniscourt         6
Label Distribution  agricultural       0.0571429
                    airplane           0.052381
                    baseballdiamond    0.0380952
                    beach              0.0619048
                    buildings          0.0809524
                    chaparral          0.047619
                    denseresidential   0.047619
                    forest             0.0428571
                    freeway            0.0809524
                    golfcourse         0.0285714
                    harbor             0.047619
                    intersection       0.0714286
                    mediumresidential  0.0571429
                    mobilehomepark     0.0428571
                    overpass           0.0333333
                    parkinglot         0.0380952
                    river              0.0380952
                    runway             0.0380952
                    sparseresidential  0.0428571
                    storagetanks       0.0238095
                    tenniscourt        0.0285714
Balanced            Yes
Per Channel Mean    [124.01575081 125.79742028 114.89969612]
Per Channel StdDev  [45.46955658 43.01082001 40.9943777 ]
Min Height          247
Min Width           247
Max Height          256
Max Width           256
Average Height      256
Average Width       256
Largest Image       (256, 256, 3)
Smallest Image      (247, 247, 3)
Duplicates          0
------------------  ----------------------------------------
MASTERFUL [14:05:25]: Cross-Dataset Analysis:
MASTERFUL [14:05:25]: Cross-Dataset analysis finished at 14:05:25 in 0 seconds (0s), returned:
--------  -----------
trainval  trainval  0
          test      1
test      trainval  1
          test      0
--------  -----------
MASTERFUL [14:05:25]: WARNING: Duplicates detected in dataset split 'test'.
MASTERFUL [14:05:25]: WARNING: You can find the duplicate entries using the tool:
MASTERFUL [14:05:25]: WARNING: python -m masterful.data.duplicate_detector --config=https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/training.yaml
MASTERFUL [14:05:25]: Meta-Learning architecture parameters...
MASTERFUL [14:05:26]: Architecture learner finished at 14:05:26 in 1 seconds (1s), returned:
------------------------------  -----------------------------
task                            Task.CLASSIFICATION
num_classes                     21
ensemble_multiplier             1
custom_objects                  {}
model_config
backbone_only                   False
input_shape                     (256, 256, 3)
input_range                     ImageRange.IMAGENET_CAFFE_BGR
input_dtype                     <dtype: 'float32'>
input_channels_last             True
prediction_logits               True
prediction_dtype                <dtype: 'float32'>
prediction_structure            TensorStructure.SINGLE_TENSOR
prediction_shape                (21,)
total_parameters                23630741
total_trainable_parameters      23577621
total_non_trainable_parameters  53120
------------------------------  -----------------------------
MASTERFUL [14:05:26]: Meta-learning training dataset parameters...
MASTERFUL [14:05:27]: Training dataset learner finished at 14:05:27 in 1 seconds (1s), returned:
-------------------------  -----------------------------
num_classes                21
task                       Task.CLASSIFICATION
image_shape                (256, 256, 3)
image_range                ImageRange.IMAGENET_CAFFE_BGR
image_dtype                <dtype: 'float32'>
image_channels_last        True
label_dtype                <dtype: 'float32'>
label_shape                (21,)
label_structure            TensorStructure.SINGLE_TENSOR
label_sparse               False
label_bounding_box_format
-------------------------  -----------------------------
MASTERFUL [14:05:27]: Meta-learning test dataset parameters...
MASTERFUL [14:05:28]: Test dataset learner finished at 14:05:28 in 1 seconds (1s), returned:
-------------------------  -----------------------------
num_classes                21
task                       Task.CLASSIFICATION
image_shape                (256, 256, 3)
image_range                ImageRange.IMAGENET_CAFFE_BGR
image_dtype                <dtype: 'float32'>
image_channels_last        True
label_dtype                <dtype: 'float32'>
label_shape                (21,)
label_structure            TensorStructure.SINGLE_TENSOR
label_sparse               False
label_bounding_box_format
-------------------------  -----------------------------
MASTERFUL [14:05:28]: Meta-Learning optimization parameters...
Callbacks: 100%|███████████████████████████████| 8/8 [01:21<00:00, 10.19s/steps]
MASTERFUL [14:06:50]: Optimization learner finished at 14:06:50 in 82 seconds (1m 22s), returned:
-----------------------  -----------------------------------------------------------------
batch_size               32
drop_remainder           False
epochs                   1000000
learning_rate            0.0012499999720603228
learning_rate_schedule
learning_rate_callback   <keras.callbacks.ReduceLROnPlateau object at 0x7f47986ea080>
warmup_learning_rate     1e-06
warmup_epochs            5
optimizer                <tensorflow_addons.optimizers.lamb.LAMB object at 0x7f479869dd68>
loss                     <keras.losses.CategoricalCrossentropy object at 0x7f47a00df160>
loss_weights
early_stopping_callback  <keras.callbacks.EarlyStopping object at 0x7f47a00716a0>
metrics                  [<keras.metrics.CategoricalAccuracy object at 0x7f47987ddba8>]
readonly_callbacks
-----------------------  -----------------------------------------------------------------
MASTERFUL [14:06:52]: Meta-Learning Regularization Parameters...
MASTERFUL [14:06:58]: Warming up model for analysis.
MASTERFUL [14:07:04]:   Warming up batch norm statistics (this could take a few minutes).
MASTERFUL [14:07:07]:   Warming up training for 505 steps.
100%|██████████████████████████████████████| 505/505 [01:55<00:00,  4.36steps/s]
MASTERFUL [14:09:03]:   Validating batch norm statistics after warmup for stability (this could take a few minutes).
MASTERFUL [14:09:51]: Analyzing baseline model performance. Training until validation loss stabilizes...
Baseline Training: 100%|█████████████████| 3900/3900 [11:46<00:00,  5.52steps/s]
MASTERFUL [14:21:56]: Baseline training complete.
MASTERFUL [14:21:56]: Meta-Learning Basic Data Augmentations...
Node 1/4: 100%|██████████████████████████| 1200/1200 [03:29<00:00,  5.72steps/s]
Node 2/4: 100%|██████████████████████████| 1200/1200 [03:32<00:00,  5.66steps/s]
Node 3/4: 100%|██████████████████████████| 1200/1200 [03:33<00:00,  5.62steps/s]
Node 4/4: 100%|██████████████████████████| 1200/1200 [03:33<00:00,  5.63steps/s]
MASTERFUL [14:37:08]: Meta-Learning Data Augmentation Clusters...
Distance Analysis: 100%|███████████████████| 143/143 [02:16<00:00,  1.05steps/s]
Node 1/10: 100%|█████████████████████████| 1200/1200 [04:23<00:00,  4.56steps/s]
Node 2/10: 100%|█████████████████████████| 1200/1200 [04:22<00:00,  4.56steps/s]
Node 3/10: 100%|█████████████████████████| 1200/1200 [04:23<00:00,  4.55steps/s]
Node 4/10: 100%|█████████████████████████| 1200/1200 [04:23<00:00,  4.55steps/s]
Node 5/10: 100%|█████████████████████████| 1200/1200 [04:23<00:00,  4.56steps/s]
Distance Analysis: 100%|█████████████████████| 66/66 [01:03<00:00,  1.04steps/s]
Node 6/10: 100%|█████████████████████████| 1200/1200 [04:40<00:00,  4.27steps/s]
Node 7/10: 100%|█████████████████████████| 1200/1200 [04:40<00:00,  4.28steps/s]
Node 8/10: 100%|█████████████████████████| 1200/1200 [04:39<00:00,  4.29steps/s]
Node 9/10: 100%|█████████████████████████| 1200/1200 [04:40<00:00,  4.28steps/s]
Node 10/10: 100%|████████████████████████| 1200/1200 [04:40<00:00,  4.27steps/s]
MASTERFUL [15:28:45]: Meta-Learning Label Based Regularization...
Node 1/2: 100%|██████████████████████████| 1200/1200 [04:27<00:00,  4.49steps/s]
Node 2/2: 100%|██████████████████████████| 1200/1200 [04:28<00:00,  4.46steps/s]
MASTERFUL [15:38:17]: Meta-Learning Weight Based Regularization...
MASTERFUL [15:38:18]: Analysis finished in 91.32490015029907 minutes.
MASTERFUL [15:38:18]: Learned parameters margin-repeated-albacore saved at /home/sam/.masterful/policies/margin-repeated-albacore.
MASTERFUL [15:38:18]: Regularization learner finished at 15:38:18 in 5488 seconds (1h 31m 28s), returned:
-------------------------  -----------------------------------------------
shuffle_buffer_size        1701
mirror                     1.0
rot90                      1.0
rotate                     0
mixup                      0.75
cutmix                     0.0
label_smoothing            0
hsv_cluster                4
hsv_cluster_to_index       [[ 1  2  6  9 11 11]
                            [ 2  4  6  8 11 11]
                            [ 1  1  3  4  5 11]
                            [ 1  1  3  4  7 11]
                            [ 2  2  3  5  8 11]]
hsv_magnitude_table        [[  0  10  20  30  40  50  60  70  80  90 100]
                            [  0  20  10  30  40  50  60  70  80  90 100]
                            [  0  10  20  30  40  50  60  70  80  90 100]
                            [  0  10  20  30  40  50  60  70  80 100  90]
                            [100   0  10  90  80  20  30  70  40  60  50]]
contrast_cluster           4
contrast_cluster_to_index  [[ 1  3  8 11 11 11]
                            [ 1  1  1  1  5 11]
                            [ 2  5  6  7  9 11]
                            [ 1  2  5 10 11 11]
                            [ 1  2  4  8 11 11]
                            [ 1  3  5  6  8 11]]
contrast_magnitude_table   [[  0  10  20  30  40  50  60  70  80  90 100]
                            [  0  10  20  30  50  40  60  70  80  90 100]
                            [  0  10  20  30  40  50  60  70  80  90 100]
                            [  0  10  20  30  40  50  60  70  80  90 100]
                            [  0  10  20  30  40  50  60  70  80  90 100]
                            [  0  10  20  30  40  50  60  70  80  90 100]]
blur_cluster               4
blur_cluster_to_index      [[ 2 11 11 11 11 11]
                            [ 8 11 11 11 11 11]]
blur_magnitude_table       [[  0  10  20  30  40  50  60  70  80  90 100]
                            [  0  60  10  50  20  40  30  70  80  90 100]]
spatial_cluster            0
spatial_cluster_to_index   [[ 2  4  6  8 10 11]
                            [ 2  4  6  8 10 11]
                            [ 1  5  8  9 11 11]
                            [ 2  3  6 10 11 11]
                            [ 3  5  6  8 11 11]
                            [ 1  1  1  1  2 11]]
spatial_magnitude_table    [[  0  10  20  30  40  50  60  70  80  90 100]
                            [  0  10  20  30  40  50  60  70  80  90 100]
                            [  0  10  20 100  80  90  70  30  40  60  50]
                            [  0  10  20  90 100  80  30  40  70  60  50]
                            [  0  10  20  30  40  50  60  70  80  90 100]
                            [  0  10  20  30  40  50  60  70  80  90 100]]
synthetic_proportion       [0.0]
-------------------------  -----------------------------------------------
MASTERFUL [15:38:18]: Learning SSL parameters...
MASTERFUL [15:38:19]: SSL learner finished at 15:38:19 in 1 seconds (1s), returned:
----------  --
algorithms  []
----------  --
MASTERFUL [15:38:19]: Training model with semi-supervised learning disabled.
MASTERFUL [15:38:20]: Performing basic dataset analysis.
MASTERFUL [15:38:26]: Masterful will use 189 labeled examples as a validation set since no validation data was provided.
MASTERFUL [15:38:26]: Training model with:
MASTERFUL [15:38:26]:   1701 labeled examples.
MASTERFUL [15:38:26]:   189 validation examples.
MASTERFUL [15:38:26]:   0 synthetic examples.
MASTERFUL [15:38:26]:   0 unlabeled examples.
MASTERFUL [15:38:27]: Training model with learned parameters margin-repeated-albacore in two phases.
MASTERFUL [15:38:27]: The first phase is supervised training with the learned parameters.
MASTERFUL [15:38:27]: The second phase is semi-supervised training to boost performance.
MASTERFUL [15:38:29]: Warming up model for supervised training.
MASTERFUL [15:38:34]:   Warming up batch norm statistics (this could take a few minutes).
MASTERFUL [15:38:38]:   Warming up training for 505 steps.
100%|██████████████████████████████████████| 505/505 [02:11<00:00,  3.85steps/s]
MASTERFUL [15:40:49]:   Validating batch norm statistics after warmup for stability (this could take a few minutes).
MASTERFUL [15:41:10]: Starting Phase 1: Supervised training until the validation loss stabilizes...
Supervised Training: 100%|███████████████| 4020/4020 [17:22<00:00,  3.86steps/s]
MASTERFUL [15:58:56]: Semi-Supervised training disabled in parameters.
MASTERFUL [15:58:57]: Training complete in 20.49317009449005 minutes.
MASTERFUL [15:59:14]: Saving model output to /home/sam/model_output/session-00344.
MASTERFUL [15:59:14]:     Saving saved_model output to /home/sam/model_output/session-00344/saved_model
MASTERFUL [15:59:32]:     Saving onnx output to /home/sam/model_output/session-00344/onnx
MASTERFUL [16:00:06]: Saving regularization params to /home/sam/model_output/session-00344/regularization.params.
MASTERFUL [16:00:06]: ************************************
MASTERFUL [16:00:06]: Evaluating model on 210 examples from the 'test' dataset split:
Evaluating: 100%|████████████████████████████| 210/210 [00:00<00:00, 314.89it/s]
MASTERFUL [16:00:07]:   Loss: 0.1323
MASTERFUL [16:00:07]:   Categorical Accuracy: 0.9714
Confusion Matrix: 100%|██████████████████████| 210/210 [00:01<00:00, 119.99it/s]
MASTERFUL [16:00:09]:   Average Precision: 0.9704
MASTERFUL [16:00:09]:   Average Recall:    0.9809
MASTERFUL [16:00:09]:   Confusion Matrix:
MASTERFUL [16:00:09]:                      |      agricultural|          airplane|   baseballdiamond|             beach|         buildings|         chaparral|  denseresidential|            forest|           freeway|        golfcourse|            harbor|      intersection| mediumresidential|    mobilehomepark|          overpass|        parkinglot|             river|            runway| sparseresidential|      storagetanks|       tenniscourt|
MASTERFUL [16:00:09]:          agricultural|                12|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:              airplane|                 0|                11|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:       baseballdiamond|                 0|                 0|                 8|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:                 beach|                 0|                 0|                 0|                13|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:             buildings|                 0|                 0|                 0|                 0|                14|                 0|                 2|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 1|                 0|
MASTERFUL [16:00:09]:             chaparral|                 0|                 0|                 0|                 0|                 0|                10|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:      denseresidential|                 0|                 0|                 0|                 0|                 0|                 0|                10|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:                forest|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 9|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:               freeway|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                16|                 0|                 0|                 0|                 0|                 0|                 1|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:            golfcourse|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 6|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:                harbor|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                10|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:          intersection|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                15|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:     mediumresidential|                 0|                 0|                 0|                 0|                 0|                 0|                 1|                 0|                 0|                 0|                 0|                 0|                10|                 0|                 0|                 0|                 0|                 0|                 1|                 0|                 0|
MASTERFUL [16:00:09]:        mobilehomepark|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 9|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:              overpass|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 7|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:            parkinglot|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 8|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:                 river|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 8|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:                runway|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 8|                 0|                 0|                 0|
MASTERFUL [16:00:09]:     sparseresidential|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 9|                 0|                 0|
MASTERFUL [16:00:09]:          storagetanks|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 5|                 0|
MASTERFUL [16:00:09]:           tenniscourt|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 6|
MASTERFUL [16:00:09]:     Confusion matrix columns represent the prediction labels and the rows represent the real labels.
MASTERFUL [16:00:09]:
MASTERFUL [16:00:09]:   Per-Class Metrics:
MASTERFUL [16:00:09]:     Class agricultural:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class airplane:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class baseballdiamond:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class beach:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class buildings:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 0.8235
MASTERFUL [16:00:09]:     Class chaparral:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class denseresidential:
MASTERFUL [16:00:09]:       Precision: 0.7692
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class forest:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class freeway:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 0.9412
MASTERFUL [16:00:09]:     Class golfcourse:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class harbor:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class intersection:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class mediumresidential:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 0.8333
MASTERFUL [16:00:09]:     Class mobilehomepark:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class overpass:
MASTERFUL [16:00:09]:       Precision: 0.8750
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class parkinglot:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class river:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class runway:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class sparseresidential:
MASTERFUL [16:00:09]:       Precision: 0.9000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class storagetanks:
MASTERFUL [16:00:09]:       Precision: 0.8333
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class tenniscourt:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]: Saving evaluation metrics to /home/sam/model_output/session-00344/evaluation_metrics.csv.
MASTERFUL [16:00:09]: Saving confusion matrix to /home/sam/model_output/session-00344/confusion_matrix.csv.
MASTERFUL [16:00:09]: Total elapsed training time: 116 minutes (1h 55m 52s).
MASTERFUL [16:00:09]: Launch masterful-gui to visualize the training results: policy name 'margin-repeated-albacore'

Analyze the Results

At the end of training, Masterful will evaluate your model based on the test (sometimes called holdout) dataset you specified in the evaluation section of your configuration file. The evaluation results are different for each task, since every computer vision task has both general metrics (such as loss) as well as task-specific metrics (IoU for semantic segmentation for example). These results are printed to the console, as well as saved into a CSV file in the output directory specified in your configuration file.

Image Classification Evaluation Metrics

For Image Classification, Masterful reports three main categories of metrics: Model Loss, Model Accuracy, and Confusion Matrix.

Model Loss

The first set of metrics Masterful reports is the overall loss of the model. The following is the loss result from the above training run (this might be different if you rerun the above cells):

MASTERFUL [16:00:06]: Evaluating model on 210 examples from the 'test' dataset split:
Evaluating: 100%|████████████████████████████| 210/210 [00:00<00:00, 314.89it/s]
MASTERFUL [16:00:07]:   Loss: 0.1323

As you can see in the above output, the total loss of the model is MASTERFUL [16:00:07]:   Loss: 0.1323. This seems like a pretty good loss value. It is relatively low and close to zero, which is the goal of the optimizer in machine learning training. However, it’s really difficult to understand intuitively what this means in terms of your models overall performance. Are these good values? Or bad ones? Let’s dive into the rest of the metrics to answer these questions.

Model Accuracy

Next Masterful reports the categorical accuracy of the model. This is an overall measure of the quality of your model, and is a good quantitative measure of the performance of your model when your test dataset is well balanced.

MASTERFUL [16:00:07]:   Categorical Accuracy: 0.9714

In this instance, the model is 97% accurate, which is a really good value!

Confusion Matrix

The final set of metrics Masterful provides for Image Classification is the confusion matrix of predictions versus labels. The confusion matrix helps figure out where and how the model makes mistakes. Masterful also provides a summary of the confusion matrix in the form of Precision and Recall values.

Below is the output for the confusion matrix from the training process:

[ ]:
MASTERFUL [16:00:09]:   Confusion Matrix:
MASTERFUL [16:00:09]:                      |      agricultural|          airplane|   baseballdiamond|             beach|         buildings|         chaparral|  denseresidential|            forest|           freeway|        golfcourse|            harbor|      intersection| mediumresidential|    mobilehomepark|          overpass|        parkinglot|             river|            runway| sparseresidential|      storagetanks|       tenniscourt|
MASTERFUL [16:00:09]:          agricultural|                12|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:              airplane|                 0|                11|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:       baseballdiamond|                 0|                 0|                 8|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:                 beach|                 0|                 0|                 0|                13|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:             buildings|                 0|                 0|                 0|                 0|                14|                 0|                 2|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 1|                 0|
MASTERFUL [16:00:09]:             chaparral|                 0|                 0|                 0|                 0|                 0|                10|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:      denseresidential|                 0|                 0|                 0|                 0|                 0|                 0|                10|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:                forest|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 9|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:               freeway|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                16|                 0|                 0|                 0|                 0|                 0|                 1|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:            golfcourse|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 6|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:                harbor|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                10|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:          intersection|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                15|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:     mediumresidential|                 0|                 0|                 0|                 0|                 0|                 0|                 1|                 0|                 0|                 0|                 0|                 0|                10|                 0|                 0|                 0|                 0|                 0|                 1|                 0|                 0|
MASTERFUL [16:00:09]:        mobilehomepark|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 9|                 0|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:              overpass|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 7|                 0|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:            parkinglot|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 8|                 0|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:                 river|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 8|                 0|                 0|                 0|                 0|
MASTERFUL [16:00:09]:                runway|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 8|                 0|                 0|                 0|
MASTERFUL [16:00:09]:     sparseresidential|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 9|                 0|                 0|
MASTERFUL [16:00:09]:          storagetanks|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 5|                 0|
MASTERFUL [16:00:09]:           tenniscourt|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 0|                 6|
MASTERFUL [16:00:09]:     Confusion matrix columns represent the prediction labels and the rows represent the real labels.

The confusion matrix gives you more insight into the predictive power of your model, and importantly, helps you categorize where the model made incorrect predictions.

Per-Class Metrics

Masterful also gives you a per-class breakdown of the confusion matrix results, as well as the average precision and recall of your model. Precision measures how often your model is correct when it predicts a class, and recall measures how often your model fails to predict a class when it should.

Below you will see the average and per-class precision and recall metrics for this model:

MASTERFUL [16:00:09]:   Average Precision: 0.9704
MASTERFUL [16:00:09]:   Average Recall:    0.9809
MASTERFUL [16:00:09]:   Per-Class Metrics:
MASTERFUL [16:00:09]:     Class agricultural:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class airplane:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class baseballdiamond:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class beach:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class buildings:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 0.8235
MASTERFUL [16:00:09]:     Class chaparral:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class denseresidential:
MASTERFUL [16:00:09]:       Precision: 0.7692
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class forest:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class freeway:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 0.9412
MASTERFUL [16:00:09]:     Class golfcourse:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class harbor:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class intersection:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class mediumresidential:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 0.8333
MASTERFUL [16:00:09]:     Class mobilehomepark:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class overpass:
MASTERFUL [16:00:09]:       Precision: 0.8750
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class parkinglot:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class river:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class runway:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class sparseresidential:
MASTERFUL [16:00:09]:       Precision: 0.9000
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class storagetanks:
MASTERFUL [16:00:09]:       Precision: 0.8333
MASTERFUL [16:00:09]:       Recall   : 1.0000
MASTERFUL [16:00:09]:     Class tenniscourt:
MASTERFUL [16:00:09]:       Precision: 1.0000
MASTERFUL [16:00:09]:       Recall   : 1.0000

As you can see, the model does really well overall. Looking at the per-class metrics however, you can see that the model is a little confused on the denseresidential class. From the confusion matrix, the model makes a few mistakes on this class, incorrectly marking a few instance of building and mediumresidential as denseresidential. This is not too bad overall, as you can see how easy it is for a human as well to mis-categorize these.

View the Predictions

While the classification metrics are a good quantitative indicator of the performance of your model, visualizing the predictions can help you get a qualititative sense of how well your trained model is performing. Below you can see the predictions for the above trained model on the examples you visualized above. The predicted labels form the title of each subplot.

[12]:
import numpy as np

# Download the pretrained model so that you can
# visualize the results.
MODEL_URL = "https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/trained_model.tar.gz"

model = None
with tempfile.TemporaryDirectory() as temp_directory:
  # Download the pretrained model from S3.
  saved_model_path = os.path.join(temp_directory, "saved_model.tar.gz")
  _ =  urllib.request.urlretrieve(MODEL_URL, saved_model_path, progress_bar_factory("Downloading Model: "))

  # Extract the model weights from the tar file.
  with tarfile.open(saved_model_path) as tar:
    tar.extractall(temp_directory)
  saved_model_path = os.path.join(temp_directory, "saved_model")

  image1 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image1_path))).read(), channels=3, dtype=tf.uint8)
  image2 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image2_path))).read(), channels=3,  dtype=tf.uint8)
  image3 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image3_path))).read(), channels=3,  dtype=tf.uint8)
  image4 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image4_path))).read(), channels=3,  dtype=tf.uint8)

  # Load the trained tensorflow saved model.
  model = tf.saved_model.load(saved_model_path)

  # The default inference function for tensorflow saved
  # models is named `serving_default`.
  inference_fn = model.signatures["serving_default"]


  # The inference function is a one-arg callable, whose
  # input argument is `image` - the image to predict on,
  # and which returns a dictionary of outputs. The dictionary
  # contains an item whose key is `prediction`, which is the
  # predictions of the model.
  image1_predictions = inference_fn(image=image1)['prediction']
  image2_predictions = inference_fn(image=image2)['prediction']
  image3_predictions = inference_fn(image=image3)['prediction']
  image4_predictions = inference_fn(image=image4)['prediction']

  # Threshold the predictions to determine the class prediction. For multi-class classification,
  # we can take the maximum prediction (argmax)
  image1_predictions = np.argmax(image1_predictions.numpy())
  image2_predictions = np.argmax(image2_predictions.numpy())
  image3_predictions = np.argmax(image3_predictions.numpy())
  image4_predictions = np.argmax(image4_predictions.numpy())

  f, axarr = plt.subplots(2, 2, figsize=(15,15))
  _ = axarr[0, 0].imshow(image1)
  _ = axarr[0, 0].title.set_text(label_map[str(image1_predictions)])
  _ = axarr[0, 1].imshow(image2)
  _ = axarr[0, 1].title.set_text(label_map[str(image2_predictions)])
  _ = axarr[1, 0].imshow(image3)
  _ = axarr[1, 0].title.set_text(label_map[str(image3_predictions)])
  _ = axarr[1, 1].imshow(image4)
  _ = axarr[1, 1].title.set_text(label_map[str(image4_predictions)])

Downloading Model: 100% (88170291 of 88170291) || Elapsed Time: 0:00:10 Time:  0:00:10
../_images/notebooks_guide_cli_classification_land_use_20_1.png

As you can see in the above predictions, the model correctly classifies all four images.

Using the Model for Inference

The Output Formats guide has more information about how to use the models output by Masterful. You made some predictions above in the View the Predictions section, so let’s recap the important parts below:

[15]:
# This image corresponds to the first image (upper left)
# in the `View the Predictions` section above.
IMAGE_URL = "https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/images/golfcourse17.png"

# Load the trained model so that you can test running inference.
MODEL_URL = "https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/trained_model.tar.gz"

model = None
with tempfile.TemporaryDirectory() as temp_directory:
  # Download the pretrained model from S3.
  saved_model_path = os.path.join(temp_directory, "saved_model.tar.gz")
  _ =  urllib.request.urlretrieve(MODEL_URL, saved_model_path, progress_bar_factory("Downloading Model: "))

  # Extract the model weights from the tar file.
  with tarfile.open(saved_model_path) as tar:
    tar.extractall(temp_directory)
  saved_model_path = os.path.join(temp_directory, "saved_model")

  # Load the trained tensorflow saved model.
  model = tf.saved_model.load(saved_model_path)

# Download the image to use for generating a prediction
# and convert it into a tensor.
http_response = requests.get(IMAGE_URL, stream=True)
image = tf.io.decode_image(http_response.raw.read())

image_height = image.shape[0]
image_width = image.shape[1]
print(f"Image Shape: height={image_height} width={image_width}.")

# The default inference function for tensorflow saved
# models is named `serving_default`.
inference_fn = model.signatures["serving_default"]

# The inference function is a one-arg callable, whose
# input argument is `image` - the image to predict on,
# and which returns a dictionary of outputs. The dictionary
# contains an item whose key is `prediction`, which is the
# predictions of the model.
predictions = inference_fn(image=image)['prediction']

# Convert the predictions to NumPY for easier processing.
predictions = predictions.numpy()

# The predictions from a classification model
# have the shape [batch size, number of classes].
# The batch size will always be 1 for inference models,
# since they expect only one image for to predict on.
print(f"Predictions shape is {predictions.shape}")

class_prediction = np.argmax(predictions)
print(f"The model predicted class '{label_map[str(class_prediction)]}'.")
Downloading Model: 100% (88170291 of 88170291) || Elapsed Time: 0:00:09 Time:  0:00:09
Image Shape: height=256 width=256.
Predictions shape is (1, 21)
The model predicted class 'golfcourse'.

Next Steps

In the above sections, you have learned how to train an image classification model using the Masterful CLI, evaluate the performance of the model, and use the model to make predictions on your production data.

If you want to look at more computer vision tasks, you can try Detecting Digits. If you want to look at another, more involved Object Detection example, see Detecting Pedestrians in Street Level Imagery for more information. Otherwise, feel free to explore Segmenting Animals for other examples of using the Masterful CLI Trainer to solve challenging computer vision problems.