Classification of Land Use with the Masterful CLI Trainer¶
In this guide, you will take a deeper look at Classification with the Masterful CLI Trainer to train a state of the art model.
This guide will use a satellite imagery dataset from UCMerced to classify different plots of land according to their usage.
Define the Application¶
In this guide, your challenge is to build a computer vision model that can classify different plots of land according to their usage. In this instance, the images have already been extracted from larger satellite imagery and each image can be classified into a single category.
Since each image can be categorized into a single class, and there are more than one possible classes, the computer vision task for this problem is multi-class classification.
TL;DR¶
Don’t want to read the rest of this guide, and want to start training immediately? The following command shows you how to start training with Masterful, using a configuration file and dataset on S3.
masterful-train --config https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/training.yaml
Prepare the Data¶
For this guide, you will use the UCMerced dataset, which is a land use classification dataset used for image classification. In this section, you will take the raw dataset and convert it into a format that Masterful can understand. Along the way, you will visualize a few examples of the data to see both the input to the model, as well as the predictions after training the model.
The UCMerced dataset contains 100 images for each of 21 different classes. Each image measures 256x256 pixels. The images were manually extracted from large images from the USGS National Map Urban Area Imagery collection for various urban areas around the country. The pixel resolution of this public domain imagery is 1 foot.
The CSV format for Image Classification can be found at Masterful Image Classification. Each line contains the relative path to the image, and the 0-indexed integer of the groundtruth class. For example, the following are the first few lines from the trainval.csv file:
images/golfcourse17.png, 9
images/intersection98.png, 11
images/agricultural59.png, 0
images/overpass28.png, 14
images/agricultural81.png, 0
images/freeway01.png, 8
images/river77.png, 16
.
.
.
The label map file helps convert the integer class indices into human readable forms. The label map for UCMerced looks like:
0,agricultural
1,airplane
2,baseballdiamond
3,beach
4,buildings
5,chaparral
6,denseresidential
7,forest
8,freeway
9,golfcourse
10,harbor
11,intersection
12,mediumresidential
13,mobilehomepark
14,overpass
15,parkinglot
16,river
17,runway
18,sparseresidential
19,storagetanks
20,tenniscourt
Converting each dataset is typically a one-off operation that is different for every dataset you want to train with. For brevity, you can use the already converted dataset located at the public S3 bucket s3://masterful-public/datasets/ucmerced/. In this bucket, you will see the following files:
ucmerced\
test.csv
trainval.csv
training.yaml
label_map.csv
images\
Explore the Data¶
You should always visually inspect your dataset to get a sense of what the model will see, and to roughly verify that your dataset conversion routine worked properly and you are not training with corrupted or incorrect data.
[2]:
# Install dependencies necessary to run the following
# code.
!pip install opencv-python-headless --quiet
!pip install masterful --quiet
# Import the packages used below.
import matplotlib.pyplot as plt
import os
import requests
import tarfile
import tempfile
import tensorflow as tf
import urllib.request
# Import and activate the Masterful package.
import masterful
masterful = masterful.activate()
# Helper function to display a progress when downloading
# a file using HTTP.
from masterful.utils.downloader import progress_bar_factory
# This is necessary for running inside of Colab/Jupyter,
# since the CLI trainer is run outside of the kernel
# (as a script command).
gpus = tf.config.experimental.list_physical_devices('GPU')
for gpu in gpus:
tf.config.experimental.set_memory_growth(gpu, True)
MASTERFUL: Your account has been successfully activated. Masterful v0.5.2 is loaded.
[11]:
DATASET_ROOT = "https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/"
TRAINING_CSV_URL = os.path.join(DATASET_ROOT, "trainval.csv")
with tempfile.TemporaryDirectory() as temp_directory:
training_csv_name = os.path.join(temp_directory, "trainval.csv")
_ = urllib.request.urlretrieve(TRAINING_CSV_URL, training_csv_name)
examples = []
with open(training_csv_name) as training_csv:
examples = training_csv.readlines()
image1_path, label1 = examples[0].split(',')
image2_path, label2 = examples[1].split(',')
image3_path, label3 = examples[2].split(',')
image4_path, label4 = examples[3].split(',')
label_map = {
"0": "agricultural",
"1": "airplane",
"2": "baseballdiamond",
"3": "beach",
"4": "buildings",
"5": "chaparral",
"6": "denseresidential",
"7": "forest",
"8": "freeway",
"9": "golfcourse",
"10": "harbor",
"11": "intersection",
"12": "mediumresidential",
"13": "mobilehomepark",
"14": "overpass",
"15": "parkinglot",
"16": "river",
"17": "runway",
"18": "sparseresidential",
"19": "storagetanks",
"20": "tenniscourt",
}
image1 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image1_path))).read(), channels=3, dtype=tf.float32)
image2 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image2_path))).read(), channels=3, dtype=tf.float32)
image3 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image3_path))).read(), channels=3, dtype=tf.float32)
image4 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image4_path))).read(), channels=3, dtype=tf.float32)
f, axarr = plt.subplots(2, 2, figsize=(15,15))
_ = axarr[0, 0].imshow(image1)
_ = axarr[0, 0].title.set_text(label_map[label1.strip()])
_ = axarr[0, 1].imshow(image2)
_ = axarr[0, 1].title.set_text(label_map[label2.strip()])
_ = axarr[1, 0].imshow(image3)
_ = axarr[1, 0].title.set_text(label_map[label3.strip()])
_ = axarr[1, 1].imshow(image4)
_ = axarr[1, 1].title.set_text(label_map[label4.strip()])
Configure the CLI Trainer¶
The Masterful CLI Trainer is a command line tool that trains a production quality model with no code required. The Masterful CLI Trainer take a YAML configuration file as input. The configuration file fully specifies everything necessary for training, including the dataset, model, export formats, and evaluation metrics.
Choosing a Model¶
Masterful provides many different state of the art image classification models that you can choose from. In general, choosing a model can have different constraints than training the model (are you deploying on server or edge? Runtime or latency constraints? Memory constraints?). You generally want to choose a model that is large enough to fit your data, but not so large as to overfit the training data and “memorize” the results, which can lead to poor generalization performance.
The Masterful Image Classification Model Zoo supports the following model architectures.
Model Name |
Year |
Description |
|---|---|---|
|
2015 |
ResNet-50 architecture from the paper Deep Residual Learning for Image Recognition |
|
2015 |
ResNet-101 architecture from the paper Deep Residual Learning for Image Recognition |
|
2015 |
ResNet-152 architecture from the paper Deep Residual Learning for Image Recognition |
|
2016 |
ResNet-50 architecture from the paper Identity Mappings in Deep Residual Networks |
|
2016 |
ResNet-101 architecture from the paper Identity Mappings in Deep Residual Networks |
|
2016 |
ResNet-152 architecture from the paper Identity Mappings in Deep Residual Networks |
|
2019 |
|
|
2019 |
|
|
2019 |
|
|
2019 |
|
|
2019 |
|
|
2019 |
|
|
2019 |
|
|
2019 |
|
|
2019 |
|
|
2019 |
The |
|
2019 |
The |
|
2015 |
The |
|
2015 |
The |
|
2017 |
The |
|
2016 |
The |
In the example below, you will select a resnet50v1 model. This is a great model to start with, as it generalizes well across most domains.
The Configuration File¶
The following section is a condensed YAML configuration file for training against this dataset with NO unlabeled data. It points to the dataset created above, and selects a resnet50v1 model, which is a good model for small datasets and edge inference use cases.
Note all of the model artifacts will be saved to the directory ~/model_output, which you can configure in the output section below.
The original YAML file can be found here.
dataset:
root_path: https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced
splits: [trainval, test]
label_map: label_map
optimize: True
model:
architecture: resnet50v1
num_classes: 21
input_shape: [256,256,3]
training:
task: classification
training_split: trainval
output:
formats: [saved_model, onnx]
path: ~/model_output
evaluation:
split: test
Train the Model¶
Training the model involves simply running the Masterful CLI Trainer (masterful-train) against the YAML file on S3.
Below, you will setup the CLI and configuration file to run in this notebook. However, if you want to train this model outside of the notebook, against the dataset on S3, you can run:
masterful-train --config https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/training.yaml
[1]:
# Use the Masterful CLI to train the model without unlabeled data
!masterful-train --config https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/training.yaml
MASTERFUL: Your account has been successfully activated. Masterful v0.5.2 is loaded.
MASTERFUL [14:04:16]: Training with configuration 'https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/training.yaml':
---------- --------------------------------------------------------------------------------
dataset root_path https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced
splits ['trainval', 'test']
label_map label_map
optimize True
model architecture resnet50v1
num_classes 21
input_shape [256, 256, 3]
training task classification
training_split trainval
output formats ['saved_model', 'onnx']
path ~/model_output
evaluation split test
---------- --------------------------------------------------------------------------------
MASTERFUL [14:04:20]: Building model 'resnet50v1'...
MASTERFUL [14:04:22]: Using model resnet50v1 with:
MASTERFUL [14:04:22]: 23630741 total parameters
MASTERFUL [14:04:22]: 23577621 trainable parameters
MASTERFUL [14:04:22]: 53120 untrainable parameters
MASTERFUL [14:04:22]: Caching I/O optimized dataset split 'trainval' to /home/sam/.masterful/datasets/2723296d09d4f0d7f7fb923db3237bc6ca4f7668...
100%|███████████████████████████████████████| 1890/1890 [00:46<00:00, 40.83it/s]
MASTERFUL [14:05:10]: Caching I/O optimized dataset split 'test' to /home/sam/.masterful/datasets/538fb0c1083cebe30f260bcc37f79adfb8bdde92...
100%|█████████████████████████████████████████| 210/210 [00:05<00:00, 40.95it/s]
MASTERFUL [14:05:17]: Dataset Summary:
MASTERFUL [14:05:17]: Training Dataset: 1890 examples.
MASTERFUL [14:05:17]: Validation Dataset: 0 examples.
MASTERFUL [14:05:17]: Test Dataset: 210 examples.
MASTERFUL [14:05:17]: Unlabeled Dataset: 0 examples.
MASTERFUL [14:05:17]: Training Dataset Analysis:
100%|██████████████████████████████████████| 1890/1890 [00:07<00:00, 258.13it/s]
MASTERFUL [14:05:24]: Training dataset analysis finished at 14:05:24 in 7 seconds (7s), returned:
------------------ ----------------------------------------
Total Examples 1890
Label Counts agricultural 88
airplane 89
baseballdiamond 92
beach 87
buildings 83
chaparral 90
denseresidential 90
forest 91
freeway 83
golfcourse 94
harbor 90
intersection 85
mediumresidential 88
mobilehomepark 91
overpass 93
parkinglot 92
river 92
runway 92
sparseresidential 91
storagetanks 95
tenniscourt 94
Label Distribution agricultural 0.0465608
airplane 0.0470899
baseballdiamond 0.0486772
beach 0.0460317
buildings 0.0439153
chaparral 0.047619
denseresidential 0.047619
forest 0.0481481
freeway 0.0439153
golfcourse 0.0497354
harbor 0.047619
intersection 0.0449735
mediumresidential 0.0465608
mobilehomepark 0.0481481
overpass 0.0492063
parkinglot 0.0486772
river 0.0486772
runway 0.0486772
sparseresidential 0.0481481
storagetanks 0.0502646
tenniscourt 0.0497354
Balanced Yes
Per Channel Mean [123.41418882 124.86677271 114.86978589]
Per Channel StdDev [44.09977763 41.55092148 39.49503547]
Min Height 247
Min Width 242
Max Height 257
Max Width 257
Average Height 256
Average Width 256
Largest Image (257, 257, 3)
Smallest Image (247, 247, 3)
Duplicates 0
------------------ ----------------------------------------
MASTERFUL [14:05:24]: Test Dataset Analysis:
100%|████████████████████████████████████████| 210/210 [00:00<00:00, 251.64it/s]
MASTERFUL [14:05:25]: Test dataset analysis finished at 14:05:25 in 1 seconds (1s), returned:
------------------ ----------------------------------------
Total Examples 210
Label Counts agricultural 12
airplane 11
baseballdiamond 8
beach 13
buildings 17
chaparral 10
denseresidential 10
forest 9
freeway 17
golfcourse 6
harbor 10
intersection 15
mediumresidential 12
mobilehomepark 9
overpass 7
parkinglot 8
river 8
runway 8
sparseresidential 9
storagetanks 5
tenniscourt 6
Label Distribution agricultural 0.0571429
airplane 0.052381
baseballdiamond 0.0380952
beach 0.0619048
buildings 0.0809524
chaparral 0.047619
denseresidential 0.047619
forest 0.0428571
freeway 0.0809524
golfcourse 0.0285714
harbor 0.047619
intersection 0.0714286
mediumresidential 0.0571429
mobilehomepark 0.0428571
overpass 0.0333333
parkinglot 0.0380952
river 0.0380952
runway 0.0380952
sparseresidential 0.0428571
storagetanks 0.0238095
tenniscourt 0.0285714
Balanced Yes
Per Channel Mean [124.01575081 125.79742028 114.89969612]
Per Channel StdDev [45.46955658 43.01082001 40.9943777 ]
Min Height 247
Min Width 247
Max Height 256
Max Width 256
Average Height 256
Average Width 256
Largest Image (256, 256, 3)
Smallest Image (247, 247, 3)
Duplicates 0
------------------ ----------------------------------------
MASTERFUL [14:05:25]: Cross-Dataset Analysis:
MASTERFUL [14:05:25]: Cross-Dataset analysis finished at 14:05:25 in 0 seconds (0s), returned:
-------- -----------
trainval trainval 0
test 1
test trainval 1
test 0
-------- -----------
MASTERFUL [14:05:25]: WARNING: Duplicates detected in dataset split 'test'.
MASTERFUL [14:05:25]: WARNING: You can find the duplicate entries using the tool:
MASTERFUL [14:05:25]: WARNING: python -m masterful.data.duplicate_detector --config=https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/training.yaml
MASTERFUL [14:05:25]: Meta-Learning architecture parameters...
MASTERFUL [14:05:26]: Architecture learner finished at 14:05:26 in 1 seconds (1s), returned:
------------------------------ -----------------------------
task Task.CLASSIFICATION
num_classes 21
ensemble_multiplier 1
custom_objects {}
model_config
backbone_only False
input_shape (256, 256, 3)
input_range ImageRange.IMAGENET_CAFFE_BGR
input_dtype <dtype: 'float32'>
input_channels_last True
prediction_logits True
prediction_dtype <dtype: 'float32'>
prediction_structure TensorStructure.SINGLE_TENSOR
prediction_shape (21,)
total_parameters 23630741
total_trainable_parameters 23577621
total_non_trainable_parameters 53120
------------------------------ -----------------------------
MASTERFUL [14:05:26]: Meta-learning training dataset parameters...
MASTERFUL [14:05:27]: Training dataset learner finished at 14:05:27 in 1 seconds (1s), returned:
------------------------- -----------------------------
num_classes 21
task Task.CLASSIFICATION
image_shape (256, 256, 3)
image_range ImageRange.IMAGENET_CAFFE_BGR
image_dtype <dtype: 'float32'>
image_channels_last True
label_dtype <dtype: 'float32'>
label_shape (21,)
label_structure TensorStructure.SINGLE_TENSOR
label_sparse False
label_bounding_box_format
------------------------- -----------------------------
MASTERFUL [14:05:27]: Meta-learning test dataset parameters...
MASTERFUL [14:05:28]: Test dataset learner finished at 14:05:28 in 1 seconds (1s), returned:
------------------------- -----------------------------
num_classes 21
task Task.CLASSIFICATION
image_shape (256, 256, 3)
image_range ImageRange.IMAGENET_CAFFE_BGR
image_dtype <dtype: 'float32'>
image_channels_last True
label_dtype <dtype: 'float32'>
label_shape (21,)
label_structure TensorStructure.SINGLE_TENSOR
label_sparse False
label_bounding_box_format
------------------------- -----------------------------
MASTERFUL [14:05:28]: Meta-Learning optimization parameters...
Callbacks: 100%|███████████████████████████████| 8/8 [01:21<00:00, 10.19s/steps]
MASTERFUL [14:06:50]: Optimization learner finished at 14:06:50 in 82 seconds (1m 22s), returned:
----------------------- -----------------------------------------------------------------
batch_size 32
drop_remainder False
epochs 1000000
learning_rate 0.0012499999720603228
learning_rate_schedule
learning_rate_callback <keras.callbacks.ReduceLROnPlateau object at 0x7f47986ea080>
warmup_learning_rate 1e-06
warmup_epochs 5
optimizer <tensorflow_addons.optimizers.lamb.LAMB object at 0x7f479869dd68>
loss <keras.losses.CategoricalCrossentropy object at 0x7f47a00df160>
loss_weights
early_stopping_callback <keras.callbacks.EarlyStopping object at 0x7f47a00716a0>
metrics [<keras.metrics.CategoricalAccuracy object at 0x7f47987ddba8>]
readonly_callbacks
----------------------- -----------------------------------------------------------------
MASTERFUL [14:06:52]: Meta-Learning Regularization Parameters...
MASTERFUL [14:06:58]: Warming up model for analysis.
MASTERFUL [14:07:04]: Warming up batch norm statistics (this could take a few minutes).
MASTERFUL [14:07:07]: Warming up training for 505 steps.
100%|██████████████████████████████████████| 505/505 [01:55<00:00, 4.36steps/s]
MASTERFUL [14:09:03]: Validating batch norm statistics after warmup for stability (this could take a few minutes).
MASTERFUL [14:09:51]: Analyzing baseline model performance. Training until validation loss stabilizes...
Baseline Training: 100%|█████████████████| 3900/3900 [11:46<00:00, 5.52steps/s]
MASTERFUL [14:21:56]: Baseline training complete.
MASTERFUL [14:21:56]: Meta-Learning Basic Data Augmentations...
Node 1/4: 100%|██████████████████████████| 1200/1200 [03:29<00:00, 5.72steps/s]
Node 2/4: 100%|██████████████████████████| 1200/1200 [03:32<00:00, 5.66steps/s]
Node 3/4: 100%|██████████████████████████| 1200/1200 [03:33<00:00, 5.62steps/s]
Node 4/4: 100%|██████████████████████████| 1200/1200 [03:33<00:00, 5.63steps/s]
MASTERFUL [14:37:08]: Meta-Learning Data Augmentation Clusters...
Distance Analysis: 100%|███████████████████| 143/143 [02:16<00:00, 1.05steps/s]
Node 1/10: 100%|█████████████████████████| 1200/1200 [04:23<00:00, 4.56steps/s]
Node 2/10: 100%|█████████████████████████| 1200/1200 [04:22<00:00, 4.56steps/s]
Node 3/10: 100%|█████████████████████████| 1200/1200 [04:23<00:00, 4.55steps/s]
Node 4/10: 100%|█████████████████████████| 1200/1200 [04:23<00:00, 4.55steps/s]
Node 5/10: 100%|█████████████████████████| 1200/1200 [04:23<00:00, 4.56steps/s]
Distance Analysis: 100%|█████████████████████| 66/66 [01:03<00:00, 1.04steps/s]
Node 6/10: 100%|█████████████████████████| 1200/1200 [04:40<00:00, 4.27steps/s]
Node 7/10: 100%|█████████████████████████| 1200/1200 [04:40<00:00, 4.28steps/s]
Node 8/10: 100%|█████████████████████████| 1200/1200 [04:39<00:00, 4.29steps/s]
Node 9/10: 100%|█████████████████████████| 1200/1200 [04:40<00:00, 4.28steps/s]
Node 10/10: 100%|████████████████████████| 1200/1200 [04:40<00:00, 4.27steps/s]
MASTERFUL [15:28:45]: Meta-Learning Label Based Regularization...
Node 1/2: 100%|██████████████████████████| 1200/1200 [04:27<00:00, 4.49steps/s]
Node 2/2: 100%|██████████████████████████| 1200/1200 [04:28<00:00, 4.46steps/s]
MASTERFUL [15:38:17]: Meta-Learning Weight Based Regularization...
MASTERFUL [15:38:18]: Analysis finished in 91.32490015029907 minutes.
MASTERFUL [15:38:18]: Learned parameters margin-repeated-albacore saved at /home/sam/.masterful/policies/margin-repeated-albacore.
MASTERFUL [15:38:18]: Regularization learner finished at 15:38:18 in 5488 seconds (1h 31m 28s), returned:
------------------------- -----------------------------------------------
shuffle_buffer_size 1701
mirror 1.0
rot90 1.0
rotate 0
mixup 0.75
cutmix 0.0
label_smoothing 0
hsv_cluster 4
hsv_cluster_to_index [[ 1 2 6 9 11 11]
[ 2 4 6 8 11 11]
[ 1 1 3 4 5 11]
[ 1 1 3 4 7 11]
[ 2 2 3 5 8 11]]
hsv_magnitude_table [[ 0 10 20 30 40 50 60 70 80 90 100]
[ 0 20 10 30 40 50 60 70 80 90 100]
[ 0 10 20 30 40 50 60 70 80 90 100]
[ 0 10 20 30 40 50 60 70 80 100 90]
[100 0 10 90 80 20 30 70 40 60 50]]
contrast_cluster 4
contrast_cluster_to_index [[ 1 3 8 11 11 11]
[ 1 1 1 1 5 11]
[ 2 5 6 7 9 11]
[ 1 2 5 10 11 11]
[ 1 2 4 8 11 11]
[ 1 3 5 6 8 11]]
contrast_magnitude_table [[ 0 10 20 30 40 50 60 70 80 90 100]
[ 0 10 20 30 50 40 60 70 80 90 100]
[ 0 10 20 30 40 50 60 70 80 90 100]
[ 0 10 20 30 40 50 60 70 80 90 100]
[ 0 10 20 30 40 50 60 70 80 90 100]
[ 0 10 20 30 40 50 60 70 80 90 100]]
blur_cluster 4
blur_cluster_to_index [[ 2 11 11 11 11 11]
[ 8 11 11 11 11 11]]
blur_magnitude_table [[ 0 10 20 30 40 50 60 70 80 90 100]
[ 0 60 10 50 20 40 30 70 80 90 100]]
spatial_cluster 0
spatial_cluster_to_index [[ 2 4 6 8 10 11]
[ 2 4 6 8 10 11]
[ 1 5 8 9 11 11]
[ 2 3 6 10 11 11]
[ 3 5 6 8 11 11]
[ 1 1 1 1 2 11]]
spatial_magnitude_table [[ 0 10 20 30 40 50 60 70 80 90 100]
[ 0 10 20 30 40 50 60 70 80 90 100]
[ 0 10 20 100 80 90 70 30 40 60 50]
[ 0 10 20 90 100 80 30 40 70 60 50]
[ 0 10 20 30 40 50 60 70 80 90 100]
[ 0 10 20 30 40 50 60 70 80 90 100]]
synthetic_proportion [0.0]
------------------------- -----------------------------------------------
MASTERFUL [15:38:18]: Learning SSL parameters...
MASTERFUL [15:38:19]: SSL learner finished at 15:38:19 in 1 seconds (1s), returned:
---------- --
algorithms []
---------- --
MASTERFUL [15:38:19]: Training model with semi-supervised learning disabled.
MASTERFUL [15:38:20]: Performing basic dataset analysis.
MASTERFUL [15:38:26]: Masterful will use 189 labeled examples as a validation set since no validation data was provided.
MASTERFUL [15:38:26]: Training model with:
MASTERFUL [15:38:26]: 1701 labeled examples.
MASTERFUL [15:38:26]: 189 validation examples.
MASTERFUL [15:38:26]: 0 synthetic examples.
MASTERFUL [15:38:26]: 0 unlabeled examples.
MASTERFUL [15:38:27]: Training model with learned parameters margin-repeated-albacore in two phases.
MASTERFUL [15:38:27]: The first phase is supervised training with the learned parameters.
MASTERFUL [15:38:27]: The second phase is semi-supervised training to boost performance.
MASTERFUL [15:38:29]: Warming up model for supervised training.
MASTERFUL [15:38:34]: Warming up batch norm statistics (this could take a few minutes).
MASTERFUL [15:38:38]: Warming up training for 505 steps.
100%|██████████████████████████████████████| 505/505 [02:11<00:00, 3.85steps/s]
MASTERFUL [15:40:49]: Validating batch norm statistics after warmup for stability (this could take a few minutes).
MASTERFUL [15:41:10]: Starting Phase 1: Supervised training until the validation loss stabilizes...
Supervised Training: 100%|███████████████| 4020/4020 [17:22<00:00, 3.86steps/s]
MASTERFUL [15:58:56]: Semi-Supervised training disabled in parameters.
MASTERFUL [15:58:57]: Training complete in 20.49317009449005 minutes.
MASTERFUL [15:59:14]: Saving model output to /home/sam/model_output/session-00344.
MASTERFUL [15:59:14]: Saving saved_model output to /home/sam/model_output/session-00344/saved_model
MASTERFUL [15:59:32]: Saving onnx output to /home/sam/model_output/session-00344/onnx
MASTERFUL [16:00:06]: Saving regularization params to /home/sam/model_output/session-00344/regularization.params.
MASTERFUL [16:00:06]: ************************************
MASTERFUL [16:00:06]: Evaluating model on 210 examples from the 'test' dataset split:
Evaluating: 100%|████████████████████████████| 210/210 [00:00<00:00, 314.89it/s]
MASTERFUL [16:00:07]: Loss: 0.1323
MASTERFUL [16:00:07]: Categorical Accuracy: 0.9714
Confusion Matrix: 100%|██████████████████████| 210/210 [00:01<00:00, 119.99it/s]
MASTERFUL [16:00:09]: Average Precision: 0.9704
MASTERFUL [16:00:09]: Average Recall: 0.9809
MASTERFUL [16:00:09]: Confusion Matrix:
MASTERFUL [16:00:09]: | agricultural| airplane| baseballdiamond| beach| buildings| chaparral| denseresidential| forest| freeway| golfcourse| harbor| intersection| mediumresidential| mobilehomepark| overpass| parkinglot| river| runway| sparseresidential| storagetanks| tenniscourt|
MASTERFUL [16:00:09]: agricultural| 12| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: airplane| 0| 11| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: baseballdiamond| 0| 0| 8| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: beach| 0| 0| 0| 13| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: buildings| 0| 0| 0| 0| 14| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 1| 0|
MASTERFUL [16:00:09]: chaparral| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: denseresidential| 0| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: forest| 0| 0| 0| 0| 0| 0| 0| 9| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: freeway| 0| 0| 0| 0| 0| 0| 0| 0| 16| 0| 0| 0| 0| 0| 1| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: golfcourse| 0| 0| 0| 0| 0| 0| 0| 0| 0| 6| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: harbor| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: intersection| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 15| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: mediumresidential| 0| 0| 0| 0| 0| 0| 1| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 1| 0| 0|
MASTERFUL [16:00:09]: mobilehomepark| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: overpass| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: parkinglot| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 8| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: river| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 8| 0| 0| 0| 0|
MASTERFUL [16:00:09]: runway| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 8| 0| 0| 0|
MASTERFUL [16:00:09]: sparseresidential| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0| 0|
MASTERFUL [16:00:09]: storagetanks| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 5| 0|
MASTERFUL [16:00:09]: tenniscourt| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 6|
MASTERFUL [16:00:09]: Confusion matrix columns represent the prediction labels and the rows represent the real labels.
MASTERFUL [16:00:09]:
MASTERFUL [16:00:09]: Per-Class Metrics:
MASTERFUL [16:00:09]: Class agricultural:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class airplane:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class baseballdiamond:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class beach:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class buildings:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 0.8235
MASTERFUL [16:00:09]: Class chaparral:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class denseresidential:
MASTERFUL [16:00:09]: Precision: 0.7692
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class forest:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class freeway:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 0.9412
MASTERFUL [16:00:09]: Class golfcourse:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class harbor:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class intersection:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class mediumresidential:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 0.8333
MASTERFUL [16:00:09]: Class mobilehomepark:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class overpass:
MASTERFUL [16:00:09]: Precision: 0.8750
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class parkinglot:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class river:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class runway:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class sparseresidential:
MASTERFUL [16:00:09]: Precision: 0.9000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class storagetanks:
MASTERFUL [16:00:09]: Precision: 0.8333
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class tenniscourt:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Saving evaluation metrics to /home/sam/model_output/session-00344/evaluation_metrics.csv.
MASTERFUL [16:00:09]: Saving confusion matrix to /home/sam/model_output/session-00344/confusion_matrix.csv.
MASTERFUL [16:00:09]: Total elapsed training time: 116 minutes (1h 55m 52s).
MASTERFUL [16:00:09]: Launch masterful-gui to visualize the training results: policy name 'margin-repeated-albacore'
Analyze the Results¶
At the end of training, Masterful will evaluate your model based on the test (sometimes called holdout) dataset you specified in the evaluation section of your configuration file. The evaluation results are different for each task, since every computer vision task has both general metrics (such as loss) as well as task-specific metrics (IoU for semantic segmentation for example). These results are printed to the console, as well as saved into a CSV file in the output directory specified in
your configuration file.
Image Classification Evaluation Metrics¶
For Image Classification, Masterful reports three main categories of metrics: Model Loss, Model Accuracy, and Confusion Matrix.
Model Loss¶
The first set of metrics Masterful reports is the overall loss of the model. The following is the loss result from the above training run (this might be different if you rerun the above cells):
MASTERFUL [16:00:06]: Evaluating model on 210 examples from the 'test' dataset split:
Evaluating: 100%|████████████████████████████| 210/210 [00:00<00:00, 314.89it/s]
MASTERFUL [16:00:07]: Loss: 0.1323
As you can see in the above output, the total loss of the model is MASTERFUL [16:00:07]: Loss: 0.1323. This seems like a pretty good loss value. It is relatively low and close to zero, which is the goal of the optimizer in machine learning training. However, it’s really difficult to understand intuitively what this means in terms of your models overall performance. Are these good values? Or bad ones? Let’s dive into the rest of the metrics to answer these questions.
Model Accuracy¶
Next Masterful reports the categorical accuracy of the model. This is an overall measure of the quality of your model, and is a good quantitative measure of the performance of your model when your test dataset is well balanced.
MASTERFUL [16:00:07]: Categorical Accuracy: 0.9714
In this instance, the model is 97% accurate, which is a really good value!
Confusion Matrix¶
The final set of metrics Masterful provides for Image Classification is the confusion matrix of predictions versus labels. The confusion matrix helps figure out where and how the model makes mistakes. Masterful also provides a summary of the confusion matrix in the form of Precision and Recall values.
Below is the output for the confusion matrix from the training process:
[ ]:
MASTERFUL [16:00:09]: Confusion Matrix:
MASTERFUL [16:00:09]: | agricultural| airplane| baseballdiamond| beach| buildings| chaparral| denseresidential| forest| freeway| golfcourse| harbor| intersection| mediumresidential| mobilehomepark| overpass| parkinglot| river| runway| sparseresidential| storagetanks| tenniscourt|
MASTERFUL [16:00:09]: agricultural| 12| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: airplane| 0| 11| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: baseballdiamond| 0| 0| 8| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: beach| 0| 0| 0| 13| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: buildings| 0| 0| 0| 0| 14| 0| 2| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 1| 0|
MASTERFUL [16:00:09]: chaparral| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: denseresidential| 0| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: forest| 0| 0| 0| 0| 0| 0| 0| 9| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: freeway| 0| 0| 0| 0| 0| 0| 0| 0| 16| 0| 0| 0| 0| 0| 1| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: golfcourse| 0| 0| 0| 0| 0| 0| 0| 0| 0| 6| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: harbor| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: intersection| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 15| 0| 0| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: mediumresidential| 0| 0| 0| 0| 0| 0| 1| 0| 0| 0| 0| 0| 10| 0| 0| 0| 0| 0| 1| 0| 0|
MASTERFUL [16:00:09]: mobilehomepark| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: overpass| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 7| 0| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: parkinglot| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 8| 0| 0| 0| 0| 0|
MASTERFUL [16:00:09]: river| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 8| 0| 0| 0| 0|
MASTERFUL [16:00:09]: runway| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 8| 0| 0| 0|
MASTERFUL [16:00:09]: sparseresidential| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 9| 0| 0|
MASTERFUL [16:00:09]: storagetanks| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 5| 0|
MASTERFUL [16:00:09]: tenniscourt| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 0| 6|
MASTERFUL [16:00:09]: Confusion matrix columns represent the prediction labels and the rows represent the real labels.
The confusion matrix gives you more insight into the predictive power of your model, and importantly, helps you categorize where the model made incorrect predictions.
Per-Class Metrics¶
Masterful also gives you a per-class breakdown of the confusion matrix results, as well as the average precision and recall of your model. Precision measures how often your model is correct when it predicts a class, and recall measures how often your model fails to predict a class when it should.
Below you will see the average and per-class precision and recall metrics for this model:
MASTERFUL [16:00:09]: Average Precision: 0.9704
MASTERFUL [16:00:09]: Average Recall: 0.9809
MASTERFUL [16:00:09]: Per-Class Metrics:
MASTERFUL [16:00:09]: Class agricultural:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class airplane:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class baseballdiamond:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class beach:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class buildings:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 0.8235
MASTERFUL [16:00:09]: Class chaparral:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class denseresidential:
MASTERFUL [16:00:09]: Precision: 0.7692
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class forest:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class freeway:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 0.9412
MASTERFUL [16:00:09]: Class golfcourse:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class harbor:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class intersection:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class mediumresidential:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 0.8333
MASTERFUL [16:00:09]: Class mobilehomepark:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class overpass:
MASTERFUL [16:00:09]: Precision: 0.8750
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class parkinglot:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class river:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class runway:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class sparseresidential:
MASTERFUL [16:00:09]: Precision: 0.9000
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class storagetanks:
MASTERFUL [16:00:09]: Precision: 0.8333
MASTERFUL [16:00:09]: Recall : 1.0000
MASTERFUL [16:00:09]: Class tenniscourt:
MASTERFUL [16:00:09]: Precision: 1.0000
MASTERFUL [16:00:09]: Recall : 1.0000
As you can see, the model does really well overall. Looking at the per-class metrics however, you can see that the model is a little confused on the denseresidential class. From the confusion matrix, the model makes a few mistakes on this class, incorrectly marking a few instance of building and mediumresidential as denseresidential. This is not too bad overall, as you can see how easy it is for a human as well to mis-categorize these.
View the Predictions¶
While the classification metrics are a good quantitative indicator of the performance of your model, visualizing the predictions can help you get a qualititative sense of how well your trained model is performing. Below you can see the predictions for the above trained model on the examples you visualized above. The predicted labels form the title of each subplot.
[12]:
import numpy as np
# Download the pretrained model so that you can
# visualize the results.
MODEL_URL = "https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/trained_model.tar.gz"
model = None
with tempfile.TemporaryDirectory() as temp_directory:
# Download the pretrained model from S3.
saved_model_path = os.path.join(temp_directory, "saved_model.tar.gz")
_ = urllib.request.urlretrieve(MODEL_URL, saved_model_path, progress_bar_factory("Downloading Model: "))
# Extract the model weights from the tar file.
with tarfile.open(saved_model_path) as tar:
tar.extractall(temp_directory)
saved_model_path = os.path.join(temp_directory, "saved_model")
image1 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image1_path))).read(), channels=3, dtype=tf.uint8)
image2 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image2_path))).read(), channels=3, dtype=tf.uint8)
image3 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image3_path))).read(), channels=3, dtype=tf.uint8)
image4 = tf.io.decode_image(urllib.request.urlopen(urllib.request.Request(os.path.join(DATASET_ROOT, image4_path))).read(), channels=3, dtype=tf.uint8)
# Load the trained tensorflow saved model.
model = tf.saved_model.load(saved_model_path)
# The default inference function for tensorflow saved
# models is named `serving_default`.
inference_fn = model.signatures["serving_default"]
# The inference function is a one-arg callable, whose
# input argument is `image` - the image to predict on,
# and which returns a dictionary of outputs. The dictionary
# contains an item whose key is `prediction`, which is the
# predictions of the model.
image1_predictions = tf.nn.softmax(inference_fn(image=image1)['prediction'], axis=-1)[0]
image2_predictions = tf.nn.softmax(inference_fn(image=image2)['prediction'], axis=-1)[0]
image3_predictions = tf.nn.softmax(inference_fn(image=image3)['prediction'], axis=-1)[0]
image4_predictions = tf.nn.softmax(inference_fn(image=image4)['prediction'], axis=-1)[0]
# Threshold the predictions to determine the class prediction. For multi-class classification,
# we can take the maximum prediction (argmax)
image1_predictions = np.argmax(image1_predictions.numpy())
image2_predictions = np.argmax(image2_predictions.numpy())
image3_predictions = np.argmax(image3_predictions.numpy())
image4_predictions = np.argmax(image4_predictions.numpy())
f, axarr = plt.subplots(2, 2, figsize=(15,15))
_ = axarr[0, 0].imshow(image1)
_ = axarr[0, 0].title.set_text(label_map[str(image1_predictions)])
_ = axarr[0, 1].imshow(image2)
_ = axarr[0, 1].title.set_text(label_map[str(image2_predictions)])
_ = axarr[1, 0].imshow(image3)
_ = axarr[1, 0].title.set_text(label_map[str(image3_predictions)])
_ = axarr[1, 1].imshow(image4)
_ = axarr[1, 1].title.set_text(label_map[str(image4_predictions)])
Downloading Model: 100% (88170291 of 88170291) || Elapsed Time: 0:00:10 Time: 0:00:10
As you can see in the above predictions, the model correctly classifies all four images.
Using the Model for Inference¶
The Output Formats guide has more information about how to use the models output by Masterful. You made some predictions above in the View the Predictions section, so let’s recap the important parts below:
[15]:
# This image corresponds to the first image (upper left)
# in the `View the Predictions` section above.
IMAGE_URL = "https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/images/golfcourse17.png"
# Load the trained model so that you can test running inference.
MODEL_URL = "https://masterful-public.s3.us-west-1.amazonaws.com/datasets/ucmerced/trained_model.tar.gz"
model = None
with tempfile.TemporaryDirectory() as temp_directory:
# Download the pretrained model from S3.
saved_model_path = os.path.join(temp_directory, "saved_model.tar.gz")
_ = urllib.request.urlretrieve(MODEL_URL, saved_model_path, progress_bar_factory("Downloading Model: "))
# Extract the model weights from the tar file.
with tarfile.open(saved_model_path) as tar:
tar.extractall(temp_directory)
saved_model_path = os.path.join(temp_directory, "saved_model")
# Load the trained tensorflow saved model.
model = tf.saved_model.load(saved_model_path)
# Download the image to use for generating a prediction
# and convert it into a tensor.
http_response = requests.get(IMAGE_URL, stream=True)
image = tf.io.decode_image(http_response.raw.read())
image_height = image.shape[0]
image_width = image.shape[1]
print(f"Image Shape: height={image_height} width={image_width}.")
# The default inference function for tensorflow saved
# models is named `serving_default`.
inference_fn = model.signatures["serving_default"]
# The inference function is a one-arg callable, whose
# input argument is `image` - the image to predict on,
# and which returns a dictionary of outputs. The dictionary
# contains an item whose key is `prediction`, which is the
# predictions of the model.
predictions = inference_fn(image=image)['prediction']
# Categorical class predictions should be run through softmax for
# correct predictions.
predictions = tf.nn.softmax(predictions, axis=-1)
# Convert the predictions to NumPY for easier processing.
predictions = predictions.numpy()
# The predictions from a classification model
# have the shape [batch size, number of classes].
# The batch size will always be 1 for inference models,
# since they expect only one image for to predict on.
print(f"Predictions shape is {predictions.shape}")
class_prediction = np.argmax(predictions[0])
print(f"The model predicted class '{label_map[str(class_prediction)]}'.")
Downloading Model: 100% (88170291 of 88170291) || Elapsed Time: 0:00:09 Time: 0:00:09
Image Shape: height=256 width=256.
Predictions shape is (1, 21)
The model predicted class 'golfcourse'.
Next Steps¶
In the above sections, you have learned how to train an image classification model using the Masterful CLI, evaluate the performance of the model, and use the model to make predictions on your production data.
If you want to look at more computer vision tasks, you can try Detecting Digits. If you want to look at another, more involved Object Detection example, see Detecting Pedestrians in Street Level Imagery for more information. Otherwise, feel free to explore Segmenting Animals for other examples of using the Masterful CLI Trainer to solve challenging computer vision problems.