Installation Tutorial

Prerequisites

Supported Linux Distributions

Masterful is supported on most Linux distributions that are supported by Tensorflow and PyTorch.

Python

Masterful supports Python 3.6+, which is installed by default in most Linux distributions.

Pip

Masterful is currently distributed on the Python Package Index here. While Python 3.x is installed by default on most Linux distributions, pip is not installed by default, so make sure to install it if it is not already installed.

sudo apt install python3-pip

Tensorflow 2

The Masterful package does not have an installation dependency on Tensorflow, so as not to overwrite the existing version of TF on the machine or in the environment. Therefore Tensorflow, Tensorflow Datasets, and Tensorflow Addons must be installed separately from Masterful. Note that the tensorflow-datasets and tensorflow-addons packages are built for specific versions of the tensorflow package, so make sure to install the appropriate version of each according to their compatibility matrices.

Masterful requires Tensorflow versions 2.4.1 or greater.

pip install --upgrade pip
pip install tensorflow
pip install tensorflow-datasets
pip install tensorflow-addons

Masterful Installation

Install Masterful with Python’s pip package manager.

pip install --upgrade pip
pip install masterful

Verify your installation by running:

python -c "import masterful; print(masterful.__version__)"

If this returns a version number like 0.4, your installation was successful.

Next Steps

Now it’s time to get started! Go to the quickstart tutorial apply Masterful to a simple convnet on CIFAR-10.

Troubleshooting

For the quickest access to help, support, and advice, please join the masterful-community slack.

I see a warning from Tensorflow Addons that my version of tensorflow is not supported

You may see the following warning if there is a version mismatch between Tensorflow and Tensorflow Addons. Typically this happens if you install a specific version of Tensorflow, but install the latest version of Tensorflow Addons without looking at the compatibility matrix. If this happens, you will see a warning in the console that looks like this:

... site-packages/tensorflow_addons/utils/ensure_tf_install.py:53: UserWarning: Tensorflow Addons supports using Python ops for all Tensorflow versions above or equal to 2.5.0 and strictly below 2.8.0 (nightly versions are not supported).
 The versions of TensorFlow you are currently using is 2.4.1 and is not supported.
Some things might work, some things might not.
If you were to encounter a bug, do not file an issue.
If you want to make sure you're using a tested and supported configuration, either change the TensorFlow version or the TensorFlow Addons's version.
You can find the compatibility matrix in TensorFlow Addon's readme:
https://github.com/tensorflow/addons
  warnings.warn(

In order to fix this, you can uninstall Tensorflow Addons, and re-install the specific version of Tensorflow Addons that is compatible with your version of Tensorflow. The latest compatibility matrix can be found at https://github.com/tensorflow/addons.

For convenience, here is the compatibility matrix for Tensorflow Addons:

TensorFlow Addons	TensorFlow	Python
tfa-nightly	2.5, 2.6, 2.7	3.7, 3.8, 3.9
tensorflow-addons-0.15.0	2.5, 2.6, 2.7	3.7, 3.8, 3.9
tensorflow-addons-0.14.0	2.4, 2.5, 2.6	3.6, 3.7, 3.8, 3.9
tensorflow-addons-0.13.0	2.3, 2.4, 2.5	3.6, 3.7, 3.8, 3.9
tensorflow-addons-0.12.1	2.3, 2.4	3.6, 3.7, 3.8
tensorflow-addons-0.11.2	2.2, 2.3	3.5, 3.6, 3.7, 3.8
tensorflow-addons-0.10.0	2.2	3.5, 3.6, 3.7, 3.8
tensorflow-addons-0.9.1	2.1, 2.2	3.5, 3.6, 3.7
tensorflow-addons-0.8.3	2.1	3.5, 3.6, 3.7
tensorflow-addons-0.7.1	2.1	2.7, 3.5, 3.6, 3.7
tensorflow-addons-0.6.0	2.0	2.7, 3.5, 3.6, 3.7

Registering Masterful raises an AlreadyExistsError when using TensorFlow 2.6

Ensure the Keras version matches the Tensorflow version. Starting in Tensorflow 2.6, Keras was moved out of Tensorflow and back into a separate package, leaving a stale copy of Keras packages under tensorflow/python/keras, which have been removed in Tensorflow 2.7. Because of this TensorFlow 2.6 is known to erroneously use Keras 2.7 as a dependency. Although this bug is not related to Masterful, registering Masterful will expose the incompatibility during their import. Downgrading Keras to 2.6 resolves this issue.

I see too much Tensorflow logging cluttering up my console

By default, Tensorflow sets the C++ logging level to INFO, so you will see lots of console output like:

2022-01-17 11:16:11.651649: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-01-17 11:16:11.651685: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublas.so.11
2022-01-17 11:16:11.651710: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcublasLt.so.11
2022-01-17 11:16:11.651734: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcufft.so.10
2022-01-17 11:16:11.651757: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcurand.so.10
2022-01-17 11:16:11.651781: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcusolver.so.10
20

You can set the environment variable to 1 in your shell, before running your code, to eliminate these INFO logs.

export TF_CPP_MIN_LOG_LEVEL=1

If setting directly in your python script, make sure to set this variable before importing Tensorflow. Otherwise, it won’t be picked up.

# Must set BEFORE importing Tensorflow (including dependencies that might also
# import Tensorflow).
import os
os.environ['TF_CPP_MIN_LOG_LEVEL'] = '1'

# Now we can import Tensorflow for the first time.
import tensorflow

I receive an AttributeError trying to access masterful.XXX

If you try to access any Masterful APIs before registering with your account ID and authorization key, you will receive an AttributeError similar to the following:

Traceback (most recent call last):
  File "test.py", line 38, in <module>
    model_spec, data_spec = masterful.spec.create_model_and_data_specs(
AttributeError: module 'masterful' has no attribute 'spec'

This is expected, and can be fixed by registering before accessing any Masterful APIs.

masterful = masterful.register()

# Now you access all Masterful API calls.

I see an error related to IProgress when running Masterful within a notebook

When Running masterful within a jupyter notebook you might encounter with the following error:

...
~\anaconda3\envs\masterfulAI\lib\site-packages\tqdm\notebook.py in status_printer(_, total, desc, ncols)
    113         # Prepare IPython progress bar
    114         if IProgress is None:  # #187 #451 #558 #872
--> 115             raise ImportError(
    116                 "IProgress not found. Please update jupyter and ipywidgets."
    117                 " See https://ipywidgets.readthedocs.io/en/stable"

ImportError: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html

Masterful uses tqdm library in order to print out progress of the trainig loop. Nevertheless for some older versions of jupyter and ipywidgets it can generate this conflict. Thus, If you happen to run into this issue, please, just update jupyter and ipywidgetslibrary to the current version.

You can see more details on this related thread: https://stackoverflow.com/questions/67998191/importerror-iprogress-not-found-please-update-jupyter-and-ipywidgets-although

I see an error related with Opencv (cv2) library when using masterful API

You could come across with the following error:

...
~/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/cv2/__init__.py in <module>
      7 
      8 from .cv2 import *
----> 9 from .cv2 import _registerMatType
     10 from . import mat_wrapper
     11 from . import gapi

ImportError: cannot import name '_registerMatType' from 'cv2.cv2' (/home/ubuntu/anaconda3/envs/tensorflow2_p38/lib/python3.8/site-packages/cv2/cv2.cpython-38-x86_64-linux-gnu.so)

Materful uses opencv-python-headless as a dependency, however this error can appear because the version of openv-python-headless must match the version of opencv-python. Thus, you can downgrade opencv-python-headless to version 4.1.2.30 and the problem will be solved. Just uninstall running pip uninstall opencv-python-headless and install it again using the command: pip install opencv-python-headless==4.1.2.30.

You can see more information on this issue here

I receive a NotImplementedError about symbolic Tensors

If you see an error similar to the one below in the stack trace:

NotImplementedError: Cannot convert a symbolic Tensor (model/masterful_rotate/rotate/_angles_to_projective_transforms/strided_slice:0) to a numpy array. This error may indicate that you're trying to pass a Tensor to a NumPy call, which is not supported

then your numpy version is incompatible with your Tensorflow version. Typically this means your numpy version is too high, and you should try downgrading your numpy version to a previous version. For example, with Tensorflow 2.4.1, numpy==1.19.5 is known to work.