FAQ¶
Frequently Asked Questions¶
How does Masterful fit into my training workflow?¶
Masterful automates the training step of building models. Masterful is used after aggregating, cleansing, and labeling your training data. Model deployment and monitoring at inference time are outside the scope and downstream of Masterful.
The best place to start for most developers is to use the Masterful autofit API, which automatically analyzes your model and dataset to discover the optimal settings and policies that previously needed to be manually guessed and checked. This includes optimizer settings, early stopping, batch size, learning rate schedules, conventional augmentations, and applying synthetic and unlabeled data.
In practice, these are the args that are passed into the optimizer, the tf.keras.models.Model.compile, and tf.keras.models.Model.fit methods. Masterful handles the discovery of these args and calls compile and fit directly.
For users who need more control, Masterful provides the advanced API. Autofit itself is implemented using the advanced api, so users can pick and choose which techniques they want from the Masterful advanced API.
Where do I run Masterful?¶
Masterful runs wherever you currently train your model, which would typically be your local machine with a GPU or your cloud provider’s GPU-optimized instances.
What are the minimum system requirements to run Masterful?¶
Masterful has identical system requirements as Tensorflow 2.
Are there versions of Masterful other than Python?¶
No, Masterful only supports Python 3. Python 3 is currently the best language to use when training models with Tensorflow 2. Other languages that TF supports is better suited for deployment and inference. Quoting the TF 2.6 documentation:
TensorFlow has APIs available in several languages both for constructing and executing a TensorFlow graph. The Python API is at present the most complete and the easiest to use, but other language APIs may be easier to integrate into projects and may offer some performance advantages in graph execution. A word of caution: the APIs in languages other than Python are not yet covered by the API stability promises. [https://www.tensorflow.org/api_docs]
What machine learning use cases does Masterful support?¶
Masterful supports most computer vision tasks. The computer vision tasks that Masterful supports are below.
Classification, aka Multilabel Classification. In each image, there is one instance of an object, and the task is to identify the class of that object. This is the task behind the MNIST, CIFAR10, and Imagenet challenges.
Binary Classification. In each image, an attribute is either present or not present, and the task is to indicate whether the atttribute is present.
Multilabel Classification. In each image, zero, one, or more classes are present.
Detection. In each image, there are multiple classes of objects and multiple instances of each class of objects. The task is to draw a bounding box around each object and identify its class.
Localization. This task is very similar to detection, except localization does not attempt to also classify the object of interest.
Semantic Segmentation. In each image, there are multiple classes of objects. The task is to identify, for each pixel, which class of object it represents. Semantic segmentation cannot help distinguish between two different instances - if one pixel is from one car, and another pixel is from another car, the semantic segmentation task will only identify that a pixel is aligned with the class car, not any individual instance of a car.
Instance Segmentation: This extends semantic segmentation to also identify specific instances of objects. Note that object detection also identifies instances, but only predicts bounding boxes rather than pixel masks.
Keypoint Detection: This is similar to detection, except instead of bounding boxes, individual pixel locations are identified for features like the corner of a mouth or the nose of an airplane.
Note that Masterful does not currently support audio, video, voxel, sensor fusion, or NLP tasks.
What machine learning frameworks does Masterful support?¶
Masterful is built to work with Tensoflow 2. Support for PyTorch is coming soon.
Do I use my own model, or do you pick an architecture for me?¶
Masterful is AutoML for data and training. The developer still has full control and responsibility for picking the correct architecture.
In general, picking architectures is not a purely technical task. In general, the bigger and more modern the architecture, the better the model accuracy. But bigger also means higher inference costs and slower response time. So business judgement must be applied to trade off accuracy versus speed and cost.
How can I see what Masterful is doing to my data and model?¶
Masterful provides a powerful visual frontend to help developers understand what is happening inside the Masterful AutoML Platform. See the guide to the front-end for more details.
What data types does Masterful support?¶
Masterful has been tested with work with a variety of image data, including:
geospatial
manufacturing
security
documents
medical
transportation
How long does Masterful’s analyze phase take to run?¶
Masterful’s analyze phase (used in autofit) typically takes around 2x times longer to run than a traditional call to tf.keras.Model.fit. The analyze phase requires multiple calls to tf.keras.Model.fit() to understand the model’s behavior. However, Masterful eliminates the need to manually run fit with various hyperparameters and policies, and finds better ways to use the available hardware, so Masterful reduces both the total GPU training time and developer time to get to a performant model.
When do I use autofit, and when should I use advanced functions on their own?¶
Users should start with the autofit API. It can often eliminate the need for tuning and maintaining hyperparameters for regularization and optimization. Users who already have a highly tuned custom techique as part of their training cycle may want to use the advanced API instead, which provides finer grained control over the training process and access to advanced training features.
In addition, some functionality is only available via the advanced api, such as ensembling, distillation, and unsupervised pretraining.
Does Masterful slow down my training loop?¶
Yes and no. Masterful’s autofit does require time to analyze the model’s behavior. And as with any technique that improves accuracy, more epochs will be required. However, Masterful works with just one call, whereas most training requires multiple attempts with manual grid search of hyperparameters and policies. So Masterful often reduces total cloud spend, developer time, and time to market.
How much labeled data do I need to get started?¶
Masterful is helpful anytime a developer has a model that is successfully trainable to a reasonable level of accuracy and can benefit from further reductions in error. In general, we find that a labeled dataset in the tens of thousands is the minimum to achieve this baseline starting point.
How much unlabeled data do I need to get started?¶
Unlabeled data is not required by Masterful, but Masterful can apply unlableed data to help improve your model accuracy. When using unlabeled data, there should be at least as much unlabeled data as labeled data, and ideally 10x more unlabeled data.
When should I use ensembling?¶
Ensembling is an advanced technique that helps regularize training of larger models. The optimal use case to try ensembling is when larger models do not seem to improve accuracy. In that case, use the ensembling api to train several smaller models separately, then ensemble them together.
When should I distill a model?¶
Distillation is an advanced technique to get even more performance out of training. When other approaches appear to have run their course, train a larger model that is otherwise too large for your inference requirements, then distill it to a model size that does meet your inference requirements. This process will often yield a more accurate but reasonably sized model.
Does my model need a minimum level of performance in order to see benefits from Masterful?¶
Yes, your model architecture, training data, and training approach should already be yielding a model with a baseline level of performance. Until you have discovered an appropriate model architecture, you have a minimum of training data, and you have worked out the bugs in your training loop, your engineering cycles are best spent achieving that basic level of performance before attempting to further improve your model with Masterful.
How does Masterful relate to CUDA, Tensorflow, PyTorch, Keras, and AutoML?¶
Masterful is built on top of Tensorflow (PyTorch coming soon) to reduce the complexity of training.
CUDA is the lowest level API, providing scientific computing access to Nvidia GPUs. The use of GPUs is one of the major breakthroughs that enabled the commercialization of deep learning.
Tensorflow and PyTorch are implemented on top of CUDA to provide a more abstract, powerful, and easier-to-use interface on top of CUDA. For example, whereas CUDA provides at best 2 dimensional, 4x4 matrix multiplication, PyTorch and Tensorflow allow users to easily multiply arbitrarily large matrices. It is PyTorch/Tensorflow’s job to break up that operation into individual CUDA ops.
Keras and PyTorch Lightning / Torchvision are yet additional layers of abstraction that again provide more power to developers. Specifically, Keras is focused on building deep neural networks, whereas Tensorflow is general enough to handle arbitrary scientific computing. Keras was integrated into Tensorflow2 as the preferred API for building and training deep neural networks.
While Keras handles much of the complexity of Tensorflow, it is still focused on model architectures, which leads to a complex and unintuitive API for training. This is why, for example, there is a separate compile and fit function.
Masterful builds on top of Keras to create an API and set of services that is more logically oriented around the end-to-end process for creating models, processing data, and training.
Many layers of abstraction above even Masterful is the AutoML provided by cloud providers. In these APIs, the developer passes data to the cloud provider and gets a trained model. As with any high-level api, there is a loss of control, visibility, and explainability. For those reasons, AutoML is generally limited to developers just getting started. Once they need to develop and improve their own unique models, they have to reach down several layers of abstraction and rebuild their models using much more primitive APIs. In a sense, Masterful helps fill in a gap between the abstract but blackbox AutoML services from cloud providers, and the low level Tensorflow API that approaches assembly programming in complexity and difficulty.