Lambda Team

Editor’s note: This post originally appeared on <a href="https://lambdalabs.com/" rel="noopener noreferrer" target="_blank">lambdalabs.com</a>. You can find a link to it <a href="https://lambdalabs.com/blog/lambda-scale-nucleus/" rel="noopener noreferrer" target="_blank">here</a>. Training accurate models typically requires a deep understanding of your training data. Through visualization, it’s important to both identify potential flaws in your ground truth labels, and to look for insights that will guide model development. Thus, visualizing your training data is the foundation of every successful deep learning project. Nucleus is a tool from Scale that allows you to visualize and debug your training data. It provides an intuitive web interface to visualize and explore a wealth of different types of data and their labels. It also gives you the tools to explore and understand the relationship between your data and the model predictions that you generate. In this post, we'll show you how you can get started with Nucleus by training a pedestrian detection model. To do this, we will use Nucleus to visualize an object detection dataset containing images of common street scenes, and compare the ground truth to the predictions from a trained neural network. This loop of inspecting your training data and looking for failure modes in a network's predictions is a crucial process in refining both your training dataset and your neural network.<h2>Getting Started</h2> The example model in this post was trained on a <a href="https://lambdalabs.com/gpu-workstations/vector" rel="noopener noreferrer" target="_blank">Lambda Vector workstation</a> running <a href="https://lambdalabs.com/lambda-stack-deep-learning-software" rel="noopener noreferrer" target="_blank">Lambda Stack</a>. If you want to follow along, please see the <a href="https://colab.research.google.com/drive/1Cu2Xw0kPwbMMSdqUsSgmMMTzQ5b7usDZ" rel="noopener noreferrer" target="_blank">example notebook here</a>. To start, we’ll download the publicly available <a href="https://www.cis.upenn.edu/~jshi/ped_html/" rel="noopener noreferrer" target="_blank">Penn-Fudan Dataset</a>, which consists of photographs of pedestrian data with labeled bounding boxes. To load these into Nucleus, we need to do two things: first, sign up for a free Scale account and corresponding API key, and second, install the Nucleus client on our Lambda workstation from a command prompt or shell:<pre class="ql-syntax" spellcheck="false">pip install scale-nucleus
</pre> <h2>Training Data</h2> Once we've installed the Nucleus client and downloaded the dataset, we can upload the images and ground truth labels to a new Dataset we created in Nucleus. (For the full details of how to do this, see <a href="https://colab.research.google.com/drive/1Cu2Xw0kPwbMMSdqUsSgmMMTzQ5b7usDZ#scrollTo=jC-728nqvD0L" rel="noopener noreferrer" target="_blank">this section</a> of the notebook.) Now that we’ve uploaded the data, we can visualize the ground truth labels in the Nucleus web dashboard without writing any code.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/edf2a44b-d1e4-4dcc-d6f4-70b3587ac100/public" alt="PennFudan pedestrian dataset with bounding boxes in Nucleus"> <h2>Model Predictions</h2> Now that we've "<a href="https://karpathy.github.io/2019/04/25/recipe/#1-become-one-with-the-data" rel="noopener noreferrer" target="_blank">become one with our data</a>," we can start thinking about training a neural network for object detection. For this example we fine-tuned a <a href="https://arxiv.org/abs/1506.01497" rel="noopener noreferrer" target="_blank">Faster R-CNN</a> model pre-trained on the <a href="https://cocodataset.org/#home" rel="noopener noreferrer" target="_blank">COCO dataset</a>. Training it for 10 epochs only takes a few minutes on an <a href="https://lambdalabs.com/blog/nvidia-rtx-a6000-benchmarks/" rel="noopener noreferrer" target="_blank">NVIDIA RTX A6000</a> in a Lambda Vector Workstation. After training is finished we can see that our model has reached an Average Precision (AP@IoU=0.50:0.95) of 0.83. This sounds OK. It's not too far from 1, but what does it mean? Do we have a good pedestrian detector yet? The AP metric of object detection is notoriously hard to get your head around, and there are endless <a href="https://jonathan-hui.medium.com/map-mean-average-precision-for-object-detection-45c121a31173" rel="noopener noreferrer" target="_blank">blog posts</a> that aim to provide an intuition for it, but you’ll ultimately need to look at our results to see how we're doing. Nucleus allows us to upload our model predictions and compare them to the ground truth. Here we can see that blue boxes predicted by our model match pretty well with the green ground truths from the dataset: <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/1dc58fa6-c049-4610-324b-aab8d114a600/public" alt="Image Grid, Ground Truth versus Fast-RCNN Bounding Boxes"> <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/005b8ff2-c8e7-44a0-42bc-615be98ba000/public" alt="Nucleus Annotation, Classification, and IoU Detail"> <h2>What to Try Next?</h2> Nucleus also gives some nice tools to help us figure out what else we should try to improve our accuracy results. Here we visualize the ground truth labels with the model predictions so that we can get a feel for where our model is going off the rails. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/08ad13c7-2552-45ce-0650-cc8e4115a200/public" alt="Correcting the Ground Truth Bounding Box in Nucleus"> In many cases, it looks like our model is already doing a pretty good job, and it's the ground truth labels that have actually missed some pedestrians! For anyone who has worked on a practical machine learning project, this is probably a familiar story—finding and fixing errors and problems in your dataset is just as much part of deep learning as designing and training models. In this case, we’ll need to go back to our data labels and fix the ground truth errors in order to get a more accurate evaluation of our model. This is also something we can do directly in Nucleus with its convenient annotation tools. <h2>Conclusion</h2> Nucleus is a convenient way to visualize and organize your training data for a machine learning project, as we showed in our object detection example. Most importantly, it makes it easy to inspect and track the predictions from your trained models, and helps you establish the virtuous cycle of training networks, evaluating predictions, identifying failure cases, and improving both data and models: all of these steps prove to be crucial in developing an effective deep learning project. To learn more about Lambda workstations, <a href="https://lambdalabs.com/" rel="noopener noreferrer" target="_blank">check out Lambda’s website</a>, or to get started managing and improving your dataset, <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">sign up for Scale Nucleus here</a>. If you want to try this workflow out on your own <a href="https://lambdalabs.com/gpu-workstations/vector" rel="noopener noreferrer" target="_blank">Lambda workstation</a> or Lambda <a href="https://lambdalabs.com/service/gpu-cloud" rel="noopener noreferrer" target="_blank">Cloud instance</a>, you can explore the <a href="https://colab.research.google.com/drive/1Cu2Xw0kPwbMMSdqUsSgmMMTzQ5b7usDZ" rel="noopener noreferrer" target="_blank">example notebook here</a>.

Scale partners with Lambda to help businesses train better models on better data.

Lambda Team

Lambda and Scale Nucleus: Empowering Your Model Training with Better Data