Overview
Revolutionizing Cellar Management
CellarEye, Inc. is on a mission to revolutionize the experience of managing private and professional wine collections. Leveraging state-of-the-art Computer Vision (CV) and Artificial Intelligence (AI) technologies, CellarEye offers a seamless management system that automatically tracks each wine bottle in a cellar, storing both the brand and location into inventory tools without manual entries into a spreadsheet or smartphone app. Ultimately, CellarEye aims to provide better and more widespread understanding of wine, and make wine more accessible to more people.
The Problem
Detecting Data Labeling Errors
To realize their vision, the team at CellarEye had to develop a reliable object detection model to recognize and track wine bottles as the bottles registered to and removed from the inventory, and to maintain an accurate inventory of which, how many, and where each bottle is in the cellar. “The cellar environment is complex with thousands of wine bottles, and we have tons of edge cases,” said Mehdi Mohseni, Co-Founder and CEO of CellarEye. “We had really bad or inconsistent annotations at the beginning that made achieving 80%+ accuracy a challenge. We needed a better way to detect problems with our data, understand our model failures, and let our ML team collaborate with our annotation team to catch labeling mistakes faster,” concluded Mohseni.
The Solution
Enhancing Datasets with Models-in-the-Loop
The CellarEye team looked to Scale Nucleus to identify where their models were underperforming and where labels were likely to be incorrect. With Nucleus, CellarEye’s team can query their labeled and unlabeled data on custom metadata, visualize model predictions alongside ground truth annotations, sort results by error metrics like Intersection Over Union (IoU), look at high-confidence false positive predictions, and fix and export corrected labels using Nucleus’s built-in annotation editor.
The Result
Driving Model Improvements and Efficiency on ML Productivity
By using Nucleus, the team at CellarEye empowered their annotation team to rapidly identify and fix bad annotations. “The feature that really helped us improve our object detection model was sorting individual model predictions by worst IoU,” said Mohseni. “This view enabled us to quickly find some of the worst cases and also surfaced similar issues or scenarios. The fact that we could then also add, remove, or fix the annotations in one place made it really easy for us to improve our model quickly,” said Mohseni. With Nucleus, the team at CellarEye saw significant improvement in average precision. “There was one particular class where our model’s average precision was 76%. We improved average precision on that class to 87%+ after just one round of data cleaning with Nucleus,” added Mohseni. Furthermore, by using Nucleus, CellarEye’s internal team experienced 5x efficiency on ML productivity by saving time on post-processing, verification, and heuristics as well as delivering a 5x savings on annotation reviews. “Before Nucleus, I tried to drive model performance by augmenting and just adding more data. While more data is better than less data, if the annotations aren’t good, simply adding more data doesn't lead to higher accuracy. By helping us discover our annotation issues and fix them, Nucleus was able to drive significant improvements,” concluded Mohseni.