Sasha Harrison

Q: Tell us a little bit about yourself and your role at Scale.A: My name is Sasha, and I'm a Machine Learning Engineer here at Scale. I've been at Scale for about a year and eight months. Previously, I was doing a master's program in computer science at Stanford, where my concentration was in artificial intelligence.Within my career at Scale, I've worn a lot of different hats. I first started working on the Nucleus team, in a backend software engineering role when the team was very young. It was&nbsp;Scale’s newest product at that point, and I helped build out a lot of Python SDK interfaces, and the backend schema for a lot of new features, including Autotag, which is the edge case detection feature.&nbsp;I later transitioned to building out other new products, including Scale Validate, which is the model scenario testing and validation product area.Most recently, I've recently transitioned to our Machine Learning team full time, where I'm on the Content Understanding team. I'm basically helping to support automation in labeling projects associated with content categorization and curation type labeling projects.Q: Could you give us a little bit of detail about your educational background, and how that led you to join Scale?A: My undergrad degree was in applied mathematics. Entering college, I didn't think of myself as a coding person at all. But during the years I was at Stanford, computer science was something that a lot of the undergraduate body talked about, and the entry level computer science classes were really fun. During my junior and senior years in college, I started taking some entry level machine learning classes. The first thing I ever learned about was logistic regression, and how you assign importance or attribution to different input features&nbsp;to answer social science questions. Like, is it age, or education, that contributes more to this downstream factor?I thought that was really interesting, and really wanted to double down on that. So, that's what led me to stay for a fifth year master's, and specialize in AI. I worked as a research assistant briefly, in a sociology lab, where the objective of the sociology lab was to develop a dataset, documenting urban change. My job as a research assistant was to label data on AWS, Mechanical Turk. We had a lot of difficulty controlling quality, and getting annotations that were detailed enough to model the downstream tasks of trash detection and vandalism detection.&nbsp;So, I knew that data collection was a hair on fire problem for a lot of application-driven teams. When I interviewed at Scale, I felt like the mission, and the problems that the company was trying to solve, were very urgent and impactful.Q: What attracted you to working at a hyper growth startup?A: When I was in college, I think a lot of my peers were talking about going to work at a big tech company; that was the prestigious choice. It was safe; it was a known outcome. At first, I was really focused on that, and I did a lot of internships at big companies.&nbsp;And then I got this career advice my sophomore junior year of college, from a woman at Grace Hopper, and her advice was get on a rocket ship. Basically, the idea was, if you could just put yourself in a place where there were interesting problems to solve, some of those were going to come your way.I really took that to heart, and my impression of myself started to shift toward somebody who really wanted to tackle big challenges and wear a lot of hats in their job. So, I joined Scale under the hypothesis that I was going to have an opportunity to wear a lot more hats and serve a lot of different functions at a very lean company.&nbsp;As a result of having the opportunity to take on difficult challenges, I've grown my skills at a very quick rate. And I think I gained seniority and experience, at a rate that far exceeds any previous work opportunity I had in the past.&nbsp;Q: What's your advice to students, or other people, who are interested in careers in machine learning?A: First of all, I think it's an excellent career to be in, in that there are a lot of opportunities in the space right now. And the number of positions for machine learning engineers and machine learning-adjacent engineers, is growing really rapidly. I think there are also a lot of exciting developments, both on the research and applications side.One thing that fascinates me about machine learning is how closely it can relate to real-world problems. If you’re interested in sociology, there's probably a place for that type of modeling for you. Or, if you're interested in online advertising, there are a lot of different application areas to concentrate on.The other thing that I would recommend is really reading a lot. There are a lot of great resources online for practicing building deep learning models, whether it be through Kaggle challenges, or what have you. Also, a lot of research papers and academic work are publicly available.One thing that I felt intimidated about when I started in the field was, how much there was to know, and how little of it I did know, as somebody that was early in their career. And I think making a habit of reading papers, and reading about approaches or problems that I didn't work on directly on a day-to-day basis, really increased my confidence.Q: Has working at Scale shifted your perspective on AI or ML applications at all?A: Definitely. When I first started at Scale, what I was really interested in was machine learning modeling work, and the idea of experimentation–building different neural network architectures, and being very scientific about, whether A works better than B.&nbsp;One thing that's become really apparent, both working at Scale, and seeing the types of problems that our engineering teams face, but also working with Scale customers, is the importance of systems design within large scale machine learning systems. I’ve seen how difficult the problems of scalability, observability and overall reliability of machine learning systems are.&nbsp;I really think that that's where a lot of innovation in the next 10 years is going to come from; not necessarily the modeling techniques themselves, but more, how you might integrate those modeling techniques into a larger scale operational framework.Q: What do you like to do outside of work?A:&nbsp;I've recently gotten into cycling. And so, I spend a lot of my time outside, whether it's running or riding my bike in the Bay Area. I also love wheel pottery. So on the weekends, you'll find me at the pottery studio.

Sasha Harrison details her experience as a Machine Learning Engineer.

Life at Scale

Today we&rsquo;re thrilled to share that Nucleus now supports 3D Sensor Fusion data, including LiDAR point clouds.
<a href="https://scale.com/nucleus" target="_blank" rel="noopener noreferrer">Scale Nucleus</a> is the mission control for your data, providing advanced tooling for curating, visualizing and exploring machine learning datasets and debugging ML models. We recently launched an all-new Nucleus for dataset management, complete with privacy features and updated Autotag capabilities, all to help teams build better ML models by building better datasets. Now, Nucleus can help teams manage 3D datasets to develop highly accurate perception models for a range of use cases, including autonomous vehicles, robotics, and augmented/virtual reality (AR and VR).
Scale Nucleus helps you visualize and debug 3D Sensor Fusion data by making it easy to:
<ul>
<li>Construct Scenes that contain a sequence of LiDAR point clouds and their corresponding camera images over time</li>
<li>Visualize, query and share sensor fusion data with colleagues</li>
<li>Integrate Sensor Fusion data with your 3D model predictions to uncover and identify:</li>
<li>Discrepancies between 3D object detection model outputs and ground truth</li>
<li>False positive, common class confusions, and cuboid predictions that have low Intersection over Union (IoU) with ground truth</li>
<li class="ql-indent-1">Model performance and improvements over time</li>
</ul>
The Nucleus API makes it easy to import your data, construct scenes, and add model predictions. You can also import data from an existing Scale project directly into Nucleus.
<h2>A Primer: Nucleus Scenes</h2>
Scenes are the building blocks of 3D datasets in Nucleus and are designed with point clouds as the primary unit of data. More formally, a Scene is a sequence of Frames over time, where each Frame contains exactly one point cloud and an optional set of camera images. Here we can see a LiDAR point cloud above two paired images (the full dataset includes more than just front and back cameras, but a full suite of sensor data), which helps explain why we designed Nucleus Scenes to encapsulate data across both time and sensor modalities:
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/00157f16-6ed9-4ffc-45cc-60b676806e00/public" alt="Structure Diagram of Nucleus 3D Scenes">One useful way to interpret a LiDAR Scene is as a table of sensor readings. A row in the table corresponds to a sequence of readings from a particular sensor, while a column in the table represents a collection of sensor data from the same timestamp (we call this a &ldquo;frame&rdquo; of the scene).
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6ea98ea3-b4d4-40eb-f36f-3562514cab00/public" alt="Complex Nucleus 3D Scene on SF's Embarcadero, Quarter Angle View">
On their own, point clouds can be difficult to interpret, due to lack of motion cues or associated camera data. Nucleus Scenes make sets of 3D and 2D data easier to interpret for the following reasons: first, any number of 2D camera frames can be attached to each 3D LiDAR frame; and second, Scenes link frames from the same timespan so that you can examine all frames in a linear fashion.
Let&rsquo;s now view the same Scene in Nucleus and see if we can identify the mystery object from above:
<iframe class="ql-video" src="https://scaleai.wistia.com/embed/medias/8ikm69knof" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
After uploading predicted cuboids from an ML model for your Scene, Nucleus automatically computes performance metrics under the hood. You can sort predicted cuboids by IoU or filter by false positives to quickly narrow in on your model&rsquo;s failure modes. (These features have been available for 2D image analysis for some time, but today we are announcing their ability for 3D data, too.)
To summarize, by creating and uploading 3D Scenes to Nucleus, you can easily visualize LiDAR point clouds and images side-by-side, quality-check your ground truth annotations, and search your data based on metadata or even ML-produced attributes. Lastly, you can use Nucleus to measure key prediction performance metrics, such as IoU and false positives.
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/6908e9df-249d-47df-4db5-a39827556c00/public" alt="Birds' Eye View of SF Embarcadero as Seen through LiDAR and Multiple Image Sensors in Nucleus Scene">
<h2>Getting Started with 3D Data in Nucleus</h2>
Visualizing, querying, and measuring the quality of sensor fusion datasets is a challenge facing even the most sophisticated ML teams. With the newly launched support of multimodal scenes, Nucleus makes it easy to navigate through massive datasets, and quickly narrow down on the most difficult examples. Unleash the full potential of your datasets by leveraging Nucleus for debugging, search, and quality related workflows.
The complete code for this tutorial is available in <a href="https://colab.research.google.com/github/brwillia/nuc-3d/blob/main/nucleus_3d_demo.ipynb#scrollTo=rr88VqEHaL2_" target="_blank" rel="noopener noreferrer">this Colab notebook</a>. If you&rsquo;d like to follow along step by step and create a personal dataset on Nucleus, you will first need to make a <a href="http://scale.com/signup" target="_blank" rel="noopener noreferrer">free Scale account</a> and obtain a test API key (which you can <a href="https://dashboard.scale.com/settings/profile" target="_blank" rel="noopener noreferrer">find here</a> if you navigate to your user icon, and then click on API keys). You can then copy the notebook and add your personal key to upload your data and try out Nucleus for yourself. At the end, you&rsquo;ll also find a section that shows you how to format and upload your own LiDAR data for curation and management in Nucleus if you&rsquo;re ready to manage your own 3D data in Nucleus.

Learn how to ingest and curate 3D LiDAR point cloud data in Nucleus.

Nucleus now supports 3D Scenes.

Managing 3D Sensor Fusion Data with Scenes in Scale Nucleus

Many AI systems rely on supervised learning methods in which neural networks train on labeled data. The challenge with supervised methods is getting models to perform well on examples not adequately represented in the training dataset. Typically, as the frequency of a particular category decreases, so does average model performance on this category (see Figure 1). It is often difficult and costly to achieve strong performance on the rare edge cases that make up the long tail of a data distribution. In this blog post, we’ll take a deeper look at how sophisticated data curation tools can help machine learning teams target their experiments toward taming the long tail.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/fa8816c4-0552-489a-3131-4577f439ab00/public" alt="Figure 1: Long Tail Distribution"> Figure 1: This type of distribution, in which there are a few common categories followed by many rare categories, is called a long tail distribution. In the majority of deep learning applications, datasets collected in the real world tend to have this long-tail shape. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/703dc6dd-ba3f-4caf-380b-d77dbffb8800/public" alt="Figure 2: Ubiquity of Long Tail Distributions"> Figure 2: Long tail distributions occur frequently in the real world. For example, the frequency of words in English writing follows a long-tail distribution. The ugly truth is that almost all AI problems worth solving are made difficult by the challenge of a long-tail. Imagine you’re an engineer at an autonomous vehicle company, working on an object detection model trained on image data captured from vehicles on the road. This real-world data provides a textbook long-tail distribution: There are many thousands of images of cars on the highway, but very few depicting bicycles at night (see Figure 3). In a preliminary experiment, the subsequent model performs badly when localizing cyclists at night (see Figure 4). Neglecting this rare scenario is likely to result in high-severity errors during testing, but it’s critical for the algorithm to detect cyclists properly in order to prevent collisions. What’s the best method to address this model failure in a targeted way? There are many approaches to improving performance, including experimenting with new architectures or additional hyperparameter tuning. But if the goal is to improve model performance on this specific edge case, the most targeted approach is to add more examples of cyclists to the training dataset. The problem is that unlabeled data is inherently difficult to search; sampling randomly, one is unlikely to find more cyclists at night, and it’s too expensive to simply label all collected data. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2e7bb4d6-f3cb-4206-5f23-a30613bac500/public" alt="Figure 3: Berkeley Deep Drive Class Distribution"> Figure 3: In the Berkeley Deep Drive dataset, the category “rider” occurs infrequently compared to the category “car.” It’s no surprise that when we view individual examples of riders, our model trained on Berkeley Deep Drive does not do a good job of localizing these objects. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/01a950ec-c9ea-4901-65d7-befbc8d51900/public" alt="Figure 4: Examples of poor localization for the class “rider”"> Figure 4: Object detection predictions from an EfficientDet D7 architecture trained on Berkeley Deep Drive. Examples of poor localization for the class “rider”, a long tail class in the training dataset. To improve performance on the long tail of edge cases, machine learning practitioners can get caught up in an endless cycle of collecting more training data, subsampling, and retraining the model. While this type of experimentation is essential to machine learning development, it is incredibly costly with respect to time, compute, and data labeling. The cost of improving performance on the long tail often threatens the economics of AI products. <a href="https://a16z.com/2020/08/12/taming-the-tail-adventures-in-improving-ai-economics/" rel="noopener noreferrer" target="_blank">This blog post</a> from Andreesen Horowitz details how AI businesses seldom have the attractive economic properties of traditional software businesses. Where software products benefit from economies of scale, AI businesses experience the opposite; as model performance improves over time, the marginal cost of improvement increases exponentially. It may require ten times the initial training data to yield any significant model improvement. Given that expensive model retraining is unavoidable, machine learning teams should focus on making iterative experimentation as efficient as possible. One way to focus experiments on improving the long tail is to use model failures to identify gaps in the training dataset and then source additional data to fill those gaps. Think of this approach to machine learning experimentation as “mining the long tail.” With each experiment, identify a failure case, find more examples of this rare scenario, add new examples to the training dataset, and repeat. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2b271a31-9926-4679-4e58-29e90178b500/public" alt="Figure 5: ML Development Lifecycle"> Figure 5: The ML development lifecycle consists of collecting, generating, annotating, managing or curating, training, and evaluating. The data curation or management step has the biggest impact on an experiment’s success, yet is most overlooked in ML discourse. It’s difficult to target experiments in the most under-represented scenarios without sophisticated tools for dataset curation. There are three operationally critical capabilities for most AI teams: <ol><li>Identify long-tail scenarios during model evaluation</li><li>Reliably and repeatedly source similar examples from unlabeled data</li><li>Add these examples to a labeled training dataset.</li></ol> Some of the most effective teams we work with at Scale face the challenge of the long tail head-on by building their tools, operations, and workflows proactively around an iterative process of data curation. Hussein Mehanna, VP &amp; Head of AI/ML at Cruise, highlighted this at Scale’s Transform conference earlier this year: “You have to have a robust, fast, continuous learning cycle, so when you find something that you haven't seen before, you add it quickly to your data, and you learn from it, and you put it back into the model.” This idea of targeted dataset curation is simple; in practice, however, it’s difficult to achieve because of the substantial engineering work required to index millions of raw images and surface difficult scenarios automatically during model evaluation. Let’s put ourselves back in the shoes of an ML engineer. Having diagnosed poor performance on cyclists at night, we now want to target the next experiment at improving this scenario. With a complete set of data curation tools, this workflow would be simple. From one difficult example, source similar images from the large set of unlabeled data and send it through the labeling pipeline for annotation. Once this new batch of data is annotated, include it in the training dataset for subsequent experiments. We call this data-centric approach to experimentation model-in-the-loop dataset curation. Not only is this a highly targeted approach for addressing long-tail performance but, unlike random sampling, it results in better balance across classes and avoids labeling redundant examples that yield little marginal improvement of model performance. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/0o9gb9d3ss"></iframe> Performance on the long tail is often a make-or-break situation when it comes to AI in production. Especially in high-impact applications like healthcare — where data is difficult to collect and equity is critical — being proactive on edge cases is of utmost importance. Given the ubiquity of long-tail challenges in real-world applications, it’s surprising that there isn’t more discussion around how to tailor machine learning experiments toward these cases. ML teams should not run the risk of leaving precious rare scenarios undiscovered in a sea of unlabeled data. When it comes to the long tail, targeted dataset curation is an important tool for achieving performant AI models robust to the complexities of the real world.

The success of real-world machine learning applications often depends on model performance on edge cases.

Sasha Harrison

Life at Scale

Managing 3D Sensor Fusion Data with Scenes in Scale Nucleus

How to Tame the Long Tail in Machine Learning

Getting Started with Nucleus