Thomas Liao

<h1>Background</h1> At Scale AI, we’re always looking for ways to apply machine learning (ML) in our internal systems to deliver the best results for our customers. We’ve previously written about Scale’s <a href="https://scale.com/blog/how-we-scale-machine-learning" rel="noopener noreferrer" target="_blank">ML-powered pre-labeling</a> as well as our <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">active tooling, like Autosegment</a>, which helps our Taskers label faster and at higher quality. In this piece, we discuss how we think about and incorporate some basic tenets of design thinking to develop ML systems that make a difference for our labeling pipeline. <a href="http://web.mit.edu/jrankin/www/engin_as_lib_art/Design_thinking.pdf" rel="noopener noreferrer" target="_blank">Design thinking</a> is a mindset and set of practices which foster the collaboration required to solve problems in human-centered ways and was notably popularized by organizations such as IDEO. It’s useful for machine learning engineers who need to understand the full scope of how the system they are building will be used and are responsible for both business and operational outcomes. We’ll describe how design thinking techniques can be applied to scoping machine learning problems and introduce some concepts to help business stakeholders communicate more effectively with ML engineers.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8a41ed7a-4287-491b-6202-1f80a4c29a00/public" alt="Phases of Developing an ML System">Idealized phases of developing an ML system. In practice, it’s not this linear.In our experience, the different phases of developing an ML system are: <ol><li>Discovery - Locate a business or operational pain point</li><li> </li><li> Design - Map the different ways a new or existing model could fit into the</li><li> extant pipeline</li><li> </li><li> </li><li> Decision &amp; Alignment - Align with operational and engineering</li><li> stakeholders on scope and resourcing</li><li> </li><li> </li><li> Development, Integration, Deployment - Deliver the system. Build data</li><li> pipelines, train models, integrate into the pipeline, etc.</li><li> </li><li> </li><li> Measurement &amp; Experimentation - Identify the effect of the ML system as</li><li> an intervention on the existing pipeline.</li><li> </li><li> </li><li> Continuous improvement - Monitor and retrain to account for distribution</li><li> shifts and new edge cases</li><li> </li><li> </li></ol> This is a descriptive mental model, not a prescriptive blueprint. In reality, a project is always moving between different phases and can often be in multiple phases at the same time. For example, measurement and development form a tight iteration loop. But, it’s still useful to think about these phases in isolation. Concepts from design thinking are mostly employed in the Discovery and Design phases when the problem statements are the most inchoate and ill-formed. In these phases, the need or specific outcome is communicated to an ML engineer and an initial strategy to address that need or deliver that outcome is designed. We focus on the Discovery and Design phases in this blog and cover the remaining phases briefly.<h1>Phase 1: Discovery</h1> An ML system gains life when a need is discovered. By need, we distinguish between a request for a specific feature, such as a linter (a tool we use to ensure data quality), and the underlying operational or business pain point or problem - such as a new labelling capability requiring ML support. It’s not uncommon for operational or business teams to stumble into the <a href="https://en.wikipedia.org/wiki/XY_problem" rel="noopener noreferrer" target="_blank">XY problem</a> by pattern matching and requesting a clone of an existing feature, without realizing that their underlying problem is actually quite different. Or, they might ask for something which is incredibly difficult, unaware that a slightly modified version would save literally months of development time. So an ML engineer should always hunt down the true problem that needs to be solved. Pain points and problems can be project-specific or exist horizontally across business teams or units. Historically, many of the problems that the ML team at Scale have tackled are project-specific and brought to us by our cross-functional stakeholders. For example, a project manager running into issues with a labeling project or a product manager looking to launch a new feature will surface these project-specific issues to our team. Tackling these isolated problems can result in marooned solutions which aren’t readily reusable. Solving horizontal issues, in comparison, maximizes our leverage, since the ML systems we build will then scale with our business. However, cross-business issues that can be solved with ML are often harder to identify because of the sheer operational complexity. These issues require a more active discovery process and take more time and deeper alignment with partners. The problems with the highest stakes are not obvious to ML engineers, who don’t get day to day exposure to them, even as these frictions impose significant pain on the operations or business side. To get a sense for this, imagine explaining the pain of managing multiple python environments and package versions to an operational lead - it can be incredibly painful for you, yet esoteric, even cryptic, to them. As another example, while working on an image semantic segmentation project, we learned that pre-labels generated by a model specific to that project were sometimes ignored by taskers, but the root causes were unclear. A close investigation identified a number of culprits, ranging from the inflexibility of pre-labels to having optimized for the wrong metric during development. <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">Autosegment</a>, developed by Sean Li, Saleh Hamadeh, and the two of us, aims to tackle these problems. Autosegment solves a challenge for all image segmentation projects and isn’t restricted to a particular project. Learning about this challenge and building the appropriate solution required input from numerous stakeholders - from the project leads, who wanted to accelerate taskers; the taskers themselves, who used our tools and knew what worked and what didn’t; to engineers, who were familiar with the implementation of image segmentation in our interface and understood the kinds of tradeoffs that different tools would be need to make.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/7866cd69-bd81-40f7-4e44-6094a84f2c00/public" alt="Stakeholder concerns">Example overlap of concerns between different stakeholders for a new ML-assisted tool. Identifying the pain points of the end user always requires talking to the end user. Instead of theorizing what could be a problem for them, it’s always more effective to speak to users directly and observe them as they encounter the problem. When building systems to accelerate labeling operations, ML engineers need to build user empathy. A simple way to build user empathy is to spend time tasking and dogfooding our own platform. There is no substitute for first hand experience for understanding what makes tasking hard.<h1>Phase II: Design</h1> After Discovery, the project enters a design phase where an initial approach is sketched out - what kind of model to build, what sort of data is needed, the timelines, and so on. Designing effective ML systems at Scale requires two things: a clear need, which we’ve discussed above, and knowing the many different ways a model can fit into the labelling pipeline. This last point is worth further elaboration. The same machine learning model can be used for a variety of operational purposes. For example, a model that predicts bounding boxes for cars can be used to:<ol><li>Estimate the number of boxes in a task (task difficulty estimation)</li><li>Prepopulate the scene with bounding boxes (pre-labelling)</li><li> </li><li> Suggest fixes to bounding boxes made in real-time by a tasker (active</li><li> tooling)</li><li> </li><li> </li><li> Identify missing boxes or unexpected boxes in a completed scene (linting)</li><li> For each of these, the inference API could have the exact same request and</li><li> return formats. However, each of the four approaches above require different</li><li> operational adjustments, not all of which may be feasible. Task difficulty</li><li> estimation makes assumptions about what makes a task difficult or easy for</li><li> humans and implicitly maps these to legible subpopulations of taskers.</li><li> Active tooling can require nontrivial operational investment in teaching</li><li> taskers how to use the new tool and engineering lift to build new frontend</li><li> components. Building a useful ML system means designing how an ML model will</li><li> interface with the rest of the operational and engineering systems, in</li><li> conjunction with actually training the ML model.</li><li> </li><li> </li></ol><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/8e6d0b0b-5185-4a7c-227e-03350b1bdd00/public" alt="One model, many uses">The same model may be applied in different ways to an existing pipeline, requiring different accommodations by engineering or operational teams. We’ve noticed two particular traps for ML engineers which can lead them on wild goose chases during the design phase. The first is misalignment between the machine learning problem and the business problem. The second is confusing the customer and the end user of a system. ML engineers are trained to use models to solve machine learning tasks, like box detection or text classification. For a box detection dataset, you’d use a box detection model. For a text classification dataset, a text classification model. When the business problem looks like an image classification problem, you might reach for an image classification model, only to discover that what you really need to solve is tabular regression. Faced with using ML to assist, say, labelling semantic segmentation training data, a common reflex is to train a semantic segmentation model to do the labelling. But if you already had a capable model, you wouldn’t need to collect training data for it! The mistake here is to apply the familiar task paradigm - semantic segmentation - to solve the business problem - accelerating semantic segmentation. Many operational questions take the form “Is something X or not X?” To an ML engineer, this sounds like a classification problem, and it’s easy to reach for a classification model which directly answers the question. Putting aside whether modeling X directly is feasible, it’s often the case that we already have an operational or engineering system which could answer this question more effectively with some additional information. In this situation, it may be faster to figure out how to extract the augmenting information rather than replicate the existing system altogether; a strong reason to apply a modeling scheme that looks very different from the modeling question.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/3505a43e-5c45-4274-cdd3-2e27242e0600/public" alt="Leveraging existing systems">Modeling the business problem to solve doesn’t always mean reaching for the most obvious machine learning task paradigm. The second trap is mistaking the end user of a system with the people who request or manage the system. When we build a tool like <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">Autosegment</a>, even though the end user of the active tooling is the Tasker, the internal customer is actually the Delivery Operations team. What we build has to meet the customer’s expectations, so even if Taskers enjoy using our tool, we need to make sure to move the needle for the Delivery Operations team. But the expectations and needs of the end user may differ or conflict from the internal customer (e.g. latency tradeoff with cost). In order to avoid falling into these two traps, ML engineers have to be deliberate in how they transport the business problem into a machine learning context; and always keep in mind the different stakeholders.<h1>Phases III - VI: Decision and Alignment Through Continuous Improvement</h1> After the initial Discovery and Design phases, the project needs a go / no-go decision by both the ML team as well as cross-functional teams. In this phase, all teams must decide whether to allocate the necessary resources to the project. If the ML team decides to dedicate resources but the cross-functional team cannot commit the time or spare the bandwidth, the effort will ultimately fall short. Some factors teams must consider when making this decision include: prioritization, level of effort needed, and available bandwidth. Once all teams commit to the project, we can develop the ML-specific components of the ML system. This is when we build data pipelines, train models, and ship an endpoint. Once the model is developed, the project nearly always needs engineering and operational integration. For example, if we are developing a new pre-labeling model, the model requires integration with our labeling pipelines to attach the pre-label to the task. Now the project is actually deployed. Deployment should also follow best practices: meaning the feature should only be enabled for a small percent of tasks or users at the beginning, making it easier to revert if there are any issues. The system should then be constantly measured. Especially in the first few weeks, close attention should be paid to ensure no issues arise. We then use the metrics and qualitative feedback gathered in the measurement step to continuously improve the model. Models often need to be retrained to account for new edge cases. As we maintain the production model, we often encounter data distribution shifts or taxonomic changes which must be addressed.<h1>Conclusion:</h1> We hope this post provided some insight on how we design our ML systems and how you and your teams can effectively partner with horizontal teams to deploy models that serve your business needs. Throughout the phases of ML development, we strive to ensure alignment with our cross-functional stakeholders. While gaining and maintaining alignment takes time and deliberate effort, it allows us to unlock capabilities on our platform that wouldn’t otherwise be possible.

Understanding operational and business context is critical to shipping ML systems that work. 

How we leverage concepts from design thinking to design ML systems in our internal processes to deliver results for customers.

Design Thinking for ML Systems at Scale

<h1> </h1><h1> Why build an ML-based workflow for data labeling?</h1> Delivering high-quality data labels quickly is a challenging task. Even for humans, image and video data commonly found in self-driving or robotic applications can be compositionally challenging to parse and correctly annotate. At Scale, we use machine learning to augment human workflows to both improve the quality and speed of labeling. Since deep learning models can struggle to perform consistently across diverse data domains like self-driving car scenes, finding the optimal balance between ML automation and human supervision is necessary to ensure we maintain high-quality.Autosegment, an ML-powered tool that can accurately segment instances within a human-defined bounding box, allows us to delegate high-level reasoning about semantics and taxonomy to Taskers (human labelers) and basic pixel-level processing to an ML model. This division of labor enables us to build a general purpose and accurate ML labeling tool. Despite using a simple neural network, Autosegment outperforms prior tools.<h1> </h1><h1> The Challenge of Semantic Segmentation Labeling</h1> Semantic segmentation involves human Taskers identifying and labeling every pixel in a visual scene — a notoriously time-consuming and tedious process. Most Tasker time in this manual approach — about 95% — is consumed by time-intensive work, such as carefully tracing the outlines of objects in a scene. For example: it takes roughly one minute to trace the outline of a car and only three seconds to click on that outline and select the label “Car.” What if we could instead use machine learning to automate the most laborious parts of semantic segmentation? What if an ML model did the semantic segmentation, and the Tasker only had to select the right label?<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/72c723b3-46ae-4384-b75b-4a87b5e40b00/public" alt="Semantic Segmentation Task"> A visual scene which has been semantically segmented. Each pixel is color coded to denote the class of object it represents, i.e. Pedestrian, Road, Bike, etc. As ML engineers, we rely heavily on quantitative validation metrics, such as label accuracy and intersection-over-union (IoU), to determine the quality of our models. However, when building models that optimize for a particular business outcome — for instance, reducing the time it takes Taskers to semantically segment images — we need to think critically about which model to use and how we measure its performance.. In this example, our objective is to produce predictions that streamline the Taskers’ job. This is not necessarily the same as “best model performance.” For example, an instance segmentation model might be consistently inaccurate by just a few pixels around an object. This model might have a high IoU score but very little practical usefulness because Taskers would need to re-do every trace of the outline. Conversely, a model that can perfectly segment an object, even if it only identifies half of it, would still be extremely useful. Despite having lower IoU validation scores, it’s still an improvement to the overall workflow. Because it takes so long to trace outlines manually, having a model produce pixel-accurate partial outlines is still a significant time saver. <h1> </h1><h1> The Active Tooling Paradigm and the Autosegment Tool</h1><h1> </h1>Autosegment is a machine learning tool that segments the most salient object within a human-provided box annotation. It incorporates Tasker feedback at annotation time in a way that is flexible and iterative. When building Autosegment, we realized early on that the only way to provide flexibility and high quality was to give Taskers access to Active Tools. Active Tools let Taskers make changes to the model inputs and parameters in real time to generate the best output for their use case. This is similar to “prompt engineering,” the process by which we can coerce a neural language model like GPT-3 to generate desired text when given precise prompts. For more information, OpenAI hosts <a href="https://beta.openai.com/docs/guides/completion/prompt-design" rel="noopener noreferrer" target="_blank">excellent documentation</a> on how to design prompts for their GPT-3 model. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/07588dc3-6a06-45e7-c61f-a51562470900/public" alt="Open-AI GPT-3 Model"> Humans working alongside models can produce better results faster than models alone Using Active Tools, we give Taskers as many options as possible to make their tasks easier. For example, they can adjust the annotation box shape and the prediction confidence threshold to configure the model to speed up their particular labeling task. <img src="https://cdn.builder.io/o/assets%2Fe0438815ba51486bbb6a202747122d4b%2Ffef154571d5846b6b87674a46a123ac5%2Fcompressed?apiKey=e0438815ba51486bbb6a202747122d4b&amp;token=fef154571d5846b6b87674a46a123ac5&amp;alt=media&amp;optimized=true" alt="Autosegment Tool Demo">A demo of the Autosegment tool. Enabling humans to freely explore the output space of the model is a key advantage of active tooling. We've already seen how this is useful. Overall, we have observed the median annotation time per scene is about 30% faster with Autosegment. We also interviewed Taskers to see how they are using the tool. While our model is trained to segment cars and motorcycles, one Tasker used the tool to segment everything. While this wasn’t a workflow we had envisioned, it halved their time per scene segmentation, from 40 minutes to 20 minutes. In a discussion we had with five Taskers, they explained that they consider Autosegment to be more accurate and efficient than previous ML solutions. Now we can simply supply robust and general online models and let Taskers optimize the rest. <h1>The Autosegment Model</h1>Autosegment is a convolutional neural network trained to segment the most salient instance within a Tasker-defined bounding box. This simple design allows us to significantly reduce our customers’ operational overhead to deploy ML, while providing pixel-perfect instance segmentations, even on previously unseen instance types. The Autosegment model does not have to be particularly complex to produce compelling results. Autosegment has no notion of different semantic types; instead, it abstractly identifies and segments based on scene composition. The model is trained to always generate a segmentation. The training data includes a few key classes like pedestrians and vehicles. The task is binary; it simply determines whether each pixel belongs to the salient object or not. We don’t train the model with negative samples. Instead, the model makes its best guess on each pixel to generate a segmentation. We define the salient instance as the object that owns the central pixel of the crop, so even if the scene is cluttered, the model can identify the target instance to segment. This binary segmentation task across instance classes and types allows the model to generalize well on new classes and taxonomies. Indeed, we can transfer not only between objects of the same class, such as cars in different countries, but between different classes altogether — for instance, moving from cars to motorcycles to street signs. This transfer can be achieved with limited retraining or, in some cases, no retraining at all, achieving zero-shot transfer. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/1a4x5jij9v"></iframe> Using salient object detection on specific crops of the scene can generate even more accurate segmentation. We were able to double the edge precision score from 0.26 to 0.52 when compared to off-the-shelf full-scene semantic segmentation models. Instead of supplying a downsampled version of the whole image to a neural network, the selected region is fed to the model at full resolution. Because the input is smaller, fewer pooling operations are required to achieve a globally receptive field. As a result, there is less downsampling and blurring. Our model achieves better fidelity around the edges of instances, a crucial part of providing high-quality data annotations to our customers. Improving full-scene labeling to the same degree would require exponentially more effort and complexity. In summary, we developed Active Tooling to create an ML-based workflow capable of solving basic computer vision problems. This workflow helps our Taskers reach higher levels of quality faster than in previous workflows. Innovations like Autosegment are critical to enable the “Moore's Law of data labeling.” At Scale AI, we’re excited to push the boundaries of human-AI interaction for our customers.

Scale AI accelerates image annotation with semantic segmentation active tooling. 

Thomas Liao

Design Thinking for ML Systems at Scale

Faster Semantic Segmentation with Machine Learning-Assisted Workflows

ML & Human Consensus: The Best of Both Worlds