Logan Ford

When Covid-19 first hit, many companies had to rapidly reorient their operations – from altering their supply chains to implementing social distancing and work-from-home orders. One sector that was particularly hard hit was the restaurant industry. When in-person dining was suspended, restaurants turned to delivery to survive. But setting up a delivery service from scratch takes time and money that many establishments, with their income drying up, did not have. Suddenly, food delivery platforms became an economic lifeline. To ensure restaurants could onboard quickly and seamlessly in these critical early days, one of the leading delivery platforms turned to Scale.<h3>Scaling labeling</h3> Every time a restaurant joins one of the delivery platforms or changes its menu, the menu’s contents need to be inputted into the platform so users can select their food. Inputting menus manually is slow, expensive, and leads to mistakes. With restaurants relying on delivery to reach customers and support their staff, a leading delivery platform knew it was vital to ensure an efficient process for its partner restaurants. This presented Scale with unique latency and quality control challenges, requiring both an incredibly high level of accuracy with a fast turnaround to provide operational efficiencies at a critical time for our customer. Our aim was to use the combination of partial automation and human quality control in our <a href="https://scale.com/document" rel="noopener noreferrer" target="_blank">Scale Document</a> labeling pipeline to make dramatic efficiency improvements over manual processes. In all, we nearly halved the time it takes our customer to process menus and reduced critical error rates to &lt;1% for all items labeled, allowing them to onboard restaurants smoothly exactly when they needed it.<h3> </h3><h3> Building the right data infrastructure</h3> While processing standardized documents such as food menus, government IDs or loan application forms might be intuitive and simple for humans, it is a surprisingly nontrivial problem for algorithms, involving much more than automatic transcription. In particular, the data often contains many dependencies that need to be captured accurately. For example, ordering a pepperoni pizza doesn’t just mean choosing the type, but also the size, the type of base, choosing to substitute or customize toppings (which can also vary in price depending on the size of the pizza, creating more interdependencies), and whether to add extras like burgers or sandwiches. It’s important to capture these dependencies in the menu correctly, otherwise inputting errors risk causing losses for both the restaurant and the delivery platform.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/e10f107d-1efe-45c5-45a3-db2291017800/public" alt="DD_decision"> Algorithmically, this has to be represented as complex decision trees made up of categories, items, and lists of options, capturing the relationships between them. Thankfully, both our document processing pipeline and form capture tools were already set up to capture these types of nested data structures automatically without the need for customization. This meant we were able to quickly pivot our tools for our customer’s requirements. <h3>Judicious Automation and Scaling</h3> Once we have the right tooling in place to pull the raw data and label complex data structures, we can then start processing the data. Given how many menus delivery platforms were processing at the start of shelter-in-place, and how important for restaurants’ continued operations it was to process this data quickly with no drop-off in quality, we need to make this process as efficient as possible. At Scale AI, we believe that a mix of judicious labeling automation with continued human oversight is the only way to provide data at the scale, quality, and low cost needed to enable many enterprise applications of AI. In particular, we aim to automate the lower-skilled parts of the data labeling pipeline to focus human review on the hard parts – quality control, edge cases, and the most complex data types. That way, we can help guarantee quality without sacrificing efficiency. We use automation to prefill menu taxonomies for expert human labelers to review and confirm. We trained our form capture tool to automatically understand document structure and predict the next transcription. And when our labelers needed to add inputs manually, we built a “smart suggestion” feature that provides them with ready prompts for common items that they have already encountered in their batch of menus. Both have helped improve both the efficiency and accuracy of labeling simultaneously: allowing us to transcribe more than 3,000 menus in a single day and return menus within short turnaround times to onboard restaurants within 24 hours from start-to-finish. By drawing on our wide network of well-trained labelers and augmenting their work with careful automation, we’re able to handle changes in demand dynamically – essential for providing a smooth service during sudden peaks in demand. <h3> </h3><h3> Custom Benchmarks and Integrations</h3><h3> </h3> Once data is flowing smoothly and efficiently through this labeling pipeline, the next step is to ensure that the data is being labeled to an incredibly high standard, with almost zero errors. We’ve developed a range of techniques to guarantee label quality, including the randomized use of benchmark quality tests, confidence-based consensus, and, for ambiguous labeling requirements, our own <a href="https://scale.com/blog/how-to-label-1m-data-points-week" rel="noopener noreferrer" target="_blank">automated benchmark generation system</a>. Benchmarking is not always an objective science – the most effective benchmarks are often task-specific. The high stakes of onboarding restaurants during a pandemic required particularly high guarantees of labeling quality. To meet them, we worked closely with our customer to build custom benchmarks for this workflow, turning the most complex mislabeled menus into benchmarks that assessed our labelers’ performance. In surveying mislabeled menus, we noticed two key groups of errors. One was associated with the cross-referencing of menu sections between many items and mistaking how they relate to each other, while the other group tended to be mistakes from larger menus where it may be less obvious when certain features of the menu are missing. By identifying these key groups and creating benchmarks out of the associated menus, we can ensure that tasker performance in such cases exceeds the necessary standard to meet menu quality needs. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/fc0a9a32-08be-4505-422b-a05d016c3f00/public" alt="Menu_dd_photos"> The menu on the left showcases large amounts of cross-referencing and item optionality, while the menu on the right shows a fairly common amount of menu feature density. <h3>What’s Next</h3> AI models are rapidly becoming critical infrastructure for a huge range of businesses – a trend of automation that the COVID pandemic has rapidly accelerated. But gathering effective, accurate, and unbiased data currently remains such a challenging task that it is preventing real-world AI systems from reaching their full potential and presenting high complexity barriers for smaller companies. We’re tackling a whole host of problems to help accelerate that progress. Right now, teams at Scale AI are making better use of machine learning to make labeling orders of magnitude more efficient, building tools to process increasingly complex data types, such as 3D point clouds in computer vision, and developing new infrastructure tools to help streamline the management of data. Our first such management tool, <a href="https://scale.com/blog/introducing-scale-nucleus" rel="noopener noreferrer" target="_blank">Scale Nucleus</a>, is already helping our computer vision customers automate time-consuming manual steps in the ML development process. We’re now working on deploying it for our customers in natural language. Ultimately, we want to make it as easy to deploy AI as any other type of software. If you’re interested in joining us in solving these problems, take a look at our <a href="https://scale.com/careers" rel="noopener noreferrer" target="_blank">careers page</a> for our latest open positions. If you have projects that require high-quality data labeling, <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">let our team know</a>!

How Scale AI worked with a leading food delivery platform to ensure restaurants could onboard quickly and seamlessly. 

Logan Ford

Scaling Menu Transcription Tasks with Scale Document