<h4 id="aerinkim">Aerin Kim:</h4>
<p>Joining us next is Andy Fang from DoorDash. Andy is the co-founder and CTO of DoorDash. As CTO, Andy is responsible for overall product vision, technology roadmap, and architectural direction of DoorDash. Andy and the DoorDash team are actually Scale’s downstairs neighbors in San Francisco. So pre COVID, we used to run into each other all the time in the elevator. Andy, we're so sad not to get to see you more often. But thank you so much for joining us today.</p>
<h4 id="andyfang">Andy Fang:</h4>
<p>Hi, everyone. I'm Andy Fang. I'm the co-founder and CTO at DoorDash. I'm excited to be here today at Scale Transform to talk with you all about how DoorDash uses AI to power our marketplace. I'll first start off by giving you all a quick introduction of DoorDash. And then I'll quickly dive into two different case studies of how we apply AI here to power our marketplace. Starting off with our founding story, so DoorDash, we were founded in 2013 out of a Stanford dorm room. The founders were driven by a mission to empower local merchants. After interviewing hundreds of local businesses in the Bay Area. And we talked to laundromats, toy stores, hair salons, restaurants, you name it. A common theme that we came across was that offering on demand delivery for merchants would alleviate them a logistical headache while funneling them more customer demand.</p>
<h4 id="andyfang-1">Andy Fang:</h4>
<p>To test this idea, we launched PaloAltodelivery.com as our minimum viable product, a website with PDF menus, and a Google Voice number that afforded to all of our personal cell phone numbers. That's where it all began. Everything was manual, routing was done by one of us on the spreadsheet while all of us were on a group call together, we also use find my friends to track each other's locations while we were out doing deliveries. Fast forward to today, we rebranded as DoorDash. And today DoorDash services over 20 million customers, over 450,000 merchants, with a million plus dashers fulfilling deliveries on the platform. We've also become the number one delivery player in both the restaurant food and convenience verticals in America. And we're currently in the United States, Canada, and Australia and looking to expand further globally.</p>
<h4 id="andyfang-2">Andy Fang:</h4>
<p>One of our early mantras was do things that don't scale, which was evident in all the manual techniques we use to power deliveries in the early days. However, as we grew we ultimately had to scale with exponential growth and we've since automated many workflows. Now we saw this as an opportunity not to only automate, but also to think about how we could apply AI techniques to do things better than we could perform manually. DoorDash processes millions of calculations per minute to determine how to optimally service all three sides in the marketplace, you have consumers, the dashers, as well as the merchants. I'll deep dive into two particular problem areas where we've applied AI to further our business and better service our constituents. Starting off with our first case study, creating a rich item taxonomy. We have over tens of millions of restaurant items in the DoorDash catalog and tens of thousands of new items are added every day, most of which have unique taste profiles that we need to differentiate. </p>
<h4 id="andyfang-3">Andy Fang:</h4>
<p>Even for the same type of food item, let's say a chicken sandwich. Some customers would prefer a chicken sandwich from McDonald's, while others would prefer a chicken sandwich from Chick-fil-A. Not to mention all the options one can customize to the item, like adding lettuce, mild, spicy, really spicy, etc. In order to help customers find what they want, we need to be able to understand item characteristics at this lower level of detail. Merchandising and product teams want to be able to create curated editorial experiences like best breakfast options near you or game night finger foods. Strategy teams may want to know if we have enough healthy food options available in the market to determine their sales strategy. Or let's say a customer searches for pad Thai but there's no available nearby options, we might want to understand what dishes with similar characteristics we can suggest instead. We can build specialized models for each of these tasks mentioned above. But that would take too much time to quickly test new ideas.</p>
<h4 id="andyfang-4">Andy Fang:</h4>
<p>Enter building a rich taxonomy to help us solve this problem. Now, in order for us to build this rich taxonomy, we decided to approach these calstar and scaling problems by looking at all the tags we're interested in and then building models to automatically tag every item in our catalog according to this taxonomy. We integrated these models into a human in the loop system defined as a model that requires human interaction, allowing us to collect data efficiently and substantially reduce annotation costs. Our final implementation was a system that grows our taxonomy as we add tags and uses our understanding of the hierarchical relationships between tags to efficiently and quickly learn new classes. Critical things we needed to consider for how to define annotation tags. One, making sure that there's different levels of item tagging specificity that don't overlap. So let's say for coffee, you can say it's a drink, you can say it's non-alcoholic, and you can say it's caffeinated. Those are three separate labels that don't overlap and categorization with each other.</p>
<h4 id="andyfang-5">Andy Fang:</h4>
<p>Second, allow annotators to pick others as an option at each level. Having others is a great catch all option that allows us to process items that were tagged in this bucket to further see how we can add new tags to enrich our taxonomy. Third, make tags as objective as possible. You want to avoid tags that are popular or convenient. Things that would require subjectivity for an annotator to determine. Now we can leverage the tags that we developed towards developing a high precision and high throughput task. High precision is critical for accurate tags, high throughput is critical to make sure that our human tasks are cost efficient. Our taxonomy naturally lends itself towards generating simple binary or multiple choice questions for annotation with minimal background information. So you can still get high precision using less experienced annotators and less detailed instructions, which makes annotator onboarding faster and reduces the risk of an annotator misunderstanding the task objective. </p>
<h4 id="andyfang-6">Andy Fang:</h4>
<p>Now, you can see this example here that with a fried chicken sub there's different kinds of options that we gave an annotator to label it, and whether it's being a sandwich or burger, or whether it's vegan or not. Now we want to talk about how we set up the human in the loop system. And so basically what we did is we had the annotations feed directly into a model. And as you can see in this diagram here, where the steps with human involvement are in red and automated steps are in green. This loop allows us to focus on generating samples we think will be most impactful for the model. Not to mention, we also have a loop to do QA on our annotations, which makes sure that our model is being given high quality data. Through this approach, we've been able to almost double recall while maintaining precision for some of our rarest tags, leading directly to substantially improved customer selection. Pictured here is an example of the difference between the old tags where only items labeled literally with the word desert will return. And the new tags were a query for desert can be expanded with query understanding and can walk down our taxonomy so that we can do beyond simple string matching. </p>
<h4 id="andyfang-7">Andy Fang:</h4>
<p>You know, as opposed to the initial query, which only indexes items with a keyword dessert, we're able to select far more items that we actually consider to be desserts without modifying the search algorithm itself. We're also able to power use cases such as screening for restricted items. 21 plus, for alcohol, for example, relatively easily. Pictured here is a sample of items our model recognizes as alcohol. Now I want to talk through a second case study which is creating an optimized delivery menu. For a restaurant creating an online experience on DoorDash the online menu is the main way to attract customers. Since the menu is a main online touch point an unattractive or poorly organized menu can have a huge negative impact on our merchants online conversion rate regardless of the food quality. If a merchant does not design its menu correctly, customers won't be as attracted to their online offerings and won't buy as often. In order to succeed online merchants need to utilize a set of menu building practices to attract new customers. Empowering local merchants is the DoorDash mission. So we strive to help merchants best present themselves to the customer base of DoorDash.</p>
<h4 id="andyfang-8">Andy Fang:</h4>
<p>In order to surface the characteristics that make for successful online menus, we utilize AI to analyze thousands of existing menus on our platform. We then translated these characteristics into a series of hypotheses for A/B tests, we saw a huge improvement in menu performance from experiments involving header photos and more info about the restaurant. And we also intend to conduct further experience about how to add different information to further improve menu performance. While staff at a restaurant can help sell the menu by crafting a story around the item, or giving live recommendations, quite frankly, this can't happen online. For example, customers might not be familiar with a particular dish and they would require much more information to glean themselves on the online menus, using photos or descriptions to really make the leap of faith and order it. Customers also need help when parsing an online menu to make their decision and we want to make this experience as easy as possible. We want to remove friction such as cutting out on popular items, adding more detailed labels or descriptions to explain menu items and also clearly categorizing menus to make the menu easy to navigate.</p>
<h4 id="andyfang-9">Andy Fang:</h4>
<p>We define a successful menu as one with a high conversion rate at the end of the day. To build a set of features for this kind of model, we looked at each layer and had a menu from the high level menu appearance to detail modifiers for each item. For each layer we brainstorm features relating to key elements such as menu structure, how customize the more the items are, visual aesthetics, like brand pictures, things of that nature. Different merchants structure their menus differently. For example, take a look at these Chicago based restaurants. Duck Duck Goat keeps to a minimalistic but comprehensive selection of categories including mini combos. Ippolito's Pizzeria on the other hand has a very long menu, kind of following a traditional Italian menu structure complete with popular item photos, header image, and a logo. And lastly, Big Star has a very simple menu showing two main categories for food. You got the classics and the tacos and you have a consistent photo layout. </p>
<h4 id="andyfang-10">Andy Fang:</h4>
<p>Another thing to consider is just how customizable these items are, what kind of modifiers are available? While some merchants provide a running list of options available for each item such as Pho 88, other menus keep the option list short to limit the online menu and to simplify preparations. To find out which menu features were the most influential in menu conversion, we use the features mentioned above as inputs to regression models predicting menu conversion. We built our initial regression models using linear regression and base tree models to achieve a baseline error, while the results there were interpretable the error rate was pretty high. And also on top of that many of the features seem to be correlated which led to an issue of collinearity, which made it difficult to determine how changes in each feature impact the target variable directionality.</p>
<h4 id="andyfang-11">Andy Fang:</h4>
<p>The lack of being able to explain this clearly was a pain point for us and is generally a pain point for blackbox models in general. To solve this problem we use Shapley values, which is a game theoretical approach towards model explainability. Shapley values represent the marginal contribution of each feature to the target variable and are calculated by computing the average marginal contribution to the prediction across all permutations before and after withholding that feature. So after examining the resulting Shapley plots of the final model, as you can see here, the top success factor was the number of photos on the menu. And this is particularly important for the top items in the menu as photo coverage of the top items appears much more prominently in the menus overall appearance. Some other top factors and recommendations we made to merchants to help make their menus perform better, one of which is giving higher customizability for items. We found that customers enjoy optionality within the top items and ability to customize provided a degree of familiarity that they could find while dining in.</p>
<h4 id="andyfang-12">Andy Fang:</h4>
<p>Another factor was menus with a healthy mix of appetizers and sides also converted better. This provides customers with more choices to complete their meal and can lead to higher carb values for merchants as well. While top factors that lead to successful online menus aren't surprising, we also have to know that there are variances by cuisine type. This variance is a classic case of where the averages can be deceiving since the average doesn't really represent a normal distribution. And also menus are not all the same. By better understanding customer's expectations around certain types of restaurants and cuisines, merchants are able to better build and customize their menu to basically be more truthful to their brand and the food that they're serving. </p>
<h4 id="andyfang-13">Andy Fang:</h4>
<p>The clearest example of customer expectations at work was actually what we observed with Chinese menus. Unlike most other menus, Chinese menus that were long and had a ton of items actually performed better. When customers dine in at Chinese restaurants with a long and complicated menu, sometimes you even have regional specialties, indicating authenticity. Take a look at this menu. Customer expectations about an authentic Chinese menu leads to these complicated menus having greater conversion rates. On the other hand, menus for wings and pizza places tended to be shorter and well photographed and also have a ton of options for customizing. You can think about it where when customers visit these types of merchants they usually have one to two items in mind or are just looking for things to customize like sauces for wings or toppings for pizzas. </p>
<h4 id="andyfang-14">Andy Fang:</h4>
<p>To wrap it up. These are just some of the ways we're using AI to scale our marketplace and to make it easier for customers to find what they want, and also make it easier for merchants to position themselves in an online world. If you're curious to learn more about these two particular case studies, or you're curious to read about more case studies, feel free to check out our DoorDash engineering blog. It's just DoorDash.engineering. It was a pleasure to speak with all of you today. I really hope you enjoyed and took something out of it and enjoy the rest of the conference.</p>

Andy Fang, CTO & Co-Founder of DoorDash discusses how DoorDash leverages artificial intelligence and machine learning to power their marketplace. 

Applied AI/ML at DoorDash with Andy Fang

<p>Andy Fang is the Co-founder and Chief Technical Officer of DoorDash, a technology company passionate about transforming local businesses and dedicated to enabling new ways of working, earning, and living. Andy leads DoorDash’s Consumer Engineering team where he is focused on growing and retaining DoorDash’s customer user base while increasing engagement for DoorDash’s marketplace products. In this role, Andy is responsible for the overall product vision, technology roadmap and architectural direction of the DoorDash consumer platform. Andy holds a BS in Computer Science from Stanford University, where he met fellow cofounders Tony Xu and Stanley Tang whereupon the concept for DoorDash was born.</p>

Products

Solutions

Resources

Company

Applied AI/ML at DoorDash with Andy Fang