Andrew Ng discusses the importance of data in developing AI applications and joins Scale AI CEO, Alexandr Wang in a fireside chat.

From Big Data to Good Data with Andrew Ng

Andrew_Ng_Headshot

Andrew Ng is Founder of DeepLearning.AI, General Partner at AI Fund and an Adjunct Professor at Stanford University.

As a pioneer both in machine learning and online education, Dr. Ng has changed countless lives through his work in AI, authoring or co-authoring over 100 research papers in machine learning, robotics and related fields. Previously, he was chief scientist at Baidu, the founding lead of the Google Brain team, and the co-founder of Coursera – the world’s largest MOOC platform. Dr. Ng now focuses his time primarily on his entrepreneurial ventures, looking for the best ways to accelerate responsible AI practices in the larger global economy.

Andrew Ng

Scale Transform 2021 | From Big Data to Good Data with Andrew Ng

Machine Learning frameworks have gone through various transitions over the years. In this talk, Soumith recaps ML frameworks to date and discusses what the future might look like

Future of ML Frameworks with Soumith Chintala

Soumith Chintala

Soumith Chintala is a Researcher at Facebook AI Research, where he works on high-performance deep learning. Soumith created PyTorch, a deep learning framework that has traction among researchers. Prior to joining Facebook in August 2014, he worked at MuseAmi, where he built deep learning models for music and vision targeted at mobile devices. He holds a Masters in CS from NYU, and spent time in Yann LeCun’s NYU lab building deep learning models for robotics, pedestrian detection, natural image OCR, depth-images among others.

Scale Transform 2021 | Future of ML Frameworks with Soumith Chintala

Dr. Fei-Fei Li discusses AI inspired by human intelligence and developed to augment human capabilities, and the importance of diversity in the field of AI.

Human-Centered AI with Fei-Fei Li

Fei-Fei Li

Dr. Fei-Fei Li is the Sequoia Professor of Computer Science at Stanford University and Denning Co-Director of the Stanford Institute for Human-Centered AI (HAI). Her research includes cognitively inspired AI, machine learning, deep learning, computer vision and AI+healthcare. Before co-founding HAI, she served as Director of Stanford’s AI Lab. During her Stanford sabbatical from 2017 - 2018, Dr. Li was a Vice President at Google and Chief Scientist of AI/ML at Google Cloud. Prior to joining Stanford, she was on faculty at Princeton University and University of Illinois Urbana-Champaign. Dr. Li is co-founder and chairperson of the national non-profit AI4ALL, which is increasing inclusion and diversity in AI education. She is an elected member of the National Academy of Engineering, among other distinctions. She holds a B.A. degree in physics from Princeton with High Honors, and a PhD degree in electrical engineering from California Institute of Technology.

Alex_Wang_2

Scale was founded by Alexandr Wang, an MIT college dropout, national programming competitor and former engineering lead at Quora. After being a teenage tech lead at Quora, Alex founded Scale, becoming the youngest founder funded by Accel as a teenager, was recognized as Forbes 30 under 30, and is a frequent speaker at conferences including Auto AI and API Mixtape.

Alexandr Wang

Scale Transform 2021 | Human-Centered AI with Fei-Fei Li

<h4 id="aerin-kim%3A" tabindex="-1">Aerin Kim:</h4>
<p>Joining us next is Andy Fang from DoorDash. Andy is the co-founder and CTO of DoorDash. As CTO, Andy is responsible for overall product vision, technology roadmap, and architectural direction of DoorDash. Andy and the DoorDash team are actually Scale’s downstairs neighbors in San Francisco. So pre COVID, we used to run into each other all the time in the elevator. Andy, we’re so sad not to get to see you more often. But thank you so much for joining us today.</p>
<h4 id="andy-fang%3A" tabindex="-1">Andy Fang:</h4>
<p>Hi, everyone. I’m Andy Fang. I’m the co-founder and CTO at DoorDash. I’m excited to be here today at Scale Transform to talk with you all about how DoorDash uses AI to power our marketplace. I’ll first start off by giving you all a quick introduction of DoorDash. And then I’ll quickly dive into two different case studies of how we apply AI here to power our marketplace. Starting off with our founding story, so DoorDash, we were founded in 2013 out of a Stanford dorm room. The founders were driven by a mission to empower local merchants. After interviewing hundreds of local businesses in the Bay Area. And we talked to laundromats, toy stores, hair salons, restaurants, you name it. A common theme that we came across was that offering on demand delivery for merchants would alleviate them a logistical headache while funneling them more customer demand.</p>
<h4 id="andy-fang%3A-1" tabindex="-1">Andy Fang:</h4>
<p>To test this idea, we launched <a href="http://PaloAltodelivery.com" target="_blank">PaloAltodelivery.com</a> as our minimum viable product, a website with PDF menus, and a Google Voice number that afforded to all of our personal cell phone numbers. That’s where it all began. Everything was manual, routing was done by one of us on the spreadsheet while all of us were on a group call together, we also use find my friends to track each other’s locations while we were out doing deliveries. Fast forward to today, we rebranded as DoorDash. And today DoorDash services over 20 million customers, over 450,000 merchants, with a million plus dashers fulfilling deliveries on the platform. We’ve also become the number one delivery player in both the restaurant food and convenience verticals in America. And we’re currently in the United States, Canada, and Australia and looking to expand further globally.</p>
<h4 id="andy-fang%3A-2" tabindex="-1">Andy Fang:</h4>
<p>One of our early mantras was do things that don’t scale, which was evident in all the manual techniques we use to power deliveries in the early days. However, as we grew we ultimately had to scale with exponential growth and we’ve since automated many workflows. Now we saw this as an opportunity not to only automate, but also to think about how we could apply AI techniques to do things better than we could perform manually. DoorDash processes millions of calculations per minute to determine how to optimally service all three sides in the marketplace, you have consumers, the dashers, as well as the merchants. I’ll deep dive into two particular problem areas where we’ve applied AI to further our business and better service our constituents. Starting off with our first case study, creating a rich item taxonomy. We have over tens of millions of restaurant items in the DoorDash catalog and tens of thousands of new items are added every day, most of which have unique taste profiles that we need to differentiate.</p>
<h4 id="andy-fang%3A-3" tabindex="-1">Andy Fang:</h4>
<p>Even for the same type of food item, let’s say a chicken sandwich. Some customers would prefer a chicken sandwich from McDonald’s, while others would prefer a chicken sandwich from Chick-fil-A. Not to mention all the options one can customize to the item, like adding lettuce, mild, spicy, really spicy, etc. In order to help customers find what they want, we need to be able to understand item characteristics at this lower level of detail. Merchandising and product teams want to be able to create curated editorial experiences like best breakfast options near you or game night finger foods. Strategy teams may want to know if we have enough healthy food options available in the market to determine their sales strategy. Or let’s say a customer searches for pad Thai but there’s no available nearby options, we might want to understand what dishes with similar characteristics we can suggest instead. We can build specialized models for each of these tasks mentioned above. But that would take too much time to quickly test new ideas.</p>
<h4 id="andy-fang%3A-4" tabindex="-1">Andy Fang:</h4>
<p>Enter building a rich taxonomy to help us solve this problem. Now, in order for us to build this rich taxonomy, we decided to approach these calstar and scaling problems by looking at all the tags we’re interested in and then building models to automatically tag every item in our catalog according to this taxonomy. We integrated these models into a human in the loop system defined as a model that requires human interaction, allowing us to collect data efficiently and substantially reduce annotation costs. Our final implementation was a system that grows our taxonomy as we add tags and uses our understanding of the hierarchical relationships between tags to efficiently and quickly learn new classes. Critical things we needed to consider for how to define annotation tags. One, making sure that there’s different levels of item tagging specificity that don’t overlap. So let’s say for coffee, you can say it’s a drink, you can say it’s non-alcoholic, and you can say it’s caffeinated. Those are three separate labels that don’t overlap and categorization with each other.</p>
<h4 id="andy-fang%3A-5" tabindex="-1">Andy Fang:</h4>
<p>Second, allow annotators to pick others as an option at each level. Having others is a great catch all option that allows us to process items that were tagged in this bucket to further see how we can add new tags to enrich our taxonomy. Third, make tags as objective as possible. You want to avoid tags that are popular or convenient. Things that would require subjectivity for an annotator to determine. Now we can leverage the tags that we developed towards developing a high precision and high throughput task. High precision is critical for accurate tags, high throughput is critical to make sure that our human tasks are cost efficient. Our taxonomy naturally lends itself towards generating simple binary or multiple choice questions for annotation with minimal background information. So you can still get high precision using less experienced annotators and less detailed instructions, which makes annotator onboarding faster and reduces the risk of an annotator misunderstanding the task objective.</p>
<h4 id="andy-fang%3A-6" tabindex="-1">Andy Fang:</h4>
<p>Now, you can see this example here that with a fried chicken sub there’s different kinds of options that we gave an annotator to label it, and whether it’s being a sandwich or burger, or whether it’s vegan or not. Now we want to talk about how we set up the human in the loop system. And so basically what we did is we had the annotations feed directly into a model. And as you can see in this diagram here, where the steps with human involvement are in red and automated steps are in green. This loop allows us to focus on generating samples we think will be most impactful for the model. Not to mention, we also have a loop to do QA on our annotations, which makes sure that our model is being given high quality data. Through this approach, we’ve been able to almost double recall while maintaining precision for some of our rarest tags, leading directly to substantially improved customer selection. Pictured here is an example of the difference between the old tags where only items labeled literally with the word desert will return. And the new tags were a query for desert can be expanded with query understanding and can walk down our taxonomy so that we can do beyond simple string matching.</p>
<h4 id="andy-fang%3A-7" tabindex="-1">Andy Fang:</h4>
<p>You know, as opposed to the initial query, which only indexes items with a keyword dessert, we’re able to select far more items that we actually consider to be desserts without modifying the search algorithm itself. We’re also able to power use cases such as screening for restricted items. 21 plus, for alcohol, for example, relatively easily. Pictured here is a sample of items our model recognizes as alcohol. Now I want to talk through a second case study which is creating an optimized delivery menu. For a restaurant creating an online experience on DoorDash the online menu is the main way to attract customers. Since the menu is a main online touch point an unattractive or poorly organized menu can have a huge negative impact on our merchants online conversion rate regardless of the food quality. If a merchant does not design its menu correctly, customers won’t be as attracted to their online offerings and won’t buy as often. In order to succeed online merchants need to utilize a set of menu building practices to attract new customers. Empowering local merchants is the DoorDash mission. So we strive to help merchants best present themselves to the customer base of DoorDash.</p>
<h4 id="andy-fang%3A-8" tabindex="-1">Andy Fang:</h4>
<p>In order to surface the characteristics that make for successful online menus, we utilize AI to analyze thousands of existing menus on our platform. We then translated these characteristics into a series of hypotheses for A/B tests, we saw a huge improvement in menu performance from experiments involving header photos and more info about the restaurant. And we also intend to conduct further experience about how to add different information to further improve menu performance. While staff at a restaurant can help sell the menu by crafting a story around the item, or giving live recommendations, quite frankly, this can’t happen online. For example, customers might not be familiar with a particular dish and they would require much more information to glean themselves on the online menus, using photos or descriptions to really make the leap of faith and order it. Customers also need help when parsing an online menu to make their decision and we want to make this experience as easy as possible. We want to remove friction such as cutting out on popular items, adding more detailed labels or descriptions to explain menu items and also clearly categorizing menus to make the menu easy to navigate.</p>
<h4 id="andy-fang%3A-9" tabindex="-1">Andy Fang:</h4>
<p>We define a successful menu as one with a high conversion rate at the end of the day. To build a set of features for this kind of model, we looked at each layer and had a menu from the high level menu appearance to detail modifiers for each item. For each layer we brainstorm features relating to key elements such as menu structure, how customize the more the items are, visual aesthetics, like brand pictures, things of that nature. Different merchants structure their menus differently. For example, take a look at these Chicago based restaurants. Duck Duck Goat keeps to a minimalistic but comprehensive selection of categories including mini combos. Ippolito’s Pizzeria on the other hand has a very long menu, kind of following a traditional Italian menu structure complete with popular item photos, header image, and a logo. And lastly, Big Star has a very simple menu showing two main categories for food. You got the classics and the tacos and you have a consistent photo layout.</p>
<h4 id="andy-fang%3A-10" tabindex="-1">Andy Fang:</h4>
<p>Another thing to consider is just how customizable these items are, what kind of modifiers are available? While some merchants provide a running list of options available for each item such as Pho 88, other menus keep the option list short to limit the online menu and to simplify preparations. To find out which menu features were the most influential in menu conversion, we use the features mentioned above as inputs to regression models predicting menu conversion. We built our initial regression models using linear regression and base tree models to achieve a baseline error, while the results there were interpretable the error rate was pretty high. And also on top of that many of the features seem to be correlated which led to an issue of collinearity, which made it difficult to determine how changes in each feature impact the target variable directionality.</p>
<h4 id="andy-fang%3A-11" tabindex="-1">Andy Fang:</h4>
<p>The lack of being able to explain this clearly was a pain point for us and is generally a pain point for blackbox models in general. To solve this problem we use Shapley values, which is a game theoretical approach towards model explainability. Shapley values represent the marginal contribution of each feature to the target variable and are calculated by computing the average marginal contribution to the prediction across all permutations before and after withholding that feature. So after examining the resulting Shapley plots of the final model, as you can see here, the top success factor was the number of photos on the menu. And this is particularly important for the top items in the menu as photo coverage of the top items appears much more prominently in the menus overall appearance. Some other top factors and recommendations we made to merchants to help make their menus perform better, one of which is giving higher customizability for items. We found that customers enjoy optionality within the top items and ability to customize provided a degree of familiarity that they could find while dining in.</p>
<h4 id="andy-fang%3A-12" tabindex="-1">Andy Fang:</h4>
<p>Another factor was menus with a healthy mix of appetizers and sides also converted better. This provides customers with more choices to complete their meal and can lead to higher carb values for merchants as well. While top factors that lead to successful online menus aren’t surprising, we also have to know that there are variances by cuisine type. This variance is a classic case of where the averages can be deceiving since the average doesn’t really represent a normal distribution. And also menus are not all the same. By better understanding customer’s expectations around certain types of restaurants and cuisines, merchants are able to better build and customize their menu to basically be more truthful to their brand and the food that they’re serving.</p>
<h4 id="andy-fang%3A-13" tabindex="-1">Andy Fang:</h4>
<p>The clearest example of customer expectations at work was actually what we observed with Chinese menus. Unlike most other menus, Chinese menus that were long and had a ton of items actually performed better. When customers dine in at Chinese restaurants with a long and complicated menu, sometimes you even have regional specialties, indicating authenticity. Take a look at this menu. Customer expectations about an authentic Chinese menu leads to these complicated menus having greater conversion rates. On the other hand, menus for wings and pizza places tended to be shorter and well photographed and also have a ton of options for customizing. You can think about it where when customers visit these types of merchants they usually have one to two items in mind or are just looking for things to customize like sauces for wings or toppings for pizzas.</p>
<h4 id="andy-fang%3A-14" tabindex="-1">Andy Fang:</h4>
<p>To wrap it up. These are just some of the ways we’re using AI to scale our marketplace and to make it easier for customers to find what they want, and also make it easier for merchants to position themselves in an online world. If you’re curious to learn more about these two particular case studies, or you’re curious to read about more case studies, feel free to check out our DoorDash engineering blog. It’s just DoorDash.engineering. It was a pleasure to speak with all of you today. I really hope you enjoyed and took something out of it and enjoy the rest of the conference.</p>


Andy Fang, CTO & Co-Founder of DoorDash discusses how DoorDash leverages artificial intelligence and machine learning to power their marketplace. 

Applied AI/ML at DoorDash with Andy Fang

Andy Fang

Andy Fang is the Co-founder and Chief Technical Officer of DoorDash, a technology company passionate about transforming local businesses and dedicated to enabling new ways of working, earning, and living. Andy leads DoorDash’s Consumer Engineering team where he is focused on growing and retaining DoorDash’s customer user base while increasing engagement for DoorDash’s marketplace products. In this role, Andy is responsible for the overall product vision, technology roadmap and architectural direction of the DoorDash consumer platform. Andy holds a BS in Computer Science from Stanford University, where he met fellow cofounders Tony Xu and Stanley Tang whereupon the concept for DoorDash was born.

Scale Transform 2021 | Applied AI/ML at DoorDash with Andy Fang

Sam Altman, Co-Founder, and CEO of Open AI joins Scale AI CEO Alexandr Wang to discuss Open AI’s latest research from DALLE to GPT-3 and discuss AI research broadly.

The Future of AI Research with Sam Altman of Open AI

Sam Altman

Sam Altman is the co-founder and CEO of OpenAI. Sam is also the Chairman of Y Combinator, a leading silicon valley startup accelerator.

Scale Transform 2021 | The Future of AI Research with Sam Altman of Open AI

James Chen, Chief Technology Officer of Flexport joins Scale AI CTO, Brad Porter, to discuss leveraging machine learning to modernize global trade.

Modernizing Global Trade with James Chen of Flexport

James Chen is responsible for driving Flexport’s product vision and technology strategy. He was most recently a Director of Technology at Amazon, and prior was a CTO at both Jenzabar and Rakuten. James holds a BS degree in Computer Science and Electrical Engineering from MIT.

James Chen

Brad Porter_Transform

Brad is the CTO of Scale AI. Prior to Scale, Brad was the VP of Robotics and a Distinguished Engineer at Amazon where he was responsible for driving improvements in the safety and autonomy of Amazon’s global operations. His experiences at Amazon as well as Tellme Networks has given him a deep understanding of AI’s engineering challenges as well as opportunities. 

Brad Porter

Scale Transform 2021 | Modernizing Global Trade with James Chen

The panel discusses how advancements in AI technologies like document understanding are increasing efficiencies, lowering costs, and disrupting traditional processes in financial services.

Panel: Transforming FinTech with AI

Bridget Frey

As Redfin’s Chief Technology Officer, Bridget leads the software engineering team. Her mission is to build technology that makes the process of buying and selling a home less complicated and less stressful. She is a leader on issues facing traditionally underrepresented people in technology, and 36% of Redfin’s technology team are women while 10% are Black or Latinx. Prior to Redfin, Frey was the director of analytics and business applications at Lithium Technologies. In addition, she has held management positions at IntrinsiQ Research, IMlogic and Plumtree Software. Since 2019, she has served on the board of directors for Premera Blue Cross. Bridget holds a bachelor's degree in computer science from Harvard University, where she graduated.

Nima Ghmasari

Nima Ghamsari is CEO and co-founder of Blend, a Silicon Valley technology company propelling consumer lending into the digital age. He leads the company's corporate and product strategy toward the vision of a more frictionless, compliant, and accessible consumer finance ecosystem. Nima is currently focused on expanding the Blend platform both in breadth and depth, bringing seamless processes to more consumers across more of their financial lives. Prior to founding Blend, Nima was one of the first employees of the commercial group at Palantir Technologies. He is a graduate of Stanford University where he earned a B.S. in computer science.

Nima Ghamsari

Henrique Dubugras

Henrique Dubugras is Co-Founder & Co-CEO of Brex — a company reimagining financial systems so every growing company can realize their full potential and take control of their spend and business as they scale. A Brazilian entrepreneur, Henrique built payments company Pagar.me — the Stripe of Brazil — when he was sixteen years old, with his co-founder Pedro Franceschi. In just three years, Pagar.me grew to $1.5 billion in volume of transactions processed. In the fall of 2016, Henrique and Pedro sold Pagar.me and enrolled at Stanford University. Henrique and Pedro founded Brex in 2017 to give small businesses access to the financial tools and services they need to grow and succeed.

Melisa Tokmak

Melisa leads the Document Processing business unit at Scale that focuses on building fine-tuned ML models with human-in-the-loop annotation to automate handling of documents at a high quality. Previously, she was the Chief of Staff for the company and started the Government business unit. Before Scale, Melisa used to build products for new product monetization at Facebook. She graduated from Stanford University with a degree in Computer Science. Melisa is from a small town in Turkey.

Scale Transform 2021 | Panel: Transforming FinTech with AI

Danielle Dean, Technical Director of ML at iRobot, Hussein Mehanna, Head of AI at Cruise, and Bart Nabbe, VP of Corporate Development at Aurora discuss the lessons learned scaling ML systems.

Panel: Lessons Learned Scaling ML Systems

Danielle Dean

Danielle Dean, PhD is the Technical Director of Machine Learning at iRobot where she is helping lead the intelligence revolution for robots. She leads a team that leverages machine learning, reinforcement learning, and software engineering to build algorithms that will result in massive improvements in our robots. Before iRobot, Danielle was a Principal Data Scientist Lead at Microsoft within the Cloud AI Platform division. There, she led an international team of data scientists and engineers to build predictive analytics and machine learning solutions with external companies utilizing Microsoft's Cloud AI Platform. Before working at Microsoft, Danielle was a data scientist at Nokia, where she produced business value and insights from big data. Danielle completed her Ph.D. in quantitative psychology with a concentration in biostatistics at the University of North Carolina at Chapel Hill.

Russell_Kaplan_Headshot

Russell Kaplan leads Scale Nucleus, Scale's Dataset IDE for machine learning engineers. He was previously founder and CEO of Helia AI, a computer vision startup for real-time video understanding, which Scale acquired in 2020. Before that, Russell was a senior machine learning scientist on Tesla's Autopilot team, and he received his M.S. and B.S. from Stanford University, where he was a researcher in the Stanford Vision Lab advised by Fei-Fei Li.

Russell Kaplan

Hussein Mehanna

Hussein Mehanna is the Head of Artificial Intelligence at Cruise. He is an expert in AI with a passion for machine learning. He has over 15 years of experience and has successfully built and led AI teams at multiple Fortune 500 companies.

Prior to Cruise, Hussein led the Cloud AI Platform organization at Google. Under his leadership, his team revamped the product line and rebuilt the organization. Cloud AI Platform became Cloud AI's fastest growing product segment. Before Google, Hussein worked at Facebook where he co-founded the Applied Machine Learning group that combined applied research in machine learning and advanced platforms. He helped democratize artificial intelligence with more than 2000 engineers using the technologies, and his team added billions of dollars of revenue.

Hussein has a Masters in Computer Speech, Text and Internet Technology from the University of Cambridge and a Bachelors of Science in Computer Science from Ain Shams University.

Bart Nabbe

Bart Nabbe is the VP of the Strategic Partnerships and Corporate Development team at Aurora, where he handles all of the company’s major corporate partnerships. Prior to joining Aurora in 2017, he was the Director of Strategic Partnerships at Faraday Future in 2016 and was a founding member of Apple’s Autonomous Systems team from 2014 to 2016. He was a Research Scientist at Toyota, Tandent Vision Science, and Intel Research. He has a B.S. in Electrical Engineering from Venlo College of Engineering and a Masters in Computer Science from the University of Amsterdam. He earned his Ph.D. in Robotics from Carnegie Mellon in 2005 where he met and worked on mobile robotics with Chris Urmson and Drew Bagnell, who are co-founders at Aurora.

Scale Transform 2021 | Panel: Lessons Learned Scaling ML Systems

Industry leaders share their perspectives on the state of NLP research and applications, and discuss where NLP goes from here. 

Panel: Breakthroughs in NLP and Future Potential

Catherine Williams

As the Global Head of iQ at Qualtrics, an experience management software platform, Dr. Catherine Williams leads the engineering and applied science teams that build advanced intelligent features into the Qualtrics experience management products and platform, leveraging cutting edge text analytics, predictive intelligence, and statistical analysis to help customers better understand and act on their data in real time. Catherine has extensive background and expertise in data science and beyond; prior to her work at Qualtrics, Catherine led data science, analytics, and product organizations at AppNexus and Xandr (part of AT&T) as Chief Data Scientist and Chief Data and Marketplace Officer. She received her BA from Grinnell College and her PhD in Mathematics at University of Washington and held postdoctoral fellowships at Stanford and Columbia Universities. Outside of work, Catherine enjoys exploring the outdoors, running, and spending quality time with her two boys.

Julien Chaumond

Julien Chaumond is Chief Technical Officer at Hugging Face, a Brooklyn and Paris-based startup working on Machine learning and Natural Language Processing.

Hugging Face has been described as the most influential platform in modern Machine learning. As a co-founder, Julien is passionate about democratizing state-of-the-art NLP and ML for everyone.

After graduating from Ecole Polytechnique and Stanford University, he has been a founder or team member on several Machine-learning based startups. He was also an advisor to the French Minister for Digital Affairs, where he managed to get the French laws published in Git!

Aatish Nayak

Aatish is the Head of Content & Language products including NLP, Speech, Cataloguing, Classification, and Search Relevance. The team focuses on empowering customers in social media, e-commerce, and broader enterprise to get diverse human insight on content quickly and fairly. Aatish previously was an early engineer at robotics startup Shield AI, focused on building AI systems to protect service members, and Skurt, an on-demand car rental marketplace acquired by Fair.com. In college, Aatish ran Autolab.com, a learning management startup used at CMU, Cornell, Rutgers, NYU, PKU, and others. He graduated with a B.S. in Computer Engineering from Carnegie Mellon University.

Scale Transform 2021 | Panel: Breakthroughs in NLP and Future Potential

<h4 id="alexandr-wang" tabindex="-1">Alexandr Wang</h4>
<p>Thank you everyone for joining us today at our inaugural Scale Transform Conference. We have an incredible group of speakers who are going to speak to the opportunities and challenges for the future of AI. I couldn’t be more thankful for everyone who’s helped make this conference possible, and I hope you learn something new through our conversations today. As we reflect on the future of AI, the macro inputs are clear. We believe the next decade of machine learning will be defined by continued exponential scaling of compute, data, and algorithms. A consistent finding in machine learning research is that performance continues scaling seemingly infinitely with larger data sets, more compute, and larger models. Model performance increases logarithmically as a function of compute and dataset size. We can expect to see exponential growth in the amount of compute and data used to train state-of-the-art machine learning models this decade.</p>
<p>Industry is tackling the challenge of increasing compute needs head-on. There’s a boom of AI chip development, and experiments can be scaled up to massive compute sizes on public clouds. The recent Nvidia A100 chip is a strong example of this trend. Another interesting trend here is the parallels between new deep learning chips and the human brain. The new chips by Cerebras and Graphcore have interspersed compute and memory, which is more reflective of how the human brain actually works. It’s exciting to see hardware trends approach biological observations, as we continue to optimize machine learning workloads.</p>
<p>This leaves the creation of massive scale high-quality datasets as the largest remaining bottleneck. At Scale, our goal is to actualize Moore’s Law for annotation, so that data can grow in tandem with exponentially growing compute and model sizes. Recent research from OpenAI has shown how data set size model size and compute must be scaled in tandem for optimal performance. Through this research, we can actually predict exactly how much data we’ll need in the future as a function of compute and model sizes that are available. The short answer is, it’s a lot. There’s an incredible need and necessity for us as a community to figure out how to continue scaling our machine learning systems with larger data sets, and more efficient methods of annotation and human feedback. We’re excited to tackle these problems at Scale to ensure that we can all continue riding these exponential curves and scaling machine learning systems.</p>
<p>In machine learning, the evidence is clear: quantity has a quality of its own. We’re observing qualitatively new behaviors from similar algorithmic methods, simply by increasing the scale of compute and data. There’s a number of examples of this. It all started in the early 2012 experiments where neural networks finally started working once data finally scaled with the advent of the image net dataset. Later on today, we’ll be speaking with Fei-Fei Li on her seminal work in starting image net. GPD 3 wowed the world last year, where it demonstrated incredible results in few shot learning, generalization, and an impressive ability to distill human knowledge through text.</p>
<p>More recently, additional research from OpenAI has shown multimodal neurons, AKA Halle Berry neurons, emerging in clip training on 300 million image language pairs. These neurons show an ability for neural networks to learn complex concepts jointly in multiple domains, such as image and language. We’ll be speaking with Sam Altman of OpenAI later today on their breakthrough results. Lastly, Gabor-like filters emerging naturally from stochastic gradient descent in the early layers of convolutional neural networks, mirroring evidence we have of how the visual cortex works in mammalian brains from experiments as early as Hubel and Wiesel.</p>
<p>Overall, AI continues to surprise us with its new capabilities, primarily enabled through advancements in the core resources of data, compute, and algorithms. What types of new AI capabilities can we expect to observe when we reach the next orders of magnitude for data and compute? We may not be sure what these new emergent behaviors will be, but they will definitely exist, and they will be enabled by the next decade of data and compute scaling. Thank you everyone. I’m excited for our speakers to share their insights from the front lines of AI, and I hope you all enjoy the conference.</p>


Join Alex Wang, CEO and Founder of Scale AI, to hear about his vision for what the future of AI has in store for us.

Opening Remarks 

Alex is the CEO and Founder at Scale AI. He was inspired to solve ML infrastructure problems and accelerate the development of AI through his work at Quora, where he worked as a technical lead. Alex worked as an algorithm developer at Hudson River Trading and as a software engineer at Addepar. He attended, and dropped out from, MIT, studying Artificial Intelligence.

Opening Remarks with Alexandr Wang, CEO and Founder of Scale AI

Scale Transform 2021 | Opening Remarks

<h4 id="brad-porter%3A" tabindex="-1">Brad Porter:</h4>
<p>Thank you, Alex. And thank you Fei-Fei. What an incredible conversation to close out the day. Before we wrap up and go into the networking and social events. I want to thank all our speakers for their time and their insights. I know, I for one I’m incredibly enthusiastic about the AI community that’s emerged and incredibly honored that our speakers graciously volunteered their time to share their insights and that you all took your time to attend our conference today.</p>
<p>I also wanted to take a moment to reinforce a couple of the key themes I’ve picked up on throughout the day today. The first is the critical importance of data. Our industry recognizes that more and more data and computer can logarithmically improve model performance. But as Dr. Andrew Ng discussed in his thought, it’s not just the volume of data that matters, but the quality of that data. For some use cases, we may have a lot of data and in others, we may not have much to work with, but bigger or small, we need to improve the quality of our datasets to ensure model accuracy.</p>
<p>And while the model work often takes center stage in ML development, your models are ultimately learning the intrinsics of your data. I hope all of us take Andrew’s words to heart and champion the importance of data quality to unlock the full potential of machine learning. The second theme that came through loud and clear was the critical importance of making our field more diverse and more inclusive. Kevin Scott spoke earlier in the day about making the field more accessible to ensure the benefits of AI and machine learning are evenly distributed. Just now Fei-Fei Li spoke about the unintended consequences of AI.</p>
<p>The example she gave was a healthcare model developed up a dataset, primarily collected in coastal states without regional diversity in the dataset, such a model may unintentionally misdiagnose patients in other regions of the US. This ultimately serves an important reminder to all of us that how the datasets are collected matters, how datasets are labeled matters, making AI accessible matters, fostering diversity in our fields is even more important because of the potential of AI and machine learning.</p>
<p>I hope all the insights shared by our speakers today leaves you as hopeful for the future of our industry as I am. If you missed any of the sessions, video recordings of all sessions will be made available on the Scale website. Thank you again to all of our speakers and thank you all for joining us for our inaugural Scale Transform Conference.</p>
<p>If you enjoyed the sessions today, I’m really excited to announce that we’ll be hosting another virtual conference this fall. For more information on our next conference, please follow us on Twitter, LinkedIn, and Facebook. With that, we encourage you all to connect in our virtual networking platform. The link is located in the conference chat to the right of your screen. Thank you again and have a great weekend.</p>


Closing Remarks with Brad Porter, Chief Technology Officer at Scale AI

Closing Remarks 

Brad is the CTO of Scale AI. Prior to Scale, Brad was the VP of Robotics and a Distinguished Engineer at Amazon, where he oversaw both driving improvements in the safety and autonomy of Amazon’s global operations and some of Amazon’s most ambitious new initiatives including the Scout sidewalk bot program and Prime Air drone delivery. His experiences at Amazon as well as Tellme Networks, where Brad was the platform architect and first engineer, has given him a deep understanding of AI’s engineering challenges as well as opportunities.

Brad Porter 

Scale Transform 2021 | Closing Remarks

Drago Anguelov discusses the state of perception and self-driving, future research, and broader trends in the autonomous vehicle industry.

Taking Autonomous Driving from Research to Reality with Drago Anguelov

Drago_Anguelov

Drago joined Waymo in 2018 to lead the Research team, which focuses on pushing the state of the art in autonomous driving using machine learning. Earlier in his career he spent eight years at Google; first working on 3D vision and pose estimation for StreetView, and later leading a research team which developed computer vision systems for annotating Google Photos. The team also invented popular methods such as the Inception neural network architecture, and the SSD detector, which helped win the Imagenet 2014 Classification and Detection challenges. Prior to joining Waymo, Drago led the 3D Perception team at Zoox.

Drago Anguelov

Drago Anguelov, Distinguished Scientist and Head of Research at Waymo joins Scale AI CEO Alexandr Wang in a fireside chat. The two discuss the state of perception and self-driving, future research, and broader trends in the autonomous vehicle industry.

Scale Transform 2021 | Taking Autonomous Driving from Research to Reality with Drago Anguelov

Francois Chollet discusses the latest developments in the Keras & TensorFlow ecosystem and how Google is preparing for the next generation of deep learning research and applications.

The Next Five Years of Keras & Tensor Flow with Francois Chollet

François Chollet

François Chollet is a software engineer at Google, where he leads the Keras team. He is the author of a popular textbook on deep learning. He also does research on abstraction, reasoning, and how to achieve greater generality in artificial intelligence.

Scale Transform 2021 | The Next Five Years of Keras & Tensor Flow with Francois Chollet

Jeremy Howard and Rachel Thomas sit down for a fireside chat explaining why they started fast.ai, how it progressed from classes to a software platform, the importance of community, and where they see the future direction of fast.ai.

fast.ai: The Why, How, and Future of Democratizing ML

Jeremy_Howard

Jeremy Howard is a data scientist, researcher, developer, educator, and entrepreneur. Jeremy is a founding researcher at fast.ai, a research institute dedicated to making deep learning more accessible. He is also a Distinguished Research Scientist at the University of San Francisco, the chair of WAMRI, and is Chief Scientist at platform.ai. Previously, Jeremy was the founding CEO Enlitic, which was the first company to apply deep learning to medicine, and was selected as one of the world’s top 50 smartest companies by MIT Tech Review two years running. He was the President and Chief Scientist of the data science platform Kaggle, where he was the top ranked participant in international machine learning competitions 2 years running. He was the founding CEO of two successful Australian startups (FastMail, and Optimal Decisions Group–purchased by Lexis-Nexis). Before that, he spent 8 years in management consulting, at McKinsey & Co, and AT Kearney. Jeremy has invested in, mentored, and advised many startups, and contributed to many open source projects.

He has many media appearances, including writing for the Guardian, USA Today, and the Washington Post, appearing on ABC (Good Morning America), MSNBC (Joy Reid), CNN, Fox News, BBC, and was a regular guest on Australia’s highest-rated breakfast news program. His talk on TED.com, “The wonderful and terrifying implications of computers that can learn”, has over 2.5 million views. He is a co-founder of the global Masks4All movement.

Jeremy Howard

Rachel Thomas

Rachel Thomas is director of the USF Center for Applied Data Ethics and co-founder of fast.ai, where she helped create the most popular free online course on deep learning, bringing people from around the world with diverse and nontraditional backgrounds into AI. Rachel earned her PhD in mathematics at Duke University and previously worked as a data scientist and software engineer. She was selected by Forbes as one of 20 Incredible Women in AI and was profiled in the book Women Tech Founders on the Rise. She wrote chapters for the books 97 Things About Ethics Everyone in Data Science Should Know and Deep Learning for Coders with fastai and PyTorch.

Rachel’s writing has been read by nearly a million people; has been translated into Chinese, Spanish, Korean, & Portuguese; and has made the front page of Hacker News 9x. Some of her most popular articles include:

- [The problem with metrics is a big problem for AI](https://www.fast.ai/2019/09/24/metrics/)
- [If you think women in tech is just a pipeline issue, you haven’t been paying attention](https://medium.com/tech-diversity-files/if-you-think-women-in-tech-is-just-a-pipeline-problem-you-haven-t-been-paying-attention-cb7a2073b996)
- [The real reason women quit tech, and how to address it](https://medium.com/tech-diversity-files/the-real-reason-women-quit-tech-and-how-to-address-it-6dfb606929fd)
- [Google’s AutoML: Cutting Through the Hype](https://www.fast.ai/2018/07/23/auto-ml-3/)
- [An Introduction to Deep Learning for Tabular Data](https://www.fast.ai/2018/04/29/categorical-embeddings/)

Rachel’s talks include:

- [AI, Medicine, and Bias: Diversifying Your Dataset is Not Enough](https://www.youtube.com/watch?v=vVRWeGlMkGk&list=PLtmWHNX-gukLQlMvtRJ19s7-8MrnRV6h6&index=3) (Stanford AI in Medicine & Imaging Symposium)
- [Getting Specific About Algorithmic Bias](https://www.youtube.com/watch?v=S-6YGPrmtYc&list=PLtmWHNX-gukLQlMvtRJ19s7-8MrnRV6h6&index=5) (featured talk at PyBay)
- [The Barriers to AI are Lower than You Think](https://www.technologyreview.com/video/610690/accessible-ai-expanding-the-reach-of-intelligent-solutions/) (MIT Technology Review conference)
- [The New Era in NLP](https://www.youtube.com/watch?v=KChtdexd5Jo&list=PLtmWHNX-gukLQlMvtRJ19s7-8MrnRV6h6&index=5&t=0s) (keynote at SciPy)
- [Some Healthy Principles About Ethics & Bias In AI](https://www.youtube.com/watch?v=WC1kPtG8Iz8&list=PLtmWHNX-gukLQlMvtRJ19s7-8MrnRV6h6) (keynote at PyBay)

Scale Transform 2021 | fast.ai: The Why, How, and Future of Democratizing ML

Dr. Hsieh discusses how we should assess a company’s AI/ML maturity from zero to five from both a top-down (executive’s) perspective and a bottom-up (scientist’s) perspective.

A Framework to Assess Your AI/ML Maturity with Chu-Cheng Hsieh of Etsy

Chu-Cheng Hsieh

Chu-Cheng manages the data org across Etsy globally, including engineering, data science, and machine learning. Partnering with Etsy’s product and business executives, he develops the data strategy, represents data science, and drives high-impact decisions. He is specialised in search engine, recommendation systems, and machine learning technology. His primary responsibility is to deliver strategic and creative data science approaches that help achieve Etsy’s mission and goals.
Chu-Cheng received PhD in computer science from UCLA, and has two master degrees. In his leisure time, he enjoys innovating and collaborating with academic researchers. He has brought cutting-edge research into products. He publishes papers in top-tier conferences, such as WWW, SIGIR, KDD, and enjoys giving talks/keynotes at a variety of academic or industrial conferences on information retrieval, recommendation systems, and data mining.

Scale Transform 2021 | A Framework to Assess Your AI/ML Maturity with Chu-Cheng Hsieh of Etsy

Kevin Scott, CTO of Microsoft, discusses recent drivers of progress for AI/ML, macro trends shaping the future, and how Microsoft is approaching the challenge of ethically and equitably scaling the benefits of AI.

Democratizing the Benefits of AI with Kevin Scott of Microsoft

Kevin Scott

Kevin Scott is executive vice president of Technology & Research, and the chief technology officer of Microsoft. He is an innovative leader driving the technical vision to achieve Microsoft’s mission, and is passionate about creating technologies that benefit everyone. He focuses on helping make the company an exceptional place for engineers, developers and researchers to work and learn. Scott’s 20-year career in technology spans both academia and industry as researcher, engineer and leader.

Prior to joining Microsoft, he was senior vice president of engineering and operations at LinkedIn, where he helped build the technology and engineering team and led the company through an IPO and six years of rapid growth.

Earlier in his career, he oversaw mobile ads engineering at Google, including the integration of Google’s $750 million acquisition of AdMob. At AdMob, Scott was responsible for engineering and operations for the world’s leading platform for mobile monetization. Before joining AdMob, Scott held numerous leadership positions at Google in search and ads engineering and helped with the company’s early efforts establishing remote engineering centers.

Scott is the host of the podcast Behind the Tech, which features interviews with technology heroes who have helped create the tech industry of today. He also authored the book “Reprogramming the American Dream”, which explores how artificial intelligence can be realistically used to serve the interests of everyone, not just the privileged few.

As co-inventor on several patents around search and information extraction, he has also authored several publications on dynamic binary rewriting that collectively have been cited hundreds of times in other scholarly research.

He has received a Google Founder’s Award, an Intel Ph.D. Fellowship and an ACM Recognition of Service Award.

He is an adviser to several Silicon Valley startups, an active angel investor, the founder of the non-profit organization Behind the Tech, a member of the Anita Borg Institute’s board of trustees and a trustee of The Scott Foundation. He also serves on the Advisory Council for Stanford University’s Institute for Human-Centered Artificial Intelligence (Stanford HAI) and the Leadership Council for Harvard’s Technology for Public Purpose (TAPP) program.

Scott holds an M.S. in computer science from Wake Forest University, a B.S. in computer science from Lynchburg College, and has completed most of his Ph.D. in computer science at the University of Virginia.

Scale Transform 2021 | Democratizing the Benefits of AI with Kevin Scott of Microsoft

Srinivas discusses how AI is used at large scale in Facebook’s products and shares some of the recent advancements in AI and how Facebook is deploying them into production quickly. 

AI at Facebook Scale with Srinivas Narayanan

Srinivas Narayanan

Srinivas Narayanan leads the Applied Research team at Facebook AI doing research and development in a wide range of areas such as computer vision, natural language, speech, and personalisation to push the state of the art in AI to advance Facebook products. He’s led several major efforts at Facebook, including creating the interest graph, launching the location product, and leading engineering for photos, where he also helped start Facebook’s efforts in computer vision and deep learning. Previously, he was a founding member of two startups and part of the database systems research group at IBM Research – Almaden.

Scale Transform 2021 | AI at Facebook Scale with Srinivas Narayanan

Raquel Urtasun discusses the remaining technological challenges and showcases the latest advancements in the field of autonomous vehicles.

A Future with Self-Driving Vehicles with Raquel Urtasun

Raquel_Urtasun

Raquel Urtasun is the Chief Scientist for Uber ATG and the Head of Uber ATG Toronto. She is also a Professor at the University of Toronto, a Canada Research Chair in Machine Learning and Computer Vision and a co-founder of the Vector Institute for AI. She received her Ph.D. from the Ecole Polytechnique Federal de Lausanne (EPFL) in 2006 and did her postdoc at MIT and UC Berkeley. She is a recipient of an NSERC EWR Steacie Award, an NVIDIA Pioneers of AI Award, a Ministry of Education and Innovation Early Researcher Award, three Google Faculty Research Awards, an Amazon Faculty Research Award, a Connaught New Researcher Award, a Fallona Family Research Award and two Best Paper Runner up Prize awarded CVPR in 2013 and 2017. She was also named Chatelaine 2018 Woman of the year, and 2018 Toronto's top influencers by Adweek magazine.

Present & Future of AI

Watch all the talks

From Big Data to Good Data with Andrew Ng

Future of ML Frameworks with Soumith Chintala

Human-Centered AI with Fei-Fei Li

Applied AI/ML at DoorDash with Andy Fang

The Future of AI Research with Sam Altman of Open AI

Modernizing Global Trade with James Chen of Flexport

Panel: Transforming FinTech with AI

Panel: Lessons Learned Scaling ML Systems

Panel: Breakthroughs in NLP and Future Potential

Opening Remarks

Closing Remarks

Taking Autonomous Driving from Research to Reality with Drago Anguelov

The Next Five Years of Keras & Tensor Flow with Francois Chollet

fast.ai: The Why, How, and Future of Democratizing ML

A Framework to Assess Your AI/ML Maturity with Chu-Cheng Hsieh of Etsy

Democratizing the Benefits of AI with Kevin Scott of Microsoft

AI at Facebook Scale with Srinivas Narayanan

A Future with Self-Driving Vehicles with Raquel Urtasun

Sam Altman

Kevin Scott

Fei-Fei Li

Andrew Ng

Drago Anguelov

Julien Chaumond

James Chen

Soumith Chintala

François Chollet

Danielle Dean

Henrique Dubugras

Andy Fang

Bridget Frey

Nima Ghamsari

Jeremy Howard

Chu-Cheng Hsieh

Russell Kaplan

Hussein Mehanna

Bart Nabbe

Srinivas Narayanan

Aatish Nayak

Brad Porter

Rachel Thomas

Melisa Tokmak

Raquel Urtasun

Alexandr Wang

Catherine Williams