experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Customers

Product

Company

General

People

Engineering

<h1>Q&amp;A with Scale's Synthetic team</h1><h1>Why did you join Scale?</h1>Vivek: Choosing a new role is a consequential and high-impact decision. I have been following the fantastic work by the Scale team for a few years now. I was intrigued by their ambition and impact in a relatively short window. Eventually, my brain caught up to my gut. Once I spoke to the team at Scale, joining turned out to be a relatively simple decision.Joel: I have spent the last decade working at the intersection of computer graphics and computer vision. I have never been more excited about the future. Never before have so many profound technological breakthroughs converged at the same time - unprecedented compute power, large amounts of visual data, powerful generative models, and scalable AI infrastructure. Together, these advancements enable many new AI applications that we could only dream of in the past. I am incredibly excited to join Scale to help accelerate that future.<h1>What aspects of AI and ML are you excited to work on at Scale?</h1>Vivek: We are still very early in the technology arc of transitioning from traditional software systems to AI. Building infrastructure to get there quickly and responsibly has fueled my passion for several years. I believe Scale will play a significant role in shaping that future.Although we have seen significant investment in model architectures over the past decade, we need a fundamental shift in data practices to unleash AI’s full power. Model code and data form the foundation of an AI system. Scale’s leading end-to-end solutions for the ML lifecycle based on real-world data will continue to set the bar for the data-centric movement.Joel: When I completed my PhD in 2015, the deep learning revolution was just getting going. My thesis focused on physically based rendering of synthetic objects in real environments. Fast forward to 2022, and deep learning has fundamentally changed how most visual computing problems are approached.The practical success of these models is often attributable to their capacity to scale to large amounts of data - a fact that I came to appreciate first hand during my time at Apple.<h1>What are the biggest challenges to implementing synthetic data?</h1>Vivek: The physical world continues to be a limitation for data collection. Enter: digital twins. Digital twins are virtual clones that recreate the real world. This synthetic world will give developers access to unlimited data quickly. But, more importantly, it will take us a step further in addressing the most pressing ethical issues in the AI world today - privacy, data bias, and the democratization of data. Scale Synthetic is our next addition towards accelerating the AI future.Joel: Most recently as Head of Machine Learning at Nines, I came to appreciate that often the fastest way to improve the performance of modern ML models is not to tweak the model to work better for less data, but to focus one’s efforts on understanding, collecting and labeling more high quality data.This insight also presents a challenge to applications where data is scarce, or difficult to collect or label. Instances of this include when there are privacy considerations, dataset bias, or rare events. Hence, the need for synthetic data.<h1>In your view, what sets Scale apart from other AI and ML companies?</h1>Vivek: Scale’s credos resonate with my core beliefs as tenets for successful teams.Two credos that stand out for me are “ambition shapes reality” and “results speak loudest”. Raising the bar, and regularly and objectively measuring progress, is foundational to great products. Once I spent time with the team, it was evident that the credos were not symbolic, but formed the day-to-day backbone.Building, and scaling a culture alongside a product is tough. However, reading Alex and Brad’s posts gave me confidence that I am joining a company where the founder and exec team cares immensely about culture. Solid culture is a necessary condition to build a generational company–one which I hope to contribute to and shape with the team.Joel: The data-centric approach is also what got me excited about Scale. Scale enables any ML developer to build better AI models, faster, by building solutions to manage the entire ML lifecycle. Scale started with high quality data annotation, and is now a clear market leader. Scale has since continued to innovate and has built an impressive suite of leading products around AI operations and infrastructure

Scale AI Welcomes New Synthetic Team Leadership

Why we joined Scale Synthetic

Today, we are thrilled to announce the Early Access Availability of <a href="https://scale.com/synthetic" rel="noopener noreferrer" target="_blank">Scale Synthetic</a>, the most efficient way to enhance ML performance with data that complements real-world datasets. With Scale Synthetic, ML teams can combine the benefits of real-world data with diverse and realistic synthetic data to achieve desired data distributions. Real-world data remains indispensable to developing performant models. ML teams, however, have encountered several challenges with relying on real-world data alone. Data privacy issues can prevent collected data from being used, collected datasets also inherit the biases of conditions under which data was collected, and collecting enough examples of edge cases and rare scenarios can be both costly and time-consuming. With Scale Synthetic, ML teams can: <ol><li> </li><li>Overcome the challenges of real-world datasets while maximizing the return on investment (ROI) of real-world data by realistically augmenting real-world data or generating fully synthetic data from scratch.</li><li> </li><li> </li><li>Bridge the sim-to-real reality gap by leveraging Scale’s global, human-in-the-loop network that ensures objects in augmented or fully synthetic scenes are realistically placed, and follow a logical path.</li><li> </li><li> </li><li>Seamlessly integrate with Scale’s full ecosystem of products for near-zero friction to add augmented or fully synthetic data into existing ML data pipelines. </li><li> </li></ol>Scale Synthetic currently supports the augmentation or generation of image, video, or 3D point cloud data but custom requests are also encouraged. To date, Scale Synthetic is trusted by leading ML teams such as Kodiak Robotics as well as the U.S. Department of Defense to deliver high-quality synthetic data. Additionally, Scale Synthetic has been recognized in the 2022 Gartner® Innovation Insight for Synthetic Data* report as a Representative Provider in Sample Synthetic Data. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/13cf537b-0b89-4b55-885f-069505c8c900/public" alt="Kodiak Robotics Quote">To continue to enhance and build on the foundations of Scale Synthetic, we are also pleased to welcome Joel Kronander as the new Head of Synthetic Data and Vivek Raju Muppalla as the Director of Synthetic Services. Joel joins Scale from Nines, a startup that applied ML and AI expertise to the field of medicine where he served as the Head of Machine Learning. Vivek joins Scale from Unity Technologies, where he was the Director of Engineering, AI and Simulation.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/9281bfec-9ad9-49d8-d658-c16381bb0c00/public" alt="Joel Kronander and Vivek Muppalla Quotes">If you would like to get early access for Scale Synthetic, <a href="https://scale.com/synthetic" rel="noopener noreferrer" target="_blank">contact us today</a>. *Gartner, “Innovation Insight for Synthetic Data”, Arun Chandrasekaran, Alexander Linden, Anthony Mullen, Farhan Choudhary, February 7, 2022. Disclaimer- GARTNER is a registered trademark and service mark of Gartner, Inc. and/or its affiliates in the U.S. and internationally and is used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner’s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.

Scale Synthetic is the most efficient way to enhance ML performance with synthetic data that complements real-world datasets.

Complementing Real-World Datasets with Diverse and Realistic Synthetic Data

Introducing: Scale Synthetic - Now Available for Early Access 

<h1>Field Engineering at Scale</h1>Scale’s Field Engineering is a global team of creative engineers who collaborate with clients to understand their challenges and architect solutions. Our solutions are rooted in our technical skill, product expertise, and a thorough understanding of our customers’ complex needs. Whether the answer is a bespoke solution to solve a unique problem, or a new developer tool that can scale across Scale, everything we do aims to ensure our customers benefit from the full potential of Scale products and expertise. Our work is at the intersection of customer success, product, and engineering, helping Scale’s customers succeed and reach their business goals with Scale by unblocking them technically. Field Engineers are subject matter experts who know all of the products and technologies Scale has inside and out, and work closely with product managers to influence new product features. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2893973b-04ee-4e9e-9c7e-6304d5b38a00/public" alt="Field Engineering Map"><h1>A Day in the Life</h1> Field Engineers start each day with a team standup. During the standup, we align on daily priorities and check in on our longer-term projects. There’s almost always an urgent customer issue to work on, but when we have downtime, we tackle longer term projects that will help the company move faster in the future. We’ll go over some of these longer term projects further down. One of the more fun parts of our daily standup is ending with the “Word of the Day”. As half of our team is located in Buenos Aires, Argentina, each day someone will share a word that they would like to know how to say in English or Spanish and we’ll share our own take, the different definitions, and any fun stories that go along with it. After the standup, each day can look pretty different. Our days will often include at least a few customer meetings, where we will consult with our existing customers (think: the world’s leading ML companies) on technical aspects of their projects, any challenges they may be facing, and new initiatives coming up. Field Engineering is the customer’s technical contact to ensure all goes well with their work. Covid-permitting, we use lunchtime to socialize with colleagues in the office while we eat all the amazing food that is provided for us. It’s a nice time to decompress and connect. After customer meetings and any urgent issues have been addressed, we focus on our longer-term projects. We’re focused on making the lives of our customers, our internal partners, and ourselves as streamlined and scalable as we can. So, we spend significant time on internal tools, customer SDks, documentation and anything else we can think of to improve Scale’s developer experience. <h1>Field Engineering Initiatives</h1> <h2>Introducing a new and improved Scale Python SDK</h2> 2021 saw the introduction of a new version of our Python SDK. Rebuilt from the ground up to deliver new features faster, this new version includes a set of new ways to interact with the Scale API and improve infrastructure for a smoother developer experience. The tool was built on a new foundation that includes novel abstractions, so you can get more out of the Scale API with less effort. This new foundation allows us to add new functionality faster and better than ever before, so expect more new features from Scale to show up in the SDK with improved documentation and release notes. <h2>Shipping a Lidar Toolkit to make sense of multi-sensor datasets.</h2> The act of fusing multiple cameras, lidar sensors, and radars from a vehicle or robot into one cohesive scene that can be annotated is no easy feat. Historically, helping customers work through the nuances of their data and how to format it for Scale’s API-based workflows was one of the most time-consuming parts of our team’s day. In 2021 we overhauled our Lidar Toolkit and introduced a set of resources, code samples, and debugging tools to enable maximum efficiency for developers working with Lidar data. The toolkit, built on Open3d and Scale’s Python SDK, is full of helper functions that make working with the geometry, camera projections, and coordinate systems much easier to understand and implement than doing it from scratch. The Lidar Toolkit our team has published is now the de facto way all customers share and upload their data to Scale today. <h2>Embracing the Operational Nuances at Scale</h2> Scale is in the business of human-in-the-loop data annotation, and humans + loads of data means a lot of operational overhead comes with the territory. We don’t want to rely on running our business based on custom scripts day in and day out. So, we continually develop, iterate and ship internal tools to make the lives of our customer delivery team as seamless as possible. Some of the tools make facilitating customer requests easier, like moving tasks (tasks are what we call the unit of work we are annotating, like an image) between customer projects, re-sending callbacks to our customer’s endpoints, or canceling tasks en masse. Other sets of tools ensure high-quality data is coming out the other side of our task pipelines. A few years ago, had we wanted to programmatically identify potential annotation errors, a custom script would need to be written, tailored to that client’s specific projects. Over the past 18 months, Field Engineering has developed a framework that can be configured by our non-technical ops counterparts to flag any erroneous tasks in their pipelines. And, better yet, this framework can proactively prevent these tasks by flagging warnings directly in the labeling interface—before the annotations are even submitted. This has become one of our most powerful tools to ensure reasonable data is consistently delivered to our clients. <h1>Could Field Engineering be a fit for you?</h1> If you enjoy leveraging both sides of your brain, the people skills side, and the deeply technical one, Field Engineering may be a promising place to build and Scale your career. We are consistently hiring and growing the team (the team has doubled in size each year) and are interested in candidates with a variety of technical backgrounds and skills. Bonus points are given to candidates who possess a background in Computer Vision or API building and troubleshooting. If you're interested, <a href="https://scale.com/careers?location=All&amp;team=Engineering" rel="noopener noreferrer" target="_blank">explore our open positions today</a>!

A Day in the Life of a Field Engineer at Scale

Field Engineering at Scale

Field Engineering at Scale | A Day in the Life

In an exciting announcement to close out the year, Scale has been featured in two of Gartner, Inc.’s 2021 Hype Cycles. We are honored to have been featured in Hype Cycles for both Artificial Intelligence and Data Science and Machine Learning. These reports detail emerging technologies in AI, data science and machine learning, and name Scale as a representative vendor for Data Labeling and Annotation Services. The reports describe an urgency for data labeling and annotation services (DLA) to become widely adopted. According to Gartner analyst Anthony Mullen’s analysis, the demand for labeled and annotated data has risen in order to remove bottlenecks in industry-specific use cases. And, due to the fact that most companies do not have in-house skills and processes for DLA, companies like Scale are the best option to provide DLA to yield optimal AI results. Why good data matters Better data leads to more performant models, and performant models lead to faster deployment. With better data, Scale helps customers deliver more value. An increased diversity of use cases in DLA will drive both AI and industry-specific innovation, according to Gartner analyst Anthony Mullen. Vendors in the market have specific offerings across industries including commerce, robotics, autonomous vehicles, retail, GiS/maps, AR/VR, agriculture, finance, transportation and more. Scale is proud to accelerate innovation for customers including PayPal, iRobot, Square, Brex, Pinterest, Etsy, SAP, Airbnb, Instacart, NVIDIA, OpenAI, Samsung, Toyota, GM, the U.S. Air Force, and more. How does Scale solve the bad data problem? Scale is your AI readiness partner. We got our start in data labeling and annotation for autonomous vehicles. Now, Scale provides a data-centric, end-to-end solution to manage the entire ML lifecycle. That means that Scale can meet you wherever you are in your data process. ML-powered, human-in-the loop data Scale’s strength is the combined power of our ML platform and expert human insights to help our customers develop their data strategy with high quality data from the ground up. According to Mullen, there are AI solutions which would be impossible without human-in-the-loop labeling, including in cases of search engine tuning and image recognition. Furthermore, DLA allows data scientists to allocate and prioritize time for higher-order tasks, instead of taking time and effort to annotate data themselves. With Scale Rapid, any machine learning developer can achieve production-quality labels with no data minimums. Receive quality labels in just a matter of hours, with detailed insights and identified edge cases. Scale Rapid can even handle subjective tasks with the utmost precision and quality. But what comes after labeled data? Gartner Inc., has made the case for why data matters to AI development. After data labeling, what’s next? Data management. Whether you have prepared the data, or we label and annotate the data for you, we’re your premiere data management partner. Scale Nucleus is the command and control for your ML dataset. Visualize, curate and debug your dataset to improve your ML model performance with Nucleus. Scale’s advantage: AI development, from data labeling, through the entire ML lifecycle. Scale’s mission is to accelerate the development of AI. We do this by providing a data-centric, end-to-end solution to annotate, manage, automate, evaluate, collect and generate your data. Our products and services meet you wherever you are in the ML lifecycle, whether you need data labeled and annotated, or need an ML engineer to help explore and refine how AI can help your business.

Gartner Hype Cycles for AI and Data Science and Machine Learning

Scale highlighted in Gartner Inc.’s 2021 AI Hype Cycles

Scale Highlighted in Gartner Inc.’s 2021 AI Hype Cycles

Innovations in the field of Machine Learning (ML) continue to be driven by a dynamic research community. The ML team at Scale is proud to contribute to the field through its own research, dataset partnerships with leading Universities such as Oxford (<a href="https://scale.com/open-datasets/oxford" rel="noopener noreferrer" target="_blank">DEBAGREEMENT: Reddit 50K Dataset</a>) and the University of Waterloo and the University of Toronto (<a href="https://scale.com/open-datasets/oxford" rel="noopener noreferrer" target="_blank">CADC Dataset</a>), and collaborations with Stanford University, the University of California, Berkeley, and the University of Washington. As <a href="https://neurips.cc/" rel="noopener noreferrer" target="_blank">NeurIPS 2021</a>, gets underway, we are pleased to share details on three papers the ML team have published as part of the conference: <ol><li> </li><li>DEBAGREEMENT: A Comment-Reply Dataset for (Dis)agreement Detection in Online Debates</li><li>Authors: John Pougué-Biyong, Valentina Semenova, Alexandre Matton, Rachel Han, Aerin Kim, Renaud Lambiotte, Doyne Farmer</li><li> </li><li>Natural Adversarial Objects</li><li>Authors: Felix Lau, Sasha Harrison, Nishant Subramani, Aerin Kim, Elliot Branson, Rosanne Liu</li><li> </li><li> </li><li>Are We Learning Yet? A Meta-Review of Evaluation Failures Across Machine Learning</li><li>Authors: Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, Ludwig Schmidt</li><li> </li></ol><h2><a href="https://openreview.net/forum?id=udVUN__gFO" rel="noopener noreferrer" target="_blank">DEBAGREEMENT: A Comment-Reply Dataset for (Dis)agreement Detection in Online Debates</a></h2> <h3>Paper Summary:</h3> DEBAGREEMENT is a dataset including 42,894 comment-reply pairings from the popular debate website Reddit, each of which has been labeled with agree, neutral, or disagree. We gathered interactions from five different forums: r/BlackLivesMatter, r/Brexit, r/Climate, r/Democrats, and r/Republican. Comment pairings for each forum were chosen in such a way that they generate a user interaction graph when taken as a whole. DEBAGREEMENT offers a challenge for Natural Language Processing (NLP) systems because it includes slang, irony, and topic-specific humor, all of which are common in online discussions. We compared the performance of state-of-the-art language models on a (dis)agreement detection task, and looked at the usage of contextual information that is accessible to the models during training (graph, authorship, and temporal information). DEBAGREEMENT provides novel opportunities for combining graph-based and text-based machine learning techniques to detect agreements as well as disagreements online, in light of recent research showing that context, such as social context or knowledge graph information, enables language models to perform better on downstream NLP tasks. <h3>Key Takeaways:</h3> <ol><li> </li><li>Language models trained on existing datasets underperform when run on real world data. This discrepancy is exposed when evaluating the models using the new DEBAGREEMENT dataset.</li><li> </li><li> </li><li>DEBAGREEMENT presents new opportunities for modeling diverse online interactions with text and context (authorship, graph, temporal information). Its graph structure allows for the combination of text-based machine learning (ML) with graph representation learning (GRL) approaches.</li><li> </li><li> </li><li>By modeling online discussion forums as graphs of user interactions, researchers can: 1) transform the agreement/disagreement detection problem into a sign link prediction task, and 2) use existing signed graph embedding techniques evaluated on publicly accessible signed graphs such as Epinions and Slashdot.</li><li>The sentiment and polarization of topics is not static in time and for the first time, this dataset exposes that and shows how realistic topic sentiment evolves over time.</li><li> </li></ol><h2><a href="https://arxiv.org/abs/2111.04204" rel="noopener noreferrer" target="_blank">Natural Adversarial Objects</a></h2> <h3>Paper Summary:</h3> Although cutting-edge object identification approaches have shown impressive performance, models are often vulnerable to adversarial assaults and out-of-distribution data. Natural Adversarial Objects (NAO) is a novel dataset for assessing the robustness of object identification techniques. NAO comprises 7,934 images and 9,943 objects that are unaltered and depict real-world circumstances, yet cause cutting-edge detection methods to misclassify with high confidence. When compared to the conventional MS-COCO validation set, the mean average precision (mAP) of EfficientDet-D7 lowers by 74.5 percent when tested on NAO. Furthermore, when evaluating a range of object detection architectures, we discovered that improved performance on the MS-COCO validation set does not always transfer to better performance on the NAO, implying that robustness cannot be gained merely by training a more accurate model. We look at why NAO cases are tough to identify and classify. Experiments using image patch shuffles demonstrate that models are too sensitive to local texture. Furthermore, when predicting class labels with integrated gradients and background replacement, we discovered that the detection model is dependent on pixel information inside the bounding box and insensitive to background context. <h3>Key Takeaways:</h3> <ol><li> </li><li>The NAO dataset presents a difficult robustness assessment challenge for detection models trained on MS-COCO. As part of our evaluation, we tested seven state-of-the-art detection models from a variety of families, and found that they consistently failed to perform accurately on NAO when compared to the MS-COCO validation set, which included both in-distribution and out-of-distribution objects, among other things.</li><li> </li><li> </li><li>We also described the process of building such a dataset yourself, which may be used to create similar datasets for any target dataset or pre-trained model.</li><li> </li><li> </li><li>The popular MS-COCO dataset has "blind-spots" that make it challenging to appropriately categorize these naturally adversarial objects. </li><li> </li><li> </li><li>Additionally, we show that detection models are overly sensitive to local texture but insensitive to background change, which makes them vulnerable to natural adversarial objects.</li><li> </li></ol><h2><a href="https://openreview.net/pdf?id=mPducS1MsEK" rel="noopener noreferrer" target="_blank">Are We Learning Yet? A Meta-Review of Evaluation Failures Across Machine Learning</a></h2> <h3>Paper Summary:</h3> Evaluation issues often undermine the validity of results in machine learning research. In collaboration with researchers from Stanford University, the University of California, Berkeley, and the University of Washington, we conducted a meta-review of 100+ survey papers to identify common benchmark evaluation problems across subfields. In some cases, several years’ worth of progress in certain fields may be misstated. Our meta-review surveys evaluation papers reporting on a broad range of subfields, ranging from computer vision to deep reinforcement learning, to recommender systems and natural language processing, and more. We found a consistent set of failure modes, which we organized into a systematic taxonomy. This taxonomy of failure modes helps researchers understand where different types of failures are common and provides a framework for future research. Our taxonomy distinguishes between “internal” and “external” validity failures. Internal validity failures occur within the context of a single benchmark and raise questions about the consistency or reliability of measurements on that specific benchmark. For example, running baseline methods with less hyperparameter tuning than a newly proposed method can underestimate the performance of existing methods and hence lead to overly optimistic claims of performance improvements. External validity failures, in contrast, describe failures that occur when dealing with multiple benchmarks. External validity failures occur when results from one benchmark do not generalize to other settings. One example is dataset misalignment: if data collected for a benchmark is not representative of the setting where models will be used, then results from that benchmark may not reliably predict performance in the real world. <h3>Key Takeaways:</h3> <ol><li> </li><li>Evaluation failures pose a risk to the validity of research findings in machine learning. Our meta-review of 100+ survey papers observed common failure modes across a wide range of subfields. </li><li> </li><li> </li><li>While some internal validity failures can be identified in the context of a single benchmark, other external validity failures only appear when looking at multiple datasets.</li><li> </li></ol><h2>Conclusion</h2> As we look ahead to 2022, we look forward to supporting more university research teams and driving more research initiatives within Scale. To learn more about our work and the work of many of our partners, our team hosts regular TechTalks on <a href="https://exchange.scale.com/public/events" rel="noopener noreferrer" target="_blank">Scale Exchange</a>, our community of ML practitioners. Join the community to learn more and register for upcoming talks. If you’re interested in joining our growing ML team, take a look at our <a href="https://scale.com/careers" rel="noopener noreferrer" target="_blank">careers page</a> for open positions.

An Overview of ML Research Papers Accepted At NeurIPS 2021

Research Papers Accepted At NeurIPS 2021

Machine Learning Papers at NeurIPS 2021

Artificial Intelligence (AI) is now a part of our everyday lives and has the potential to massively impact the future of our society. Leaders in the private sector are making a real-world impact with AI today - from autonomous vehicles, medical diagnostics, vaccine distribution, and eCommerce. We also recognize the importance of the federal government, academia, and the private sector to work collaboratively to drive innovation and ensure continued U.S. leadership in technology. In early October, we hosted our second virtual conference: TransformX. Over the two days, we brought together more than 22,000 researchers, practitioners, and executives to hear from leaders across industries, academia, and government. We’re excited to share some key insights and takeaways that the foremost government leaders have on the future of AI. <h1>Fireside Chats With Key Leaders</h1>Mark “Drifter” Valentine, Head of Federal at Scale AI hosted fireside discussions with key government and private sector leaders including Lieutenant General Kirk S. Pierce, Commander, Continental U.S. North American Aerospace Defense Command Region-1st Air Force, Rear Admiral Lorin Selby, Chief of Naval Research, and Steven Escaravage, Senior Vice President at Booz Allen Hamilton. <h2>Lieutenant General Kirk S. Pierce</h2> LTG Pierce spoke with Mark on operationalizing AI for National Security. For Pierce, homeland defense is a chief interest. To work effectively and with prudence, “We have this grand, sweeping concept of joint all domain command and control.” To achieve this, Pierce says interoperability is key. The key to interoperability? Data. “We cannot continue to have data stovepipes,” he said. “Depending on wherever that data comes from—whether it's a sensor, above water, below water, in space, whatever it is, we want to have the data to be able to make decisions.” He also said that he sees AI and ML operability as “an opportunity to reset, fundamentally, how we look at information, how we aggregate information and be able to evaluate and assess the validity of the information.” “The companies that are working on that are gonna change how we fundamentally, as senior leaders, make decisions.” <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/vgxbm41ptn"></iframe> <h2>Rear Admiral Lorin Selby</h2> RADM Selby discussed the importance of fostering an AI innovation ecosystem. As a leader who is positioned at the cross-section of government, academia and industry, Selby is optimistic about the utility of AI to help humans reach their potential, and the Navy achieve its goals. He foresees a future where AI can “unburden the mind” from tedious bureaucratic work, “So we can actually focus on the things the human mind is really good at focusing on— which is kind of more strategic thinking, more abstract thinking.” “This is a pivotal moment in history,” he said of AI and ML advancements. “I think we’ll look back on this time as a key transition phase in history.” Selby said that building confidence in AI systems is essential to innovation, as is remaining anchored in the strength of human decision making. “We have to develop trust,” he said, while emphasizing the need to keep humans in the loop of critical decisions. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/h3hikuszel"></iframe> <h2>Steve Escaravage</h2> Steve Escaravage shared his insights on developing realistic approaches to deploying machine learning (ML) in federal environments. Escaravage said that for advancements in ML, the private sector is leading, and the government has to follow. “The innovation is coming from outside—it’s coming from the private sector, from commercial use cases” he said. “For the public sector—for the government to be successful, we simply cannot ignore that investment and we need to channel that into the public sector.” He said that although there has been real progress and momentum since 2017, there are pieces of the puzzle that are missing. Friction points include: access to real world data assets and data sets to train AI, approved computing environments that are configurable and scalable and systems with authorization to operate (ATO) accreditation. “We all have a responsibility as citizens of our governments to try to get them to better outcomes,” he remarked.<iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/0b1pm4gn25"></iframe> <h2>Expert Panel:</h2> In addition to the fireside chats, Mark also led an expert panel discussion on The State of AI for National Security Mission Applications featuring Michael “Rabbi” Harasimowicz and Rachael Martin. Martin said that she hopes that in the future, AI will be a seamless part of her agency’s workflow, and as commonplace, critical and functional as something like email. “The new warfighting demand is time,” she said. “When you have machine speed, you need to have trust and confidence that what the machine is telling you is correct.” As the NGA relies on overhead imagery and data to compel decision-making, she said it’s important to have a systematic data workflow from input to warfighter. “We do think that AI and automation is really the best way of accomplishing that,” Martin said. She added that she sees the time investment and the leadership investment in bringing AI to the mission space as heartening. “You have to change culture by getting people involved in the change,” she reflected. Harasimowicz said that if our government doesn’t update its systems, the systems and the country are at risk. “Our national security challenges are really getting to the point where they're beyond human scale,” he said. “And I say that because the amount of data that is flowing in and around all of our domains is just insurmountable for just a commander.” The level of support required to manage the data at a tactical and strategic level drives us to have a level of assistance, he said. “So artificial intelligence can provide that backdrop.” “Our adversaries are challenging themselves in aggressive ways,” Harasimowicz said. “And we have to be as aggressive, if not more so aggressive…We have to go faster. We have to find ways to solve solutions in weeks, as opposed to years.”<iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/2yn8oyur22"></iframe>

Highlights from TransformX 2021: U.S. Government Related Sessions

Highlights from TransformX 2021: U.S. Government Sessions

<h1>Natural Language Processing and Online Discourse:</h1>Online debates shape our society, and have a profound impact on how people perceive the world. Over 4.2 billion people actively use social media, and many of them participate in online discussions. The breadth and depth of online discourse provide insights and data for researchers and curious people alike.One way to understand data from online discussions is through the field of Natural Language Processing (NLP). NLP offers techniques to understand textual interactions, and to gain a better understanding of polarization online.However, current NLP models often struggle with nuanced language in online exchanges, such as slang, sarcasm and topic-specific jokes, making it difficult to understand diverse online interactions. A task called stance detection is useful in this arena to address this challenge.Stance detection is a text understanding task that consists of classifying the position (or stance) of the author of a piece of text, towards an initial message or statement. In an online setting, these statements might include comments or messages in a thread. The stance is classified into one of three classes: in favor, against, or neutral.<h1>Methodology:</h1>Recent work has shown that stance detection would benefit from context-sensitive approaches. We were not able to find realistic, large, open datasets with contextual information which do not have platform-specific features.Thus, we created <a href="https://scale.com/open-datasets/oxford" rel="noopener noreferrer" target="_blank">DEBAGREEMENT</a> with conversations from Reddit to better understand the evolution of online debates, and to provide <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotated data</a> for agreement and disagreement detection.The comment or reply function on social media sites does not provide sufficient information to classify an interaction. So, we looked at actual online conversations. Unlike existing datasets for stance detection, DEBAGREEMENT provides realistic online discussions, with diverse writing styles, genres and topics of discussion.The set is comprised of 42,894 comment-reply pairs from Reddit, annotated with agree, neutral, or disagree labels.The dataset collects data from five forums on Reddit:<ol><li>r/BlackLivesMatter</li><li>r/Brexit</li><li>r/Climate</li><li>r/Democrats</li><li>r/Republican</li></ol>We pulled from three subreddits of social movements r/Brexit, r/BlackLivesMatter, r/Climate, and two subreddits of political affiliations r/Republican and r/Democrats.To ensure we were only using high-quality interactions, we excluded: empty comments, comments from deleted authors, comments that were hidden for user privacy reasons, and comments containing hyperlinks. Further, in order to <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotate</a> impactful discussions in a given subreddit, we removed posts with fewer than 10 words and few comments.This dataset presents opportunities to detect (dis)agreements by leveraging context beyond just text. The set does not rely on platform-specific features, such as retweets or hashtags. The data was collected such that, for each subreddit, the resulting set of interactions formed a multi-edge, temporal graph where nodes are users, and edges represent a comment-reply interaction between two users.<h1>Challenges:</h1>Despite the preprocessing we performed to ensure <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">high-quality annotations</a>, the dataset annotation came with certain obstacles.We used methods that were developed in-house such as dynamic consensus to ensure high quality annotations. (For more details on dynamic consensus, take a look at our blog on <a href="https://scale.com/blog/ml-human-consensus" rel="noopener noreferrer" target="_blank">ML + Human Consensus</a>.) Annotating short online exchanges is tough; only 33% of the pairs were annotated with a full agreement among annotators. Accounting for nuanced statements, sarcasm, diverse writing genres, and specific socio-economic contexts is a tall order.<h1>Results:</h1>This research presents a challenge for NLP systems, and attempts to argue:<ul><li>Current language models struggle with messy data when it is either scraped directly from social media, or pulled from the human dialogue.</li><li>Clean and tailored datasets may not have transferable insights for online (dis)agreement detection.</li></ul>We evaluated the performance of state-of-the-art language models against our new stance detection dataset and assessed the transfer learning potential with other stance detection datasets already available.State-of-the-art language models are faring well on simple examples, but fail on more complex ones. These models tend to overly rely on superficial cues, such as the first word of the reply.Moreover, it appears that models trained on clean stance datasets behave poorly on our new dataset, which speaks to the new challenges that our dataset offers.This research presents promising avenues. Our findings highlight ways to improve language models with contextual information.We see promise in DEBAGREEMENT as it presents an opportunity to train socially aware language models.<h1>Further Research and Recommendations:</h1>We believe expanding annotations to other areas of discussion, such as consumer products reviews, investor forums or wider political forums could lead to further discoveries and insights about future consumer preferences or investor decision-making.The dataset also provides the possibility to investigate social theories, understand polarisation, study online social movements and examine how people express their opinions and change their views on social media.<h1>Read The Full Research:</h1>This work was achieved in collaboration with Oxford University, who are the first authors of the paper.The paper has been accepted to the “Datasets and Benchmarks Track” of <a href="https://neurips.cc/" rel="noopener noreferrer" target="_blank">Neurips 2021</a>. The paper details each of these issues further and is accessible on <a href="https://openreview.net/forum?id=udVUN__gFO" rel="noopener noreferrer" target="_blank">OpenReview</a>.

New Research Paves The Way to Train Socially Aware Language Models

Scale AI And Oxford University's Reddit Dataset Provides Comprehensive Online Discourse Data

At Scale, we help our customers accelerate the development of their AI applications. We do this through a variety of products and services, but here we’ll focus on one: <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">high-quality labels</a> for training data. We’ve developed a process for delivering model-improving data quickly and efficiently, and we highlight parts of this process below.To move fast, we align with our customers early. We strive to understand what our customers ask for and why, so that we can use the best combination of product and operational expertise to <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">label their data</a> with high quality - with their ML goals in mind. We take a collaborative and thoughtful approach to architecting and building data annotation pipelines. It’s one of the many reasons that customers continue to trust Scale as their AI partner.<h1>We architect data annotation pipelines that map to our customers’ ML goals</h1>At Scale, we start by understanding what our customers need to accomplish. What specifically will the AI model need to do? Is it for an <a href="https://scale.com/customers/toyota" rel="noopener noreferrer" target="_blank">Autonomous Vehicle</a> (AV) use case, perhaps predicting the action of another agent? Or is it for an application that can recognize and track objects for <a href="https://scale.com/customers/cellareye" rel="noopener noreferrer" target="_blank">inventory management</a>? From this, we can begin to understand:<ol><li>What data will the model ‘learn’ from?</li><li>What data will be used for inference, once the model is deployed in production?</li><li>What should the ideal output look like?</li><li>What determines a good (or bad) inference output?</li></ol>We’ve found that the success of an AI project is highly correlated with the quality and specificity of the training data used. This means that a broad or general training dataset won’t usually be the most efficient or quickest way to build an ML model. As a result, we collaborate with customers to set up goal-oriented annotation pipelines that map directly to their ML roadmaps. This approach helps them achieve their KPIs and project milestones.As an example, it is common for new customers to ask us to replicate a broad-purpose public dataset, such as the AV-centric <a href="https://scale.com/open-datasets/nuscenes" rel="noopener noreferrer" target="_blank">NuScenes</a>, which we released with <a href="https://www.aptiv.com/" rel="noopener noreferrer" target="_blank">Aptiv</a>. Once we deep dive on their ML roadmap however, we discover the sub-goals (such as lane planning and pedestrian intent recognition) and the desired timelines for each. We then build specialized pipelines that generate the required training data at higher quality and at faster speeds than the customer would otherwise have done themselves. In fact, our speed of operation has frequently helped customers meet key deadlines, like public conferences or demos.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/673f8157-ef5b-4b51-a76d-ee5f8b9d0100/public" alt="Common Autonomous Vehicle ML Roadmap and Data Needs.">Scale seeks to understand our customers' timelines of ML goals and to spin up label pipelines targeted at the specific data needs for each goal.It’s important to note that few customers come to us with neatly-defined, single pipeline needs (they used to!). Today, our customers often bring multi-modal data from multiple sensors. For an AV customer, this could mean LiDAR, radar, and a combination of regular and fisheye cameras. Having a clear understanding of our customers' goals helps us be a thought partner on the types of objects or events to label across these sensors. It also helps us propose a pipeline architecture that incorporates the best product capabilities to produce a comprehensive, goal-oriented training dataset.For example, we might propose a string of dependent pipelines that weave together categorization, 3D LiDAR cuboids, and LiDAR linking. Weather attributes and 3D data defects might be caught and categorized in the categorization pipeline, vehicles might be tracked in the 3D LiDAR cuboids project, and camera-based attributes like blinker status might be generated in the linking project.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2e43fe86-4639-495c-3096-34676043a300/public" alt="Using specialized label pipelines to turn multi-sensor inputs into training data. ">AV customers often come to Scale with data from multiple sensors and with an idea of what labels they need. Scale's team then architects specialized label pipelines that bridge the two, producing high-quality labeled data quickly and efficiently.<h1>We test and fine tune pipeline architecture by creating a golden dataset and vetting it with our customers</h1>Following this thoughtful approach to customer alignment helps us achieve a data milestone: the creation of a golden dataset. A golden dataset is used to define what ‘good’ looks like. It is used as a reference for us, our human Taskers, and customers’ own stakeholders.Creating a golden dataset is a journey of discovery. We set out to create the golden dataset by annotating a representative sample of data in the pipeline we architected. Annotating some data always generates observations and insights about nuances in the data and requirements, which we surface to the customer along with our recommendations. We then review the resultant annotated data with our customer, tying open questions and considerations back to the customer’s ML goals.Reviewing the proposed golden dataset together with the customer is key. For example, we were working with an autonomous drone customer who needed polygon labeling of flyable space, ie where their drones could safely navigate. We found ourselves drawing pixel-perfect polygons around every tree branch, only to have the customer reject every task. Only once we debriefed live with the customer did we realize that for them, quality training data meant polygons that avoided tree pixels by a wide margin - ie, rough, simple polygons that just didn’t clip the treetops! That one collaborative sync put the project back on track with a clear path to consistent, relevant, and accurate labeled data.Attaining this mutually-vetted golden dataset is powerful. It powers our internal (human) Tasker training and assessment tools, giving both Taskers and the operations team a clear understanding of what success looks like. It also aligns our Quality Assurance (QA) team, whose job is to effectively mirror customers’ QA teams to reduce customers’ operational data-management burden.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/ecd95dc4-c137-4a9e-4024-569dd8df3d00/public" alt="Achieving and leveraging golden data">Reviewing label samples together with customers helps establish a golden dataset, which in turn makes is possible to scale up production of high-quality labels.<h1>We engage all relevant stakeholders to ensure customers are aligned with Scale - and amongst themselves</h1>We’ve recognized that our customers are running complex initiatives involving many stakeholders on many teams, including operations, engineering, legal, and procurement. Alignment isn’t complete until Scale has aligned with each customer subteam, AND the customer subteams are internally aligned amongst themselves.Projects can fail if the customer’s operations team is grading labeled data according to a different set of standards than what the ML team expected. This can take many forms - operations might be grading too easily (for example, approving polygons that are too swaggy), or grading too harshly (for example, rejecting boxes with acceptable sizing), or just grading according to a different or outdated rubric (for example, applying an old definition of “truck” that included small pickup trucks).The result of inconsistent labeling standards is that the ML team gets training data that doesn’t improve model performance. Or, the ML team gets starved for data, even though there is data available that would have improved performance! Few things are worse than delivering weeks or months of data, with a consistent green light from the customer’s operations team - only to get a mass rejection from the ML team once it tests model performance.Scale drives stakeholder alignment by bringing the right people into the discussion, especially at steps that we know to be high risk. We also actively monitor for hints that there might be misalignment (like radio silence from some stakeholders at key milestones) and work with our customers to drive alignment internally and with Scale. We pride ourselves at being a knowledgeable partner who knows how to avoid situations that de-ccelerate the development of AI applications.<h1>Alignment is never won and done; we continue to re-assess alignment to our customers’ ML goals</h1>It takes a deliberate effort to maintain alignment. Very often, model evaluations lead customers to update their data requirements and labeling guidelines. Even when the labeling guidelines remain constant, customer data itself can change. For example, we once built a reliable high-quality pipeline to label street scenes, only to find the next batch of customer data contained scenes of people lying on a racetrack.Throughout the lifecycle of the data annotation pipeline, we continue to apply our alignment process. We continue to check in on customer ML goals, refine pipeline setup, maintain the accuracy of the golden dataset, and ensure alignment between teams on the customer, Scale, and Tasker side. This accelerates the speed with which our customers receive high-quality labeled data, which in turn accelerates their development of AI applications.To experiment with Scale’s data labelling at a small scale, sign up for <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a>, the fastest way to production-quality labels, with no data minimums. To set up custom pipelines for your ML needs, <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">talk to our team</a>.

We strive to understand what our customers ask for and why,  to label their data at high quality.

Partnering with customers to architect data pipelines

How Does Scale AI Partner with Customers to Produce High-Quality Training Data?

Every year, nearly 200,000 service members transition to civilian life. As a fighter pilot who served for over two decades in the U.S. Air Force, I know the challenges these men and women face. From adjusting to “normal” life, to mental health struggles, to determining a new career path, it can be overwhelming and difficult to navigate. One of my favorite things about Scale is our CEO’s hiring philosophy. Simply put, he says to hire people who give a shit. That philosophy has brought an amazing group of people to this company who not only care about technology, but also care about our country, our government, and our service members. As we prepare to celebrate Veterans Day, we wanted to share some of our Scale veterans’ stories to honor them and their commitment to our country, and to also offer advice for those navigating the transition from service to civilian life. <h1>What did you do in the military?</h1>Jason Schmid: I was a U.S. Army Military Intelligence Analyst, and deployed to Iraq twice. The first time was as a part of the Iraq Survey Group, determining the disposition of Iraq’s suspected weapons of mass destruction programs. On that mission, I was part of the team interrogating former high ranking Iraqi Regime officials. On my second deployment, I was embedded as a Tactical Intelligence Trainer in an Iraqi infantry battalion. Dan Tadross: I was an Air Traffic Control Officer responsible for coordinating aircraft requests and the deconfliction of airspace. In my last few years of Active Duty, I had the opportunity to serve as an Innovation Advisor to the Secretary of the Navy on how to integrate AI into Aviation Command and Control (specifically how to shorten planning cycles). I also had the privilege of being a founding member of the Department of Defense (DOD) Joint Artificial Intelligence Center (JAIC). During my two years at the JAIC, I ran a number of programs from supporting testing and evaluation to establishing Project Gargoyle for Force Protection. Brianna Hester: I was a Geospatial Analyst for 24 years. My most recent position was the Operations Superintendent of the European Partner Integration Enterprise (EPIE) where I managed a 40-person, international intelligence team operating a $5.7M Intelligence, Surveillance, Reconnaissance (ISR) coalition program for combat mission planning, execution, and overall intelligence gathering and research. I oversaw ISR mission operations and imagery exploitation of U.S. ISR platforms employed for collection of full motion video imagery in support of the DOD, NATO, and coalition forces. I deployed to Qatar and Afghanistan and also traveled to numerous other countries for my duties. Jacob Sheehan: I was a U.S. Army Special Forces Officer, also known as a Green Beret, where I deployed on combat missions to Iraq and Afghanistan and on training missions in India, Thailand, and Korea. In addition to leading operational teams, I also had an academic assignment at West Point where I taught economics courses, ran the graduate scholarship program (Rhodes, Marshall, Fulbright, etc.) and designed a two-semester critical thought curriculum. Matt Beck: I was commissioned as a Naval Officer in 2008. I spent my first two years as a Surface Warfare Officer on a destroyer. I was selected to attend SEAL training after that tour and completed training in 2012. I led teams of various sizes on deployments to Europe, Africa, South America, and the Middle East. Tyler Haley: I was a Special Operations Communicator in the U.S. Army for 10 years. I spent most of my career inside of Special Operations doing everything from tactical communications to designing and deploying data centers. I’ve spent a large portion of my career in support of overseas operations from Afghanistan, Iraq, Syria, to most of Central Africa. I was lucky enough to get to finish my career in doing what I love most, which is any and all things cloud! <h1>Why Scale?</h1>Jason Schmid: When I left government service, it was important to me to join a company that values its contribution to national security. All of the Scale senior leaders I spoke to in the interview process made it clear that Scale is dedicated to using its technology to serve the nation. I think it is hard to overstate how important the work Scale is doing for the nation, both on the commercial and federal sides of the business. Dan Tadross: One of the most challenging aspects for the U.S. government to adopt and implement AI responsibly is the lack of structured data. Scale is at the forefront of developing structured data and can do so at a volume and quality that can help catapult government efforts and ensure the competitive edge for the U.S. Brianna Hester: I was intrigued by the job description. When I opened it and looked at the requirements, I had all the requested experience so I applied. After chatting with Austin Woo about some more specifics of the position, I knew I wanted to be a part of this program. Jacob Sheehan: I was primarily drawn to Scale through my hiring manager, David Kinsella. I was headed towards working at a financial technology company in my job search, but David introduced me to Scale’s compelling vision in the AI industry and the role it could play in making the U.S. government more nimble to respond to pressing challenges. He viewed my military experience as an asset and imagined my future impact at Scale rather than solely focusing on my non-traditional business operations background. Matt Beck: I was drawn to Scale’s mission of accelerating the adoption of AI. A lot of the work we are doing on the federal team has a direct application to our nation’s defense, and I wanted to be a part of that work. Tyler Haley: I love the team dynamics of start-ups. Everyone wears different hats depending on the meeting or the time of day. This is super relatable to being on small unit teams inside of the Special Operations community. Scale, in particular, pushes the boundaries inside of technology and solves hard problems, which is exactly what I did in the U.S. military. <h1>What do you do at Scale?</h1>Jason Schmid: I am Scale’s Head of Government Relations. I work to help Congress understand AI and ML capabilities so that they can build programs and policies that will help the U.S. government accelerate the adoption of AI. Dan Tadross: I am an Engagement Manager helping to bridge the gap between U.S. government requirements and experimentation and Scale’s expertise. Brianna Hester: I am the Synthetic Aperture Radar (SAR) Imagery Intelligence Subject Matter Expert (SME). Jacob Sheehan: I’m a Business Operations and Strategy Manager that supports the federal team. My main projects include growing a compounding network of labeling experts and labeler performance management. Matt Beck: I am a Federal Deployment Strategist focused on developing solutions to support the modernization of government systems. Tyler Haley: I am the Infrastructure and Platform Lead for our federal team. I partner closely with our software engineering and security teams to help design infrastructure that works on various AWS classification regions. <h1>How was your transition from service to civilian life?</h1>Jason Schmid: I transitioned from the Army 15 years ago, and the circumstances of my transition were pretty unique. I went from being in literal combat to being a civilian in about three weeks. I was very fortunate to have a job lined up already, but it would have been much better if I had time to take advantage of all of the resources available to transitioning service members. Such a jarring transition was not easy. We now know so much more about successfully transitioning veterans back into civilian life that would have made my transition so much smoother - not just for me, but for my family as well. Dan Tadross: I transitioned off Active Duty service after 12 years during the height of the pandemic in 2020. So the transition process was stressful and challenging, but when I eventually landed at Scale my level of apprehension subsided. Brianna Hester: My transition was pretty easy, as I was ready to begin my post-military life. The hardest thing to get used to was not having to tell someone my every move. Taking PTO without telling someone the address of where I would be for the duration of the PTO was a little unnerving the first time. Jacob Sheehan: I gave myself sufficient time, but there seems to be no such thing as too much time when it comes to major life transitions, especially if you have a family to take care of! I’m glad I had the space to think about what I wanted to do next instead of having to manufacture it quickly. Transitioning during COVID was a challenge from the standpoint of young kids being around all the time, so I would recommend finding a quiet space to do your transition thinking. Matt Beck: I had a wonderful transition departing the military. I met Shands Pickett early on in the process and kept in touch as positions at Scale became available. My preparation and exploration of many different post-military careers went well and resulted in me working at Scale. Tyler Haley: Transitioning was a bit tough for me. I got out of the service in May 2020 right as the pandemic was really setting in. Transitioning from being in a room with people that you are close with to only seeing people via Zoom and Slack was difficult at first. I was accustomed to seeing my daily contributions make a real impact, but when I got out, that faded away rather quickly. The good part about transitioning was being able to be at home and start a family. My wife and I welcomed our first kiddo into the world shortly after I got out. It gave me a new purpose and perspective in life. It helped me fill the void that the military left. <h1>Can you speak to your experience as a veteran at Scale?</h1>Jason Schmid: I think one of the best qualities of Scale’s culture is a genuine curiosity and passion to solve hard problems as a team. I think anyone who has served can connect with that. Dan Tadross: One of the aspects I miss most about serving in the U.S. Marine Corps is the comradery and sense of purpose. While nothing can compare with serving in a Marine unit, Scale has a culture that shares some similarities and the team I am a part of is motivated to succeed and support the U.S. government toward the responsible adoption of AI. Brianna Hester: Scale has become my new family. I still have my military family and always will but now I have added another branch to my family tree with everyone from Scale. They may be very different families but I wouldn’t trade either of them for anything and I can’t wait to see how much we accomplish together! Jacob Sheehan: Spending time with the many veterans on the federal team has maintained some of the camaraderie I enjoyed in the U.S. Army. I’m also pleasantly surprised how genuinely curious Scaliens across all of Scale are in veteran experiences! Matt Beck: Scale is extremely supportive of its veterans. My work keeps me engaged on a mission I believe in. Furthermore, Scale is extremely flexible in letting me maintain my connection to the U.S. Navy by supporting my drill requirements. Tyler Haley: Although I am still new to the team at Scale, I felt value-added from day one. As I mentioned earlier, feeling like what you do matters is a huge point of pride for most veterans. I instantly felt as if I was a part of the team and also was surrounded by other team members that have served. At Scale, you really get the experience of being back on those small teams that have a mission and purpose. <h1>What advice would you share with veterans making the transition?</h1>Jason Schmid: Take advantage of every resource available to support your transition, whether you think it applies to you or not, because you’ve earned it. It’s normal for recently separated veterans to feel some loss of purpose. It’s important to find ways to continue serving something larger than yourself, even if it's not part of a career path. Dan Tadross: There are a lot of opportunities for those who want to find alternative ways to continue to support the U.S. government. Many of the skills I learned in the U.S. Marines continue to help me to succeed in the private sector. Those experiences and capabilities that made you successful as a service member are in demand in the private sector, so when making the transition, don’t forget the value you bring to any organization. Brianna Hester: My recommendation would be to take a break and relax for a minute before jumping right into another job, if you can. It’s a huge change going from complete structure in your everyday life to no structure, and it can be hard on some people. Enjoy a small (or even large) break and use that time as a way to prepare yourself for the great things to come. Jacob Sheehan: Spend a lot of time exploring the industry you want to work in, understanding the companies uniquely positioned in that industry, and thinking about the strengths you can bring to roles in those companies. I’ve had many veteran friends get it backwards by over-indexing on the role (title) and barely giving any thought to the company or larger industry. Matt Beck: Always seek to improve your position as you look for opportunities. Forecast your worst case scenario out as far as possible. Once you are comfortable with your worst case scenario, seek out the roles and companies that you are most excited about and get connected early. Tyler Haley: Figure out what fills your voids. Find a routine and stick to it. A lot of the service members that I speak with feel the same way. Whether that’s getting up every morning and hitting the Peloton or cooking dinner for the family every night, find something that you can do for YOU. Find a new skill or hobby and try to master it. Get out of the house and away from the keyboard. Find time in your days to step away and get lunch with another veteran. Transitioning is different for everyone. Make time to see those who understand what you are going through. If you are interested in joining our team, check out our <a href="https://scale.com/careers#jobs" rel="noopener noreferrer" target="_blank">open positions</a>.

As we prepare to celebrate Veterans Day, we wanted to share some of our Scale veterans’ stories.

Scale Team Spotlight: Veterans at Scale

<h1> </h1><h1> Why build an ML-based workflow for data labeling?</h1> Delivering high-quality data labels quickly is a challenging task. Even for humans, image and video data commonly found in self-driving or robotic applications can be compositionally challenging to parse and correctly annotate. At Scale, we use machine learning to augment human workflows to both improve the quality and speed of labeling. Since deep learning models can struggle to perform consistently across diverse data domains like self-driving car scenes, finding the optimal balance between ML automation and human supervision is necessary to ensure we maintain high-quality.Autosegment, an ML-powered tool that can accurately segment instances within a human-defined bounding box, allows us to delegate high-level reasoning about semantics and taxonomy to Taskers (human labelers) and basic pixel-level processing to an ML model. This division of labor enables us to build a general purpose and accurate ML labeling tool. Despite using a simple neural network, Autosegment outperforms prior tools.<h1> </h1><h1> The Challenge of Semantic Segmentation Labeling</h1> Semantic segmentation involves human Taskers identifying and labeling every pixel in a visual scene — a notoriously time-consuming and tedious process. Most Tasker time in this manual approach — about 95% — is consumed by time-intensive work, such as carefully tracing the outlines of objects in a scene. For example: it takes roughly one minute to trace the outline of a car and only three seconds to click on that outline and select the label “Car.” What if we could instead use machine learning to automate the most laborious parts of semantic segmentation? What if an ML model did the semantic segmentation, and the Tasker only had to select the right label?<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/72c723b3-46ae-4384-b75b-4a87b5e40b00/public" alt="Semantic Segmentation Task"> A visual scene which has been semantically segmented. Each pixel is color coded to denote the class of object it represents, i.e. Pedestrian, Road, Bike, etc. As ML engineers, we rely heavily on quantitative validation metrics, such as label accuracy and intersection-over-union (IoU), to determine the quality of our models. However, when building models that optimize for a particular business outcome — for instance, reducing the time it takes Taskers to semantically segment images — we need to think critically about which model to use and how we measure its performance.. In this example, our objective is to produce predictions that streamline the Taskers’ job. This is not necessarily the same as “best model performance.” For example, an instance segmentation model might be consistently inaccurate by just a few pixels around an object. This model might have a high IoU score but very little practical usefulness because Taskers would need to re-do every trace of the outline. Conversely, a model that can perfectly segment an object, even if it only identifies half of it, would still be extremely useful. Despite having lower IoU validation scores, it’s still an improvement to the overall workflow. Because it takes so long to trace outlines manually, having a model produce pixel-accurate partial outlines is still a significant time saver. <h1> </h1><h1> The Active Tooling Paradigm and the Autosegment Tool</h1><h1> </h1>Autosegment is a machine learning tool that segments the most salient object within a human-provided box annotation. It incorporates Tasker feedback at annotation time in a way that is flexible and iterative. When building Autosegment, we realized early on that the only way to provide flexibility and high quality was to give Taskers access to Active Tools. Active Tools let Taskers make changes to the model inputs and parameters in real time to generate the best output for their use case. This is similar to “prompt engineering,” the process by which we can coerce a neural language model like GPT-3 to generate desired text when given precise prompts. For more information, OpenAI hosts <a href="https://beta.openai.com/docs/guides/completion/prompt-design" rel="noopener noreferrer" target="_blank">excellent documentation</a> on how to design prompts for their GPT-3 model. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/07588dc3-6a06-45e7-c61f-a51562470900/public" alt="Open-AI GPT-3 Model"> Humans working alongside models can produce better results faster than models alone Using Active Tools, we give Taskers as many options as possible to make their tasks easier. For example, they can adjust the annotation box shape and the prediction confidence threshold to configure the model to speed up their particular labeling task. <img src="https://cdn.builder.io/o/assets%2Fe0438815ba51486bbb6a202747122d4b%2Ffef154571d5846b6b87674a46a123ac5%2Fcompressed?apiKey=e0438815ba51486bbb6a202747122d4b&amp;token=fef154571d5846b6b87674a46a123ac5&amp;alt=media&amp;optimized=true" alt="Autosegment Tool Demo">A demo of the Autosegment tool. Enabling humans to freely explore the output space of the model is a key advantage of active tooling. We've already seen how this is useful. Overall, we have observed the median annotation time per scene is about 30% faster with Autosegment. We also interviewed Taskers to see how they are using the tool. While our model is trained to segment cars and motorcycles, one Tasker used the tool to segment everything. While this wasn’t a workflow we had envisioned, it halved their time per scene segmentation, from 40 minutes to 20 minutes. In a discussion we had with five Taskers, they explained that they consider Autosegment to be more accurate and efficient than previous ML solutions. Now we can simply supply robust and general online models and let Taskers optimize the rest. <h1>The Autosegment Model</h1>Autosegment is a convolutional neural network trained to segment the most salient instance within a Tasker-defined bounding box. This simple design allows us to significantly reduce our customers’ operational overhead to deploy ML, while providing pixel-perfect instance segmentations, even on previously unseen instance types. The Autosegment model does not have to be particularly complex to produce compelling results. Autosegment has no notion of different semantic types; instead, it abstractly identifies and segments based on scene composition. The model is trained to always generate a segmentation. The training data includes a few key classes like pedestrians and vehicles. The task is binary; it simply determines whether each pixel belongs to the salient object or not. We don’t train the model with negative samples. Instead, the model makes its best guess on each pixel to generate a segmentation. We define the salient instance as the object that owns the central pixel of the crop, so even if the scene is cluttered, the model can identify the target instance to segment. This binary segmentation task across instance classes and types allows the model to generalize well on new classes and taxonomies. Indeed, we can transfer not only between objects of the same class, such as cars in different countries, but between different classes altogether — for instance, moving from cars to motorcycles to street signs. This transfer can be achieved with limited retraining or, in some cases, no retraining at all, achieving zero-shot transfer. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/1a4x5jij9v"></iframe> Using salient object detection on specific crops of the scene can generate even more accurate segmentation. We were able to double the edge precision score from 0.26 to 0.52 when compared to off-the-shelf full-scene semantic segmentation models. Instead of supplying a downsampled version of the whole image to a neural network, the selected region is fed to the model at full resolution. Because the input is smaller, fewer pooling operations are required to achieve a globally receptive field. As a result, there is less downsampling and blurring. Our model achieves better fidelity around the edges of instances, a crucial part of providing high-quality data annotations to our customers. Improving full-scene labeling to the same degree would require exponentially more effort and complexity. In summary, we developed Active Tooling to create an ML-based workflow capable of solving basic computer vision problems. This workflow helps our Taskers reach higher levels of quality faster than in previous workflows. Innovations like Autosegment are critical to enable the “Moore's Law of data labeling.” At Scale AI, we’re excited to push the boundaries of human-AI interaction for our customers.

Scale AI accelerates image annotation with semantic segmentation active tooling. 

Reducing image labeling time by up to 30% and doubling segmentation quality.

Faster Semantic Segmentation with Machine Learning-Assisted Workflows

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

customers

Display order on the page /customers. Higher values show first

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

Products

Enterprise

Government

Resources