experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Customers

Company

General

People

Engineering

<h1>Natural Language Processing and Online Discourse:</h1>Online debates shape our society, and have a profound impact on how people perceive the world. Over 4.2 billion people actively use social media, and many of them participate in online discussions. The breadth and depth of online discourse provide insights and data for researchers and curious people alike.One way to understand data from online discussions is through the field of Natural Language Processing (NLP). NLP offers techniques to understand textual interactions, and to gain a better understanding of polarization online.However, current NLP models often struggle with nuanced language in online exchanges, such as slang, sarcasm and topic-specific jokes, making it difficult to understand diverse online interactions. A task called stance detection is useful in this arena to address this challenge.Stance detection is a text understanding task that consists of classifying the position (or stance) of the author of a piece of text, towards an initial message or statement. In an online setting, these statements might include comments or messages in a thread. The stance is classified into one of three classes: in favor, against, or neutral.<h1>Methodology:</h1>Recent work has shown that stance detection would benefit from context-sensitive approaches. We were not able to find realistic, large, open datasets with contextual information which do not have platform-specific features.Thus, we created <a href="https://scale.com/open-datasets/oxford" rel="noopener noreferrer" target="_blank">DEBAGREEMENT</a> with conversations from Reddit to better understand the evolution of online debates, and to provide <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotated data</a> for agreement and disagreement detection.The comment or reply function on social media sites does not provide sufficient information to classify an interaction. So, we looked at actual online conversations. Unlike existing datasets for stance detection, DEBAGREEMENT provides realistic online discussions, with diverse writing styles, genres and topics of discussion.The set is comprised of 42,894 comment-reply pairs from Reddit, annotated with agree, neutral, or disagree labels.The dataset collects data from five forums on Reddit:<ol><li>r/BlackLivesMatter</li><li>r/Brexit</li><li>r/Climate</li><li>r/Democrats</li><li>r/Republican</li></ol>We pulled from three subreddits of social movements r/Brexit, r/BlackLivesMatter, r/Climate, and two subreddits of political affiliations r/Republican and r/Democrats.To ensure we were only using high-quality interactions, we excluded: empty comments, comments from deleted authors, comments that were hidden for user privacy reasons, and comments containing hyperlinks. Further, in order to <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotate</a> impactful discussions in a given subreddit, we removed posts with fewer than 10 words and few comments.This dataset presents opportunities to detect (dis)agreements by leveraging context beyond just text. The set does not rely on platform-specific features, such as retweets or hashtags. The data was collected such that, for each subreddit, the resulting set of interactions formed a multi-edge, temporal graph where nodes are users, and edges represent a comment-reply interaction between two users.<h1>Challenges:</h1>Despite the preprocessing we performed to ensure <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">high-quality annotations</a>, the dataset annotation came with certain obstacles.We used methods that were developed in-house such as dynamic consensus to ensure high quality annotations. (For more details on dynamic consensus, take a look at our blog on <a href="https://scale.com/blog/ml-human-consensus" rel="noopener noreferrer" target="_blank">ML + Human Consensus</a>.) Annotating short online exchanges is tough; only 33% of the pairs were annotated with a full agreement among annotators. Accounting for nuanced statements, sarcasm, diverse writing genres, and specific socio-economic contexts is a tall order.<h1>Results:</h1>This research presents a challenge for NLP systems, and attempts to argue:<ul><li>Current language models struggle with messy data when it is either scraped directly from social media, or pulled from the human dialogue.</li><li>Clean and tailored datasets may not have transferable insights for online (dis)agreement detection.</li></ul>We evaluated the performance of state-of-the-art language models against our new stance detection dataset and assessed the transfer learning potential with other stance detection datasets already available.State-of-the-art language models are faring well on simple examples, but fail on more complex ones. These models tend to overly rely on superficial cues, such as the first word of the reply.Moreover, it appears that models trained on clean stance datasets behave poorly on our new dataset, which speaks to the new challenges that our dataset offers.This research presents promising avenues. Our findings highlight ways to improve language models with contextual information.We see promise in DEBAGREEMENT as it presents an opportunity to train socially aware language models.<h1>Further Research and Recommendations:</h1>We believe expanding annotations to other areas of discussion, such as consumer products reviews, investor forums or wider political forums could lead to further discoveries and insights about future consumer preferences or investor decision-making.The dataset also provides the possibility to investigate social theories, understand polarisation, study online social movements and examine how people express their opinions and change their views on social media.<h1>Read The Full Research:</h1>This work was achieved in collaboration with Oxford University, who are the first authors of the paper.The paper has been accepted to the “Datasets and Benchmarks Track” of <a href="https://neurips.cc/" rel="noopener noreferrer" target="_blank">Neurips 2021</a>. The paper details each of these issues further and is accessible on <a href="https://openreview.net/forum?id=udVUN__gFO" rel="noopener noreferrer" target="_blank">OpenReview</a>.

New Research Paves The Way to Train Socially Aware Language Models

Scale AI And Oxford University's Reddit Dataset Provides Comprehensive Online Discourse Data

At Scale, we help our customers accelerate the development of their AI applications. We do this through a variety of products and services, but here we’ll focus on one: <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">high-quality labels</a> for training data. We’ve developed a process for delivering model-improving data quickly and efficiently, and we highlight parts of this process below.To move fast, we align with our customers early. We strive to understand what our customers ask for and why, so that we can use the best combination of product and operational expertise to <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">label their data</a> with high quality - with their ML goals in mind. We take a collaborative and thoughtful approach to architecting and building data annotation pipelines. It’s one of the many reasons that customers continue to trust Scale as their AI partner.<h1>We architect data annotation pipelines that map to our customers’ ML goals</h1>At Scale, we start by understanding what our customers need to accomplish. What specifically will the AI model need to do? Is it for an <a href="https://scale.com/customers/toyota" rel="noopener noreferrer" target="_blank">Autonomous Vehicle</a> (AV) use case, perhaps predicting the action of another agent? Or is it for an application that can recognize and track objects for <a href="https://scale.com/customers/cellareye" rel="noopener noreferrer" target="_blank">inventory management</a>? From this, we can begin to understand:<ol><li>What data will the model ‘learn’ from?</li><li>What data will be used for inference, once the model is deployed in production?</li><li>What should the ideal output look like?</li><li>What determines a good (or bad) inference output?</li></ol>We’ve found that the success of an AI project is highly correlated with the quality and specificity of the training data used. This means that a broad or general training dataset won’t usually be the most efficient or quickest way to build an ML model. As a result, we collaborate with customers to set up goal-oriented annotation pipelines that map directly to their ML roadmaps. This approach helps them achieve their KPIs and project milestones.As an example, it is common for new customers to ask us to replicate a broad-purpose public dataset, such as the AV-centric <a href="https://scale.com/open-datasets/nuscenes" rel="noopener noreferrer" target="_blank">NuScenes</a>, which we released with <a href="https://www.aptiv.com/" rel="noopener noreferrer" target="_blank">Aptiv</a>. Once we deep dive on their ML roadmap however, we discover the sub-goals (such as lane planning and pedestrian intent recognition) and the desired timelines for each. We then build specialized pipelines that generate the required training data at higher quality and at faster speeds than the customer would otherwise have done themselves. In fact, our speed of operation has frequently helped customers meet key deadlines, like public conferences or demos.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/673f8157-ef5b-4b51-a76d-ee5f8b9d0100/public" alt="Common Autonomous Vehicle ML Roadmap and Data Needs.">Scale seeks to understand our customers' timelines of ML goals and to spin up label pipelines targeted at the specific data needs for each goal.It’s important to note that few customers come to us with neatly-defined, single pipeline needs (they used to!). Today, our customers often bring multi-modal data from multiple sensors. For an AV customer, this could mean LiDAR, radar, and a combination of regular and fisheye cameras. Having a clear understanding of our customers' goals helps us be a thought partner on the types of objects or events to label across these sensors. It also helps us propose a pipeline architecture that incorporates the best product capabilities to produce a comprehensive, goal-oriented training dataset.For example, we might propose a string of dependent pipelines that weave together categorization, 3D LiDAR cuboids, and LiDAR linking. Weather attributes and 3D data defects might be caught and categorized in the categorization pipeline, vehicles might be tracked in the 3D LiDAR cuboids project, and camera-based attributes like blinker status might be generated in the linking project.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2e43fe86-4639-495c-3096-34676043a300/public" alt="Using specialized label pipelines to turn multi-sensor inputs into training data. ">AV customers often come to Scale with data from multiple sensors and with an idea of what labels they need. Scale's team then architects specialized label pipelines that bridge the two, producing high-quality labeled data quickly and efficiently.<h1>We test and fine tune pipeline architecture by creating a golden dataset and vetting it with our customers</h1>Following this thoughtful approach to customer alignment helps us achieve a data milestone: the creation of a golden dataset. A golden dataset is used to define what ‘good’ looks like. It is used as a reference for us, our human Taskers, and customers’ own stakeholders.Creating a golden dataset is a journey of discovery. We set out to create the golden dataset by annotating a representative sample of data in the pipeline we architected. Annotating some data always generates observations and insights about nuances in the data and requirements, which we surface to the customer along with our recommendations. We then review the resultant annotated data with our customer, tying open questions and considerations back to the customer’s ML goals.Reviewing the proposed golden dataset together with the customer is key. For example, we were working with an autonomous drone customer who needed polygon labeling of flyable space, ie where their drones could safely navigate. We found ourselves drawing pixel-perfect polygons around every tree branch, only to have the customer reject every task. Only once we debriefed live with the customer did we realize that for them, quality training data meant polygons that avoided tree pixels by a wide margin - ie, rough, simple polygons that just didn’t clip the treetops! That one collaborative sync put the project back on track with a clear path to consistent, relevant, and accurate labeled data.Attaining this mutually-vetted golden dataset is powerful. It powers our internal (human) Tasker training and assessment tools, giving both Taskers and the operations team a clear understanding of what success looks like. It also aligns our Quality Assurance (QA) team, whose job is to effectively mirror customers’ QA teams to reduce customers’ operational data-management burden.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/ecd95dc4-c137-4a9e-4024-569dd8df3d00/public" alt="Achieving and leveraging golden data">Reviewing label samples together with customers helps establish a golden dataset, which in turn makes is possible to scale up production of high-quality labels.<h1>We engage all relevant stakeholders to ensure customers are aligned with Scale - and amongst themselves</h1>We’ve recognized that our customers are running complex initiatives involving many stakeholders on many teams, including operations, engineering, legal, and procurement. Alignment isn’t complete until Scale has aligned with each customer subteam, AND the customer subteams are internally aligned amongst themselves.Projects can fail if the customer’s operations team is grading labeled data according to a different set of standards than what the ML team expected. This can take many forms - operations might be grading too easily (for example, approving polygons that are too swaggy), or grading too harshly (for example, rejecting boxes with acceptable sizing), or just grading according to a different or outdated rubric (for example, applying an old definition of “truck” that included small pickup trucks).The result of inconsistent labeling standards is that the ML team gets training data that doesn’t improve model performance. Or, the ML team gets starved for data, even though there is data available that would have improved performance! Few things are worse than delivering weeks or months of data, with a consistent green light from the customer’s operations team - only to get a mass rejection from the ML team once it tests model performance.Scale drives stakeholder alignment by bringing the right people into the discussion, especially at steps that we know to be high risk. We also actively monitor for hints that there might be misalignment (like radio silence from some stakeholders at key milestones) and work with our customers to drive alignment internally and with Scale. We pride ourselves at being a knowledgeable partner who knows how to avoid situations that de-ccelerate the development of AI applications.<h1>Alignment is never won and done; we continue to re-assess alignment to our customers’ ML goals</h1>It takes a deliberate effort to maintain alignment. Very often, model evaluations lead customers to update their data requirements and labeling guidelines. Even when the labeling guidelines remain constant, customer data itself can change. For example, we once built a reliable high-quality pipeline to label street scenes, only to find the next batch of customer data contained scenes of people lying on a racetrack.Throughout the lifecycle of the data annotation pipeline, we continue to apply our alignment process. We continue to check in on customer ML goals, refine pipeline setup, maintain the accuracy of the golden dataset, and ensure alignment between teams on the customer, Scale, and Tasker side. This accelerates the speed with which our customers receive high-quality labeled data, which in turn accelerates their development of AI applications.To experiment with Scale’s data labelling at a small scale, sign up for <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a>, the fastest way to production-quality labels, with no data minimums. To set up custom pipelines for your ML needs, <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">talk to our team</a>.

We strive to understand what our customers ask for and why,  to label their data at high quality.

Partnering with customers to architect data pipelines

How Does Scale AI Partner with Customers to Produce High-Quality Training Data?

Every year, nearly 200,000 service members transition to civilian life. As a fighter pilot who served for over two decades in the U.S. Air Force, I know the challenges these men and women face. From adjusting to “normal” life, to mental health struggles, to determining a new career path, it can be overwhelming and difficult to navigate. One of my favorite things about Scale is our CEO’s hiring philosophy. Simply put, he says to hire people who give a shit. That philosophy has brought an amazing group of people to this company who not only care about technology, but also care about our country, our government, and our service members. As we prepare to celebrate Veterans Day, we wanted to share some of our Scale veterans’ stories to honor them and their commitment to our country, and to also offer advice for those navigating the transition from service to civilian life. <h1>What did you do in the military?</h1>Jason Schmid: I was a U.S. Army Military Intelligence Analyst, and deployed to Iraq twice. The first time was as a part of the Iraq Survey Group, determining the disposition of Iraq’s suspected weapons of mass destruction programs. On that mission, I was part of the team interrogating former high ranking Iraqi Regime officials. On my second deployment, I was embedded as a Tactical Intelligence Trainer in an Iraqi infantry battalion. Dan Tadross: I was an Air Traffic Control Officer responsible for coordinating aircraft requests and the deconfliction of airspace. In my last few years of Active Duty, I had the opportunity to serve as an Innovation Advisor to the Secretary of the Navy on how to integrate AI into Aviation Command and Control (specifically how to shorten planning cycles). I also had the privilege of being a founding member of the Department of Defense (DOD) Joint Artificial Intelligence Center (JAIC). During my two years at the JAIC, I ran a number of programs from supporting testing and evaluation to establishing Project Gargoyle for Force Protection. Brianna Hester: I was a Geospatial Analyst for 24 years. My most recent position was the Operations Superintendent of the European Partner Integration Enterprise (EPIE) where I managed a 40-person, international intelligence team operating a $5.7M Intelligence, Surveillance, Reconnaissance (ISR) coalition program for combat mission planning, execution, and overall intelligence gathering and research. I oversaw ISR mission operations and imagery exploitation of U.S. ISR platforms employed for collection of full motion video imagery in support of the DOD, NATO, and coalition forces. I deployed to Qatar and Afghanistan and also traveled to numerous other countries for my duties. Jacob Sheehan: I was a U.S. Army Special Forces Officer, also known as a Green Beret, where I deployed on combat missions to Iraq and Afghanistan and on training missions in India, Thailand, and Korea. In addition to leading operational teams, I also had an academic assignment at West Point where I taught economics courses, ran the graduate scholarship program (Rhodes, Marshall, Fulbright, etc.) and designed a two-semester critical thought curriculum. Matt Beck: I was commissioned as a Naval Officer in 2008. I spent my first two years as a Surface Warfare Officer on a destroyer. I was selected to attend SEAL training after that tour and completed training in 2012. I led teams of various sizes on deployments to Europe, Africa, South America, and the Middle East. Tyler Haley: I was a Special Operations Communicator in the U.S. Army for 10 years. I spent most of my career inside of Special Operations doing everything from tactical communications to designing and deploying data centers. I’ve spent a large portion of my career in support of overseas operations from Afghanistan, Iraq, Syria, to most of Central Africa. I was lucky enough to get to finish my career in doing what I love most, which is any and all things cloud! <h1>Why Scale?</h1>Jason Schmid: When I left government service, it was important to me to join a company that values its contribution to national security. All of the Scale senior leaders I spoke to in the interview process made it clear that Scale is dedicated to using its technology to serve the nation. I think it is hard to overstate how important the work Scale is doing for the nation, both on the commercial and federal sides of the business. Dan Tadross: One of the most challenging aspects for the U.S. government to adopt and implement AI responsibly is the lack of structured data. Scale is at the forefront of developing structured data and can do so at a volume and quality that can help catapult government efforts and ensure the competitive edge for the U.S. Brianna Hester: I was intrigued by the job description. When I opened it and looked at the requirements, I had all the requested experience so I applied. After chatting with Austin Woo about some more specifics of the position, I knew I wanted to be a part of this program. Jacob Sheehan: I was primarily drawn to Scale through my hiring manager, David Kinsella. I was headed towards working at a financial technology company in my job search, but David introduced me to Scale’s compelling vision in the AI industry and the role it could play in making the U.S. government more nimble to respond to pressing challenges. He viewed my military experience as an asset and imagined my future impact at Scale rather than solely focusing on my non-traditional business operations background. Matt Beck: I was drawn to Scale’s mission of accelerating the adoption of AI. A lot of the work we are doing on the federal team has a direct application to our nation’s defense, and I wanted to be a part of that work. Tyler Haley: I love the team dynamics of start-ups. Everyone wears different hats depending on the meeting or the time of day. This is super relatable to being on small unit teams inside of the Special Operations community. Scale, in particular, pushes the boundaries inside of technology and solves hard problems, which is exactly what I did in the U.S. military. <h1>What do you do at Scale?</h1>Jason Schmid: I am Scale’s Head of Government Relations. I work to help Congress understand AI and ML capabilities so that they can build programs and policies that will help the U.S. government accelerate the adoption of AI. Dan Tadross: I am an Engagement Manager helping to bridge the gap between U.S. government requirements and experimentation and Scale’s expertise. Brianna Hester: I am the Synthetic Aperture Radar (SAR) Imagery Intelligence Subject Matter Expert (SME). Jacob Sheehan: I’m a Business Operations and Strategy Manager that supports the federal team. My main projects include growing a compounding network of labeling experts and labeler performance management. Matt Beck: I am a Federal Deployment Strategist focused on developing solutions to support the modernization of government systems. Tyler Haley: I am the Infrastructure and Platform Lead for our federal team. I partner closely with our software engineering and security teams to help design infrastructure that works on various AWS classification regions. <h1>How was your transition from service to civilian life?</h1>Jason Schmid: I transitioned from the Army 15 years ago, and the circumstances of my transition were pretty unique. I went from being in literal combat to being a civilian in about three weeks. I was very fortunate to have a job lined up already, but it would have been much better if I had time to take advantage of all of the resources available to transitioning service members. Such a jarring transition was not easy. We now know so much more about successfully transitioning veterans back into civilian life that would have made my transition so much smoother - not just for me, but for my family as well. Dan Tadross: I transitioned off Active Duty service after 12 years during the height of the pandemic in 2020. So the transition process was stressful and challenging, but when I eventually landed at Scale my level of apprehension subsided. Brianna Hester: My transition was pretty easy, as I was ready to begin my post-military life. The hardest thing to get used to was not having to tell someone my every move. Taking PTO without telling someone the address of where I would be for the duration of the PTO was a little unnerving the first time. Jacob Sheehan: I gave myself sufficient time, but there seems to be no such thing as too much time when it comes to major life transitions, especially if you have a family to take care of! I’m glad I had the space to think about what I wanted to do next instead of having to manufacture it quickly. Transitioning during COVID was a challenge from the standpoint of young kids being around all the time, so I would recommend finding a quiet space to do your transition thinking. Matt Beck: I had a wonderful transition departing the military. I met Shands Pickett early on in the process and kept in touch as positions at Scale became available. My preparation and exploration of many different post-military careers went well and resulted in me working at Scale. Tyler Haley: Transitioning was a bit tough for me. I got out of the service in May 2020 right as the pandemic was really setting in. Transitioning from being in a room with people that you are close with to only seeing people via Zoom and Slack was difficult at first. I was accustomed to seeing my daily contributions make a real impact, but when I got out, that faded away rather quickly. The good part about transitioning was being able to be at home and start a family. My wife and I welcomed our first kiddo into the world shortly after I got out. It gave me a new purpose and perspective in life. It helped me fill the void that the military left. <h1>Can you speak to your experience as a veteran at Scale?</h1>Jason Schmid: I think one of the best qualities of Scale’s culture is a genuine curiosity and passion to solve hard problems as a team. I think anyone who has served can connect with that. Dan Tadross: One of the aspects I miss most about serving in the U.S. Marine Corps is the comradery and sense of purpose. While nothing can compare with serving in a Marine unit, Scale has a culture that shares some similarities and the team I am a part of is motivated to succeed and support the U.S. government toward the responsible adoption of AI. Brianna Hester: Scale has become my new family. I still have my military family and always will but now I have added another branch to my family tree with everyone from Scale. They may be very different families but I wouldn’t trade either of them for anything and I can’t wait to see how much we accomplish together! Jacob Sheehan: Spending time with the many veterans on the federal team has maintained some of the camaraderie I enjoyed in the U.S. Army. I’m also pleasantly surprised how genuinely curious Scaliens across all of Scale are in veteran experiences! Matt Beck: Scale is extremely supportive of its veterans. My work keeps me engaged on a mission I believe in. Furthermore, Scale is extremely flexible in letting me maintain my connection to the U.S. Navy by supporting my drill requirements. Tyler Haley: Although I am still new to the team at Scale, I felt value-added from day one. As I mentioned earlier, feeling like what you do matters is a huge point of pride for most veterans. I instantly felt as if I was a part of the team and also was surrounded by other team members that have served. At Scale, you really get the experience of being back on those small teams that have a mission and purpose. <h1>What advice would you share with veterans making the transition?</h1>Jason Schmid: Take advantage of every resource available to support your transition, whether you think it applies to you or not, because you’ve earned it. It’s normal for recently separated veterans to feel some loss of purpose. It’s important to find ways to continue serving something larger than yourself, even if it's not part of a career path. Dan Tadross: There are a lot of opportunities for those who want to find alternative ways to continue to support the U.S. government. Many of the skills I learned in the U.S. Marines continue to help me to succeed in the private sector. Those experiences and capabilities that made you successful as a service member are in demand in the private sector, so when making the transition, don’t forget the value you bring to any organization. Brianna Hester: My recommendation would be to take a break and relax for a minute before jumping right into another job, if you can. It’s a huge change going from complete structure in your everyday life to no structure, and it can be hard on some people. Enjoy a small (or even large) break and use that time as a way to prepare yourself for the great things to come. Jacob Sheehan: Spend a lot of time exploring the industry you want to work in, understanding the companies uniquely positioned in that industry, and thinking about the strengths you can bring to roles in those companies. I’ve had many veteran friends get it backwards by over-indexing on the role (title) and barely giving any thought to the company or larger industry. Matt Beck: Always seek to improve your position as you look for opportunities. Forecast your worst case scenario out as far as possible. Once you are comfortable with your worst case scenario, seek out the roles and companies that you are most excited about and get connected early. Tyler Haley: Figure out what fills your voids. Find a routine and stick to it. A lot of the service members that I speak with feel the same way. Whether that’s getting up every morning and hitting the Peloton or cooking dinner for the family every night, find something that you can do for YOU. Find a new skill or hobby and try to master it. Get out of the house and away from the keyboard. Find time in your days to step away and get lunch with another veteran. Transitioning is different for everyone. Make time to see those who understand what you are going through. If you are interested in joining our team, check out our <a href="https://scale.com/careers#jobs" rel="noopener noreferrer" target="_blank">open positions</a>.

As we prepare to celebrate Veterans Day, we wanted to share some of our Scale veterans’ stories.

Scale Team Spotlight: Veterans at Scale

<h1> </h1><h1> Why build an ML-based workflow for data labeling?</h1> Delivering high-quality data labels quickly is a challenging task. Even for humans, image and video data commonly found in self-driving or robotic applications can be compositionally challenging to parse and correctly annotate. At Scale, we use machine learning to augment human workflows to both improve the quality and speed of labeling. Since deep learning models can struggle to perform consistently across diverse data domains like self-driving car scenes, finding the optimal balance between ML automation and human supervision is necessary to ensure we maintain high-quality.Autosegment, an ML-powered tool that can accurately segment instances within a human-defined bounding box, allows us to delegate high-level reasoning about semantics and taxonomy to Taskers (human labelers) and basic pixel-level processing to an ML model. This division of labor enables us to build a general purpose and accurate ML labeling tool. Despite using a simple neural network, Autosegment outperforms prior tools.<h1> </h1><h1> The Challenge of Semantic Segmentation Labeling</h1> Semantic segmentation involves human Taskers identifying and labeling every pixel in a visual scene — a notoriously time-consuming and tedious process. Most Tasker time in this manual approach — about 95% — is consumed by time-intensive work, such as carefully tracing the outlines of objects in a scene. For example: it takes roughly one minute to trace the outline of a car and only three seconds to click on that outline and select the label “Car.” What if we could instead use machine learning to automate the most laborious parts of semantic segmentation? What if an ML model did the semantic segmentation, and the Tasker only had to select the right label?<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/72c723b3-46ae-4384-b75b-4a87b5e40b00/public" alt="Semantic Segmentation Task"> A visual scene which has been semantically segmented. Each pixel is color coded to denote the class of object it represents, i.e. Pedestrian, Road, Bike, etc. As ML engineers, we rely heavily on quantitative validation metrics, such as label accuracy and intersection-over-union (IoU), to determine the quality of our models. However, when building models that optimize for a particular business outcome — for instance, reducing the time it takes Taskers to semantically segment images — we need to think critically about which model to use and how we measure its performance.. In this example, our objective is to produce predictions that streamline the Taskers’ job. This is not necessarily the same as “best model performance.” For example, an instance segmentation model might be consistently inaccurate by just a few pixels around an object. This model might have a high IoU score but very little practical usefulness because Taskers would need to re-do every trace of the outline. Conversely, a model that can perfectly segment an object, even if it only identifies half of it, would still be extremely useful. Despite having lower IoU validation scores, it’s still an improvement to the overall workflow. Because it takes so long to trace outlines manually, having a model produce pixel-accurate partial outlines is still a significant time saver. <h1> </h1><h1> The Active Tooling Paradigm and the Autosegment Tool</h1><h1> </h1>Autosegment is a machine learning tool that segments the most salient object within a human-provided box annotation. It incorporates Tasker feedback at annotation time in a way that is flexible and iterative. When building Autosegment, we realized early on that the only way to provide flexibility and high quality was to give Taskers access to Active Tools. Active Tools let Taskers make changes to the model inputs and parameters in real time to generate the best output for their use case. This is similar to “prompt engineering,” the process by which we can coerce a neural language model like GPT-3 to generate desired text when given precise prompts. For more information, OpenAI hosts <a href="https://beta.openai.com/docs/guides/completion/prompt-design" rel="noopener noreferrer" target="_blank">excellent documentation</a> on how to design prompts for their GPT-3 model. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/07588dc3-6a06-45e7-c61f-a51562470900/public" alt="Open-AI GPT-3 Model"> Humans working alongside models can produce better results faster than models alone Using Active Tools, we give Taskers as many options as possible to make their tasks easier. For example, they can adjust the annotation box shape and the prediction confidence threshold to configure the model to speed up their particular labeling task. <img src="https://cdn.builder.io/o/assets%2Fe0438815ba51486bbb6a202747122d4b%2Ffef154571d5846b6b87674a46a123ac5%2Fcompressed?apiKey=e0438815ba51486bbb6a202747122d4b&amp;token=fef154571d5846b6b87674a46a123ac5&amp;alt=media&amp;optimized=true" alt="Autosegment Tool Demo">A demo of the Autosegment tool. Enabling humans to freely explore the output space of the model is a key advantage of active tooling. We've already seen how this is useful. Overall, we have observed the median annotation time per scene is about 30% faster with Autosegment. We also interviewed Taskers to see how they are using the tool. While our model is trained to segment cars and motorcycles, one Tasker used the tool to segment everything. While this wasn’t a workflow we had envisioned, it halved their time per scene segmentation, from 40 minutes to 20 minutes. In a discussion we had with five Taskers, they explained that they consider Autosegment to be more accurate and efficient than previous ML solutions. Now we can simply supply robust and general online models and let Taskers optimize the rest. <h1>The Autosegment Model</h1>Autosegment is a convolutional neural network trained to segment the most salient instance within a Tasker-defined bounding box. This simple design allows us to significantly reduce our customers’ operational overhead to deploy ML, while providing pixel-perfect instance segmentations, even on previously unseen instance types. The Autosegment model does not have to be particularly complex to produce compelling results. Autosegment has no notion of different semantic types; instead, it abstractly identifies and segments based on scene composition. The model is trained to always generate a segmentation. The training data includes a few key classes like pedestrians and vehicles. The task is binary; it simply determines whether each pixel belongs to the salient object or not. We don’t train the model with negative samples. Instead, the model makes its best guess on each pixel to generate a segmentation. We define the salient instance as the object that owns the central pixel of the crop, so even if the scene is cluttered, the model can identify the target instance to segment. This binary segmentation task across instance classes and types allows the model to generalize well on new classes and taxonomies. Indeed, we can transfer not only between objects of the same class, such as cars in different countries, but between different classes altogether — for instance, moving from cars to motorcycles to street signs. This transfer can be achieved with limited retraining or, in some cases, no retraining at all, achieving zero-shot transfer. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/1a4x5jij9v"></iframe> Using salient object detection on specific crops of the scene can generate even more accurate segmentation. We were able to double the edge precision score from 0.26 to 0.52 when compared to off-the-shelf full-scene semantic segmentation models. Instead of supplying a downsampled version of the whole image to a neural network, the selected region is fed to the model at full resolution. Because the input is smaller, fewer pooling operations are required to achieve a globally receptive field. As a result, there is less downsampling and blurring. Our model achieves better fidelity around the edges of instances, a crucial part of providing high-quality data annotations to our customers. Improving full-scene labeling to the same degree would require exponentially more effort and complexity. In summary, we developed Active Tooling to create an ML-based workflow capable of solving basic computer vision problems. This workflow helps our Taskers reach higher levels of quality faster than in previous workflows. Innovations like Autosegment are critical to enable the “Moore's Law of data labeling.” At Scale AI, we’re excited to push the boundaries of human-AI interaction for our customers.

Scale AI accelerates image annotation with semantic segmentation active tooling. 

Reducing image labeling time by up to 30% and doubling segmentation quality.

Faster Semantic Segmentation with Machine Learning-Assisted Workflows

AI is no longer the exclusive domain of large enterprises. It&rsquo;s entering the business mainstream&mdash;but it&rsquo;s successful only sometimes, says Salesforce President and Chief Operating Officer Bret Taylor. A thoughtful AI strategy should be customer-focused, with clearly defined business outcomes, and the key to all of this is data.
In his talk at the <a href="https://scale.com/events/transformx" target="_blank" rel="noopener noreferrer">Scale TransformX 2021 conference</a>, Taylor discussed the key considerations that businesses of all sizes should keep in mind to ensure that they delight their customers and achieve sustainable business outcomes.
<iframe class="ql-video" src="https://fast.wistia.com/embed/medias/42dyrzqs4c" frameborder="0" allowfullscreen="allowfullscreen"></iframe>
Here&rsquo;s a summary of the seven key takeaways from Taylor&rsquo;s presentation. For more details, watch the session video above.
<h3>Business Today Is All About the Data</h3>
&ldquo;If you don't have a digital business, you don't have a business,&rdquo; says Taylor.
Enterprises need to maintain constant engagement with customers, supply chains, partners, and employees, in granular detail. Each of these interactions produces data that businesses can use to create more intelligent and personalized relationships. This <a href="https://insidebigdata.com/2017/02/16/the-exponential-growth-of-data/" target="_blank" rel="noopener noreferrer">growing tsunami of data</a> requires intelligent algorithms and automation to make sense of it all. As Marc Andreessen, co-founder of venture capital firm Andreessen Horowitz, famously said, <a href="https://a16z.com/2011/08/20/why-software-is-eating-the-world/" target="_blank" rel="noopener noreferrer">&ldquo;Software is eating the world.&rdquo;</a>
Today, however, it&rsquo;s AI that has become central to everything that a company such as Salesforce does.
<h3>Build a Data Culture</h3>
AI and deep learning can&rsquo;t be the exclusive domain of technology companies. To get there, every company must strive to <a href="https://www.mckinsey.com/~/media/McKinsey/Business%20Functions/McKinsey%20Analytics/Our%20Insights/Why%20data%20culture%20matters/Why-data-culture-matters.ashx" target="_blank" rel="noopener noreferrer">develop a data culture</a> that allows employees to see and understand the flow of information moving through the business. A <a href="https://www.gartner.com/smarterwithgartner/do-you-need-a-chief-data-officer" target="_blank" rel="noopener noreferrer">chief data officer (CDO)</a> should act as a steward for all that information, without trapping data in internal silos. Instead, everyone needs to be empowered to make data-informed decisions based on objective views of customers, supply chains, and the flow of resources.
Ultimately, companies that create open data cultures will easily outcompete those that don&rsquo;t.
<h1>Ethical AI Builds Trust</h1>
Shifting toward a data culture requires building in <a href="https://www.salesforce.com/blog/what-is-ethics-by-design/" target="_blank" rel="noopener noreferrer">ethics by design</a>, in much the same way that enterprises have incorporated <a href="https://patchstack.com/security-design-principles-owasp/" target="_blank" rel="noopener noreferrer">security by design</a>.
To deliver an ethical deployment of AI, the number one value must be trust. With that in mind, Salesforce created the position of <a href="https://www.cnbc.com/2018/12/10/salesforce-hires-its-first-chief-ethical-and-humane-use-officer.html" target="_blank" rel="noopener noreferrer">chief ethical and humane use officer</a> to guide the company&rsquo;s adoption of AI.
Salesforce wanted to ensure that all of its AI is responsible, accountable, transparent, empowering, and inclusive. Taylor contends that failing to take this into account from the start inevitably leads to unintended negative consequences and general degradation of trust in technology.
Society is now holding software and AI to a higher standard. Companies that develop AI today must embrace conversations about ethics and trust, and they must embed those conversations into their development processes. Building trust also requires bringing in stakeholders who aren&rsquo;t data scientists to help shape the responsible use of these technologies.
<h3>Focus on Strategy, Not AI for AI&rsquo;s Sake</h3>
While every company needs to become an AI company to remain competitive, too many organizations are succumbing to the hype about digital transformation without having <a href="https://www.thetilt.com/content/stages-of-digital-transformation" target="_blank" rel="noopener noreferrer">clear goals</a> in mind. According to the <a href="https://www.gartner.com/en/information-technology/insights/cio-agenda" target="_blank" rel="noopener noreferrer">2021 Gartner CIO Agenda survey</a>, while 94% of CIOs are working to deploy AI, only a quarter have done so successfully.
&ldquo;Don&rsquo;t start with the technology; start with customers,&rdquo; Taylor says. Focus on their experience and what your customers need and want. Clarify the project&rsquo;s scope and look for the easy wins first. &ldquo;If AI becomes the purpose of the project, it becomes a bottomless pit of investment,&rdquo; he says.
<h3>One Shortcut to Good AI: Don&rsquo;t Build It; Buy It</h3>
Salesforce&rsquo;s <a href="https://www.salesforce.com/products/einstein/faq/" target="_blank" rel="noopener noreferrer">AI platform Einstein</a> is a prime example of Salesforce focusing on building AI technology that is customer-focused and that &ldquo;can be turned on with clicks, not code,&rdquo; says Taylor. Salesforce customer <a href="https://www.salesforce.com/blog/orvis-ai-customer-engagement/" target="_blank" rel="noopener noreferrer">Orvis uses Einstein</a> to do simple analyses of the potential engagement of its marketing messages prior to their publication.
That approach increased Orvis&rsquo; click-through rate by 22% and boosted traffic by 30%. This is one way an enterprise can deploy customer-focused experiences with the help of AI, without having to invest in the substantial skills and technology normally associated with developing it.
<h3>Align Your AI Investment with Business Priorities</h3>
A common key strategy, especially for B2C enterprises, starts with identifying use cases that increase personalization and automation, thus streamlining the customer experience. Then, identify the data and AI tools that could help create those use cases.
For its part, Orvis&rsquo; goal is to keep its support calls short, and <a href="https://www.salesforce.com/news/press-releases/2019/11/19/hey-einstein-salesforce-brings-voice-to-every-customer-experience/" target="_blank" rel="noopener noreferrer">Einstein can help by transcribing and analyzing customer questions</a> and suggesting answers to call center personnel in real time.
The lesson here: Bring in just a few machine learning experts and combine them with off-the-shelf SaaS platforms to discover unique value in AI that can be customized around your business.
<h3>Don&rsquo;t Forget the Quick Wins</h3>
Getting value from AI isn&rsquo;t science fiction, Taylor says. Simply step back and focus on the low-hanging fruit to achieve tangible business outcomes.
<h3>Learn More</h3>
To learn more about how Salesforce built its AI strategy and lessons learned,&nbsp;<a href="https://exchange.scale.com/public/videos/enterprise-ai-strategy-bret-taylor-on-lessons-learned-at-salesforce" target="_blank" rel="noopener noreferrer">watch Taylor&rsquo;s talk and view the full transcript here</a>.

Customer-first AI strategy starts with clear goals and reliable data — the key to driving real business outcomes.

AI Strategy: Data & Customer Focus

7 Elements of a Well-Thought-Out AI Strategy: Salesforce COO Bret Taylor

img_fallback.png

Dr. Li shares explores how recent AI research helps robots  perform both short-horizon and long-horizon tasks.

Dr. Fei-Fei Li On Teaching Robots Complex Tasks - TransformX 2021 Conference - Scale AI

On Teaching Robots To Perform Complex Tasks with Dr. Fei-Fei Li 

Today we are excited to welcome SiaSearch, a leading data management platform, to Scale. We acquired SiaSearch, a spin-off of the European AI venture studio Merantix, to enable us to expand Scale’s suite of tools for every AI team, broaden our footprint in European markets, and deepen the technical expertise of the company. We know that the autonomous vehicle (AV) industry is one of the best breeding grounds for AI technology. SiaSearch’s Berlin-based team brings experienced, high-caliber engineers from the European AV industry to Scale, which will help us deliver world-class AI solutions to support the complete machine learning lifecycle. The SiaSearch team will join Scale's Nucleus business unit as we grow our platform to support more data types and use cases across a wide array of industries and markets. SiaSearch’s unique interval querying engine, which is inspired by algorithms originally designed for large-scale analysis of DNA sequences, allows for lightning-fast search across massive video and LiDAR datasets and complements what Scale is building with <a href="https://studio.plasmic.app/scale.com/nucleus" rel="noopener noreferrer" target="_blank">Nucleus</a>. We are excited to weave the SiaSearch technology into the Nucleus product, accelerating development and improving the developer user experience. We see even more potential down the line to incorporate SiaSearch’s world-class team and technology throughout our entire suite of products. Merantix and the SiaSearch team have a strong network in Europe and have partnered with the world’s leading automakers, including Porsche and Volkswagen. We’re thrilled to expand Scale’s footprint to Germany and look forward to working with SiaSearch’s customers and networks to significantly increase our ability to serve European-based customers. We’re excited for what the future holds with this latest addition to our team, and we’re always looking for new opportunities. If you are interested in joining this incredible team, check out our <a href="https://scale.com/careers#jobs" rel="noopener noreferrer" target="_blank">open positions</a>. Or, if you’re interested in partnering with us, please <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">reach out</a>.

Scale has acquired SiaSearch to further enhance Nucleus, its data management platform.

The SiaSearch team and Nucleus will collaborate on Dataset Managment for Nucleus.

SiaSearch Acquisition: Delivering the Full Data Engine to Our Customers

Waymo’s co-CEO Dmitri Dolgov shares Waymo’s approach to building powerful machine learning models for self-driving vehicles

Dmitri Dolgov of Waymo Discusses Data-centric AI - TransformX 2021 Conference - Scale AI

Dmitri Dolgov of Waymo Discusses Data-centric AI

The co-founder of Schmidt Futures and a former CEO of Google, discusses how AI will shape our global future. Eric was joined by Scale AI CEO Alexandr Wang

Eric Schmidt Discusses the Geopolitics of AI - TransformX 2021 Conference - Scale AI

Eric Schmidt Discusses the Geopolitics of AI

Justin Basilico, Director of Machine Learning and Recommender Systems at Netflix describes how ‘everything at Netflix is a recommendation’.

Netflix on Recommendations and Personalization - TransformX 2021 Conference - Scale AI

Netflix Explains Recommendations and Personalization

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

custom-fonts

homepage-event

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

summit-2023-speakers

customers

Display order on the page /customers. Higher values show first

summit-2023-agenda

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Blog