experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

People

Engineering

When setting up a new machine learning (ML) system, collecting and annotating data can be a time-consuming process that requires a significant upfront investment. Since data quality is strongly correlated with ML model performance, it’s crucial to invest in high-quality solutions for data annotation from the onset of your project. The process of annotating data for an ML project can be surprisingly complicated – There are a variety of factors you’ll need to consider, from how you’ll manage your approach to annotation to what tooling and automation approaches you’ll use.<h2>How to Build Your Data Annotation Pipeline</h2>Annotating data at high quality requires significant upfront and continued investment over the lifespan of your ML system. While there are automated solutions to speed up and streamline the data annotation process, having humans-in-the-loop is key to producing high-quality results. When setting up a data annotation pipeline, your company will need to decide how you’ll find and train annotators, as well as how you’ll measure their performance. To ensure annotators produce high-quality annotations, you’ll need to provide them with performance incentives that align with your most important annotation metrics, whether that’s throughput or quality, or a combination of both. You’ll also need to provide annotation tools for your workers, as well as design a pipeline that encourages your annotators to output consistent, high-quality annotations.<h3>Hiring Annotators While Minimizing Overhead</h3>Finding annotators to complete your task is one of the first challenges you’ll face in the data annotation process. You’ll need to identify workers who are well-suited to annotate your data, whether that involves selecting people who speak a certain language, are from a specific location, or have a particular type of expertise. You’ll also need to decide how to approach the hiring process. Your choice of workforce influences whether you have access to a large selection of annotators who work fewer hours, or a small selection of annotators who work longer hours and generate more consistent results. If you expect a continuous flow of production-quality data, you’ll also need to potentially hire workers across time zones to meet the demands of your project.&nbsp;You should decide whether to hire a larger pool of diverse annotators versus a smaller set of consistent annotators based on your use case. If you anticipate that you’ll need annotations for small, irregular projects, for example, you’ll likely want to rely on a larger selection of annotators that can accommodate short-term surges in need. Alternatively, if you’re pursuing a long-term production project with consistent annotation needs, a smaller team of annotators with regular annotation hours may deliver better results.Once you’ve hired annotators, you’ll also need to devote resources to communication and expectation setting to ensure that your annotators provide you high-quality results. In short, just finding workers to annotate your data and keeping them on your platform can be a time-consuming process with logistical overhead! These challenges are often why many ML teams opt to outsource the entire process to an external company.<h3>Training Annotators and Monitoring Performance</h3>Once you’ve hired your annotators, your work is not yet over – You need to provide training to teach them to accurately annotate your data across different use cases, task types, and annotation types. To ensure high-quality annotations, you’ll need to provide workers with clear and comprehensive instructions, so that different workers can annotate data consistently. You may also find over time that there are several edge cases within your data that your instructions do not cover, as it can be difficult to write non-ambiguous and comprehensive instructions that cover all potential edge cases.As the administrator of the data pipeline, you’ll need to create a “golden dataset” that has accurate annotations for a subset of your data. You can use this dataset to check that workers understand the instructions, as well as to periodically re-evaluate their performance against a gold standard. You should also regularly update this dataset with new examples to provide annotators with up-to-date examples that reflect real-world data.&nbsp;Suppose your goal is to achieve 95%+ quality in your overall dataset. If all your annotators consistently achieve 95% quality, you should be able to achieve this goal. However, if there is a particular edge case that only appears 1% of the time, your annotators can get 100% of that 1% wrong and still average 95% quality overall. While you are still hitting your goal of 95%+ quality, this is not ideal, as your model will not be able to account for this edge case if trained on this dataset.To mitigate this challenge, you will need to add examples to your golden dataset to ensure that it remains well-balanced across all classes and edge cases. By updating and adding new examples to your dataset, you can effectively assess annotator performance over time, and achieve a holistic view of their performance across various classes and edge cases.<h3>Reviewing Annotations for Quality</h3>After your annotators complete tasks, you’ll need to evaluate the final output. To do this, there are two main methods for evaluating annotation quality. For classification or text-based tasks where there is more subjectivity, you can implement a consensus algorithm. Suppose you want to develop a sentiment analysis model, you can have multiple people determine if a person in an image looks happy or sad and the option the majority selected will win. If there isn’t enough confidence in the voted result, you can have a higher-level reviewer determine the final result. In some cases, you may also not want to weigh every person’s vote the same. If you have higher confidence in some people’s votes over others, you may want to implement a weighted voting system that biases the final response more towards the votes of people you have higher confidence in.&nbsp;For image-based or other computer vision tasks, you can have one preliminary attempter annotate the task, then have a higher-level reviewer either review the work and accept the results or review the work and fix annotations.&nbsp;You can select reviewers from past annotators who consistently produce high-quality results. Selecting reviewers, however, can be complicated, since someone’s annotation performance can fluctuate over time. You’ll need to constantly identify reviewers who are no longer performing well so that you can demote them to attempts, while also identifying well-performing annotators so that you can promote them to reviewers. Since this approach removes high-quality annotators from the labeling pool, it will naturally reduce your throughput, so you’ll want to find a balance between quality and throughput.Whatever approach you follow for evaluating label quality, you’ll need to be on the lookout for people who consistently underperform and label data poorly. You should automatically identify underperforming annotators, and either retrain them so they produce higher-quality labels, or remove them from the project entirely.<h3>Selecting the Best Annotation Tool For Your Task</h3>Another big decision you’ll need to make is on what labeling tools your workers will use. You can either build your own tool set or use available commercial tools. While you can specialize in-house tools for your specific task, they can be difficult and time-consuming to build, especially if they need to support different annotation types. For example, image, audio, and text data all require different annotation approaches – For image annotation, you’ll need a system to draw boxes, polygons, or other geometries around identified objects; whereas, for audio annotation, your system will need to support transcription and associating timestamps with event attributes. For classification tasks, on the other hand, your system will need to permit multiple-choice selections and potentially large taxonomies. For video annotation tasks, video files can be very large in size, making them technically challenging to support, especially if your workers have less powerful laptops or slow internet connections.&nbsp;&nbsp;Compared to in-house tooling, commercial data tools make it easier to jump into labeling your data, although they may be less specialized for your use case. Another consideration when selecting the right annotation tool is ML-assisted tooling. Some tools may have ML-assisted tooling, like our <a href="https://scale.com/blog/autosegment" rel="noopener noreferrer" target="_blank">Autosemseg tool</a>, to improve annotation efficiency and speed while others do not. Building and selecting the appropriate labeling tools is vital for optimizing labeler efficiency, so you should ensure the tools you select are well-suited for your use case.<h2>What Next?</h2>Needless to say, getting data annotation right can be a challenging and time-consuming process. You’ll need to consider a wide range of factors, from how you’ll manage your labeling workforce to what automation approaches you’ll use. Your data labeling pipeline is a process that’s incredibly important to get right since ML system performance is correlated with data quality.Instead of building a data pipeline from scratch, you can leverage a product like <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Scale Rapid</a> or <a href="https://scale.com/studio" rel="noopener noreferrer" target="_blank">Scale Studio</a> to quickly create production-level ratings. It’s designed so that you can set up your projects in minutes and get high-quality labels within hours. Trusted by AI teams like OpenAI, Square, and Pinterest, this tool can help you hit the ground running on your next ML project!

image.jpg

Data annotation, or data labeling, is a more challenging problem than most teams realize but investing in annotation is key to unlocking the power of AI.

Scale AI’s How To Guide to Data Annotation/Data Labeling

How to Setup Data Annotation Pipelines

At first glance, human-in-the-loop processes seem like a relic of older times. Before technological advancements, humans effectively performed functions completed by computers today. For example, NASA hired large teams of arithmetic processors- men and women whose only job was to evaluate numerical expressions quickly and accurately. These calculators were pivotal to enabling space exploration.At Scale, our labeling teams are just as critical to achieving our mission. We’ve found we’re able to effectively apply the principles behind generative adversarial networks to our human-in-the-loop processes to deliver the highest quality annotations to customers.<h2>How does this work, exactly?</h2>A GAN, or generative adversarial network, is given an unlabeled dataset and learns to produce new examples that are not already in the dataset but are indistinguishable from the items in the dataset. For example, a net meant to produce new images of flowers would be fed as input images of real flowers. To achieve this goal, the net has to achieve two things:<ol><li>The generated flowers must not be distinguishable from real flowers. That is to say, if given a set of both generated and real flower images, one should not be able to tell which is which.</li><li>In achieving the above, the GAN must not simply produce copies of the real flower images.</li></ol>Therefore, the net must be trained to generate new flowers, discriminate between real and fake flowers, and avoid simply replicating training data. The training loop jointly optimizes the generator (which produces new images that are not copies of the training set), and the discriminator (which attempts to discriminate between training data and generated images). The net is successful when the generator becomes accurate enough to fool the discriminator, i.e. the discriminator cannot outperform random choice when classifying generated vs non-generated examples.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F018ccca0549b448bb3686f7d5703244e">In the context of human-in-the loop processes, we can apply this same principle to produce quality labels on subjective tasks. Let’s say the pipeline we’ve chosen has a minimum of two annotators seeing each task, one attempter and one reviewer.Example Task:<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F4a26d7e4818f49dfa8acd7616a2bf50c">Instructions: Please describe the image as accurately and descriptively as possible with a short phrase.Possible attempter responses: <ul><li>Attempter 1: ‘asdf’</li><li>Attempter 2: ‘trees’</li><li>Attempter 3: ‘green hills’</li><li>Attempter 4: ‘grassy hills with mountains, trees, houses’</li><li>Attempter 5: ‘grassy hills with trees and 3 houses, mountains and clouds in background’</li><li>Attempter 6: ‘tall trees on green grassy hill, small red and gray houses, blue mountains in background’</li></ul>The attempters act as generators, and can generate any one of the above responses for this task. Attempters 1-3 have written low-quality responses. On a regular rapid project, these attempters would be disabled via evaluation tasks. However, on a generative task like the above, exact-match evaluation tasks would not guarantee quality. For this image, both Attempters 5 and 6 provided accurate and descriptive responses. Therefore, we turn instead to the discriminator to filter out the better responses.Our discriminators are reviewers. A reviewer is the final person to look at a task and is able to see the attempters’ responses. Reviewers are the final stage before tasks are returned to the customer If we allow reviewers to edit an attempter’s response directly and write a caption themselves, we run into the same problem as with attempter evaluation tasks.&nbsp;However, once we move more into thinking of the reviewer as a GAN discriminator, the solution becomes clear. Reviewers must discriminate between good and bad responses, but they need not write any captions themselves. Although there is no 1:1 mapping between a task and a certain correct answer, we do know upon encountering a caption whether it is acceptable or unacceptable. So we give the reviewers the choice of either accepting or rejecting an attempter’s response, but we do not give them the option to change it. Now the process of evaluating their quality is simple- create evaluation tasks with pre-generated responses that are expected to be accepted or rejected appropriately.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fb8be702f8ef547cba93bf016ee11d500">The two parts of a GAN, generator and discriminator, work together to pass information between them. In our human network, reviewers may provide feedback upon rejecting an attempt, to help the generator learn. The discriminator, composed of both the reviewers and the reviewer benchmarks managed by the customer, learns in turn from the generator. From a variety of generated data, the customer and in turn the reviewers are able to converge on a similar understanding of the desired result.Now that we are sure the discriminator is accurately discriminating between low and high-quality responses, we are able to train the generator. What this means is that attempters are now disabled based on how many of their responses are rejected. Over time, this leads to a state where the attempters are effective, high-quality generators- essentially a trained GAN.This pipeline is available via Rapid’s <a href="https://docs.scale.com/reference/text-collection" rel="noopener noreferrer" target="_blank">text collection</a> and <a href="https://docs.scale.com/reference/categorization" rel="noopener noreferrer" target="_blank">categorization</a> projects. <a href="https://scale.com/rapid" rel="noopener noreferrer" target="_blank">Get started with Scale Rapid</a> today!

Applying the principles behind generative adversarial networks (GANs) to our human-in-the-loop data annotation processes.

Using generative adversarial networks (GANs) to align human metrics

Using GANs to Align Human Metrics

At<a href="https://scale.com/defense" rel="noopener noreferrer" target="_blank"> Scale</a>, we are committed to partnering with the U.S. government and technology leaders to ensure U.S. investments in AI solve critical challenges. It’s imperative the U.S. government has access to, and is deploying, the best quality data and AI technology available.&nbsp;That's why we are excited to announce Scale has partnered with <a href="https://www2.deloitte.com/us/en.html" rel="noopener noreferrer" target="_blank">Deloitte</a>, an awardee of the Joint Artificial Intelligence Center's Data Readiness Artificial Intelligence Development (DRAID) Basic Ordering Agreement (BOA). The JAIC DRAID is a five-year, $241 million BOA that enables Scale and Deloitte to support AI data preparation for the U.S. government. This partnership brings nearly $500M in&nbsp;<a href="https://www.businesswire.com/news/home/20220131005304/en/Scale-AI-Selected-by-U.S.-Department-of-Defense-to-Accelerate-Government%E2%80%99s-AI-Capabilities" rel="noopener noreferrer" target="_blank">total JAIC ceiling value</a>&nbsp;to Scale. Scale’s technology will be readily available to all federal agencies and will empower the government to feel confident in leveraging AI and ML in their most important workflows.With this partnership, Scale will provide services focused on data management, AI modeling and innovative development including:<ul><li>High quality data integration, prep, labeling and management</li><li>Synthetic data generation&nbsp;</li><li>NLP-enabled processing</li><li>Algorithm management</li><li>Innovative AI application development</li></ul>Scale’s data-centric approach to its end-to-end AI platform provides the U.S. government with the support needed to manage the entire AI lifecycle, all while ensuring the highest quality data is incorporated. Using our technology, analysts and operators will be able to understand, visualize, curate, and collaborate on their data through process automation, enhanced decision support, and insight generation, faster and with greater accuracy.

Scale has partnered with Deloitte, an awardee of the Joint Artificial Intelligence Center's Data Readiness Artificial Intelligence Development (DRAID)

Scale AI Partners with Deloitte

Scale partners with Deloitte to accelerate AI adoption across the U.S. government

I spent the last 6 years of my career at Uber across both Finance and Business leadership roles. In this time, I witnessed first hand how AI and ML drove improved customer experiences, more efficient operations, and will be instrumental in unlocking a future with self-driving cars and trucks.&nbsp;As I considered joining Scale AI, I got deeper into the latest trends and potential of AI. I have become even more excited about the many applications of AI and AI’s ability to improve decision making, drive productivity leaps, as well as deliver improved customer experiences and outcomes across every industry and domain on the planet.&nbsp;What I found in Alex and his team at Scale is a vision to help all companies and the U.S. government realize the potential of AI. Scale’s mission is to accelerate the development of AI applications and to do it by first tackling one of the hardest problems in AI, better data. Scale started by supporting the autonomous vehicle industry but has since expanded across many industries and applications. They have built products and solutions that support customers at the leading edge of AI as well as companies that are just getting started. In short, this convinced me that the opportunity and potential are unparalleled.&nbsp;As I spent more time getting to know Alex, we talked about what he was looking for in a CFO. We agreed that Scale needs a business-minded finance partner to foster continued, rapid growth of the Finance function that enables the company through increased strategic alignment, business model evolution, and increased operational discipline.&nbsp;Today, I am excited to announce that I am joining Scale as their first CFO to help the team deliver on their vision of building and scaling what will be one of the most impactful enterprise companies in the world by accelerating the adoption of AI. I will be leading Finance, Accounting, Corporate Development, and Strategic Partnerships. We are hiring across the team. Come join us!&nbsp;

Scale Welcomes Dennis Cinelli as First CFO

Why I Joined Scale

At Scale, we have a handful of internal Employee Resource Groups (ERGs) that connect Scaliens around shared identities. That means we spend a lot of time thinking about lowering barriers, leveling the playing field, and fostering communities of belonging and shared interests. For members of the LGBTQ+ community, having space at work to be authentic and open is so important. And, it’s part of why we built Scale Out, where Scaliens rely on each other to create a community of inclusion.Maverick Eason, Office Coordinator at SFHQ and Co-chair of Scale Out, shares their experience at Scale and the history of the Pink Triangle project.<img src="https://lh4.googleusercontent.com/vWM13AfeZMo1LX_inQDmjX9hKbrz4E8mkio7yyasFIyG-2kAeBkeVPp3WrZ8I5eVL8ljXuxEVwKLtWlrD_KOd3uTvNah45CuzSQ3vlEBxjhL-NKdyyXqtTmRPrbODpJCQytmjzdCJM80X261ig">Every year since 1995, a pink triangle has been constructed at the top of Twin Peaks in San Francisco during the month of June for Pride. That first year, under the cover of darkness, community members climbed the Peaks and adorned it with the pink triangle– a symbol of gay power and pride for the LGBTQ+ community. Where did this symbol originate? Why a pink triangle?During World War 2, Nazis would force people to wear different signifiers of who they were– yellow stars for Jews, blue for immigrants, and pink triangles for homosexual men. The pink triangles were the largest and brightest of these patches, and those wearing them received the most severe brutality from both Nazis and other prisoners. When people were released from the concentration camps, many of those wearing the pink triangle were still kept imprisoned, some until as late as the 1970’s.&nbsp;In the 1980’s, as the AIDS crisis was killing off record numbers of gay men and other LGBT people, the pink triangle was used on posters to call attention to the pandemic that the United States largely ignored– the symbol was adopted by ACT UP, a grassroots organization that works to end the pandemic and to help those living with HIV/AIDS. The pink triangle continues to be not only a symbol of power, but also serves as a reminder for what hatred and intolerance can do to us all.&nbsp;The pink triangle installation has developed into a San Francisco community event that is kicked off the first day of June, and includes a lighting ceremony by the mayor and other local politicians, organizers, and volunteers. I had the privilege of volunteering on May 26th as a site ambassador, along with the marketing and business developer of TransClinique, where we talked with visitors at the site and told them about the history and importance of the installation. Starting June 1st at sunset, the pink triangle will be visible every day at the top of Twin Peaks through the entire month. It serves as a reminder to us all of the impact of hatred, ignorance, and intolerance, and also the strength and power that the LGBTQ+ community continues to demonstrate every day across the globe.&nbsp;<img src="https://lh5.googleusercontent.com/WdwUpmz36oS9ctweHy3MRU-sOvClflsQgZs4ThqvTcatMhkrVQUZgX8u9t-CrWyajt7i08sD5JFcczD0p20a3LFMVhOOFtBku6MOFrqU5lzbJpQ7I89WsSl1lkzByamEZcO10Z3w8sR3ri-LeA">

Maverick Eason, Office Coordinator at SFHQ and Co-chair of Scale Out, shares their experience at Scale and the history of the Pink Triangle project.

The Pink Triangle  

Honoring Pride: Scale Out & The Pink Triangle  

This Memorial Day, join us on Jacob's journey as he shares his experiences with the costs of war.My college application process was based on a single announcement. During a senior-year English class, my high school principal walked in and announced two passenger airplanes had struck the World Trade Center buildings. While West Point was already my top choice, I went home and threw away applications I had started to other colleges. Witnessing 9/11 was a formative experience for me and my response to it was fully naive. At the time, I didn't know much about al-Qaeda. I likely couldn’t point out Afghanistan on a map. And I certainly couldn’t have a discussion about different sects within Islam. My framework for responding was just as naively simple: do something.&nbsp;Four days after graduating high school, I arrived at West Point. Our freshman class underwent our first summer of military training with a palpable sense of urgency that we would – not might – go to war. As the academic year started, I caught my first glimpse of the war’s toll in an unlikely place – the cadet mess hall. West Point’s mess hall is a cavernous and historic cafeteria of sorts, a place seemingly pulled straight from Hogwarts, where the operational feat of feeding 4,000 hungry cadets in 20 minutes happens not once, but three times a day. Seconds before lunch, the cacophony of noises that filled the mess hall was exchanged with perfect silence as a cadet leader barked daily instructions from a central, two-story platform. These instructions were usually about something mundane like uniform or schedule changes. Yet some days the instructions were replaced with a solemn announcement that a West Point graduate died during combat operations in Afghanistan.&nbsp;These announcements increased toward the end of my freshman year as the United States launched another campaign in the Global War on Terror with the invasion of Iraq. By the time I was a senior, some announcements were graduates I knew well. First Lieutenant Gary Avery was my teammate from an arduous cadet military skills competition. Major Bill Hecker was my eccentric freshman year English composition instructor who left behind his wife to raise their four children. As a 19-year-old kid, I struggled to make sense of news like this. I would hang my head for the rest of the meal, then return to my routine of cramming for an afternoon exam.While I no longer heard lunchtime announcements after I graduated from West Point, the news of more deaths did not stop. Through text messages from friends, I would hear about other graduates I knew – like Emily Perez, Tom Martin, and Matt Ferrara – who were killed in combat. Eventually, it was my classmates’ turn to deploy to Iraq and Afghanistan. Tim Cunningham and Nick Dewhirst were the first two from my class not to make it back home. During my second combat deployment as a Special Forces team leader in southeastern Afghanistan, I heard frantic radio traffic detailing an insider attack in an adjacent province that killed three American servicemembers. Later that day I would learn one of those killed was my West Point classmate and fellow Special Forces team leader Andrew-Pedersen Keel, better known as “PK.”&nbsp;Even after I returned home from Afghanistan, the announcements continued. The flow of Taliban from Pakistan increased into the area my team had previously secured only months prior. These new fighters killed 11 Afghan police at one of their checkpoints in the middle of the night. A few weeks later, the Taliban fired a rocket at an American convoy in the same area, killing a soldier previously assigned to my team and my best interpreter. The soldier left behind three children and the interpreter was only a few weeks away from going home on his last of several deployments to Afghanistan.&nbsp;A couple years later while I was in graduate school, I woke up to another announcement, this time in the New York Times. A Taliban suicide bomber had walked into a crowded volleyball match in the exact area where my team had been in Afghanistan, killing over 40 people, eight of them local police my team had trained. I remember watching a volleyball tournament there two years prior – the first since the Taliban had outlawed all sporting events. It was one achievement I used to rationalize the costs of the war. I thought this progress would never be reversed, but it was erased instantly.I still struggle to make sense of the many announcements that defined my transition into adulthood and my early professional life. I think about the families of those killed for whom there is no such thing as a return to normal. I often feel guilty for making it back from war alive or for tactical decisions I made that contributed to the loss of life. I don’t understand how I escaped unscathed from the ambushes and other violence I encountered. The role of chance in war is maddening and absurd.&nbsp;Aggregating the costs of war is also an exercise in absurdity. The Watson Institute at Brown University reports that over 7,000 American service members have been killed in post-9/11 combat operations. Over 30,000 American service members – four times the number of those who died in combat – have taken their own lives over the same time period. Beyond the U.S., Western allies who sent troops to support American wars account for roughly 1,000 more deaths. I’m acutely aware of the estimated 200,000 deaths of partner military and national police forces in places like Iraq, Afghanistan, and Syria. Fighting alongside them, I quickly realized how remarkable their bravery and sacrifice were, especially considering how under-resourced they were by their governments. Even more hidden from public view are the estimated 300,000 civilians who died violent deaths directly related to the wars. This doesn’t include the indirect impact on civilians, like those who lost access to basic resources or were displaced from their homes as refugees. Financially, the Watson Institute estimates the U.S. spent nearly $8 trillion related to post-9/11 wars.You might have strong reactions as you process the costs of these wars. You might bristle with pride at those who gave the last full measure of devotion for their country. You might feel angry at the cost of America’s wars and view them as misadventures marked by hubris and strategic failures. You might feel a sense of guilt as you try to understand an experience from which you are so detached. All of these reactions aren’t merely okay – they are fundamentally human. Whatever your reaction to these costs of war, I encourage you to take two actions as Memorial Day approaches.&nbsp;The first action is reflection. I find it useful to reflect on an obvious, yet unsettling fact: we are all finite. Even though much of my adult life was filled with announcements reminding me of life’s brevity and fragility, it is not natural to linger on this reality for more than a few moments. Set aside time to reflect on this and ask yourself: “What causes are worth my life?”, “What acts of service will I undertake with the finite time I have left?”, and “In what ways am I living sacrificially for others?” Reflecting on questions like these can be helpful in living an examined life to deliberately grow into the person you hope to become.&nbsp;The second action is remembrance. My most powerful experience of remembrance came while I was serving as an instructor at West Point and visited its historic cemetery on Memorial Day a few years ago. When I approached a row of tombstones of West Point alumni who graduated around the time I was a cadet, I paused on one that belonged to First Lieutenant Laura Walker. I noticed the date of her death and realized she was one of the announcements I heard at lunch during my junior year as a cadet. As I gazed at some photos placed near her tomb, two people walked toward me who had been sitting under a nearby tree. They were Laura’s parents. They make an annual pilgrimage to West Point’s cemetery on Memorial Day to meet others who come across Laura in the cemetery and to share her stories with them. What struck me the most while talking with them were the human terms – like “goofball” – they used to describe Laura. There was no mention of the word “hero,” just relatable stories that highlighted Laura’s similarities to the rest of us.&nbsp;In that spirit, my act of remembrance for this Memorial Day was to learn more about the stories of those who were announcements in my life. I learned that Gary Avery liked sewing and even started a nonprofit for Iraqi orphans while he was a cadet. I learned that Bill Hecker was a Saint Louis Cardinals fanatic and loved gathering his four kids on the couch before bedtime to give dramatic readings from Little House on the Prairie. I learned that Emily Perez was known during her days as a cadet for jingling her tambourine through the hallways of the barracks on Sunday mornings as she walked to a Baptist church service.&nbsp;I encourage you to do the same kind of remembrance. As we remember the fallen for the ordinary people they were – for how similar they are to you and me – we realize that the ordinary human agency they had to make selfless choices with their lives is also available to the rest of us.&nbsp;<h3>About Veterans@Scale</h3>We’re investing in a diverse and inclusive workforce at Scale, and our workforce is as diverse as the customers we serve. We have different backgrounds and varied experiences. But wherever we work, whatever we do, we share a single set of values that unite our company.&nbsp;One of the pillars of our diversity, inclusion, and belonging efforts is Employee Resource Groups (ERGs). We currently have five ERGs, and each one plays a vital role in ensuring that those different communities have a voice and can actively influence our company policies and culture. Today, we’re happy to shine a spotlight on Veterans@Scale and hear from one of its veteran members, Jacob Sheehan.<h3>About Jacob</h3>Jacob leads the Federal labeling workforce at Scale as the Head of Federal Supply. He served in the U.S. Army as a Special Forces Officer and an assistant professor of Economics at the United State Military Academy (West Point). He is a graduate of Stanford’s Graduate School of Business and a Fulbright Scholar to the United Kingdom.

This Memorial Day, join us on Jacob's journey as he shares his experiences with the costs of war.

Memorial Day Reflection

Veterans@Scale: Reflection and Remembrance on Memorial Day

Q: Tell us a little bit about yourself and your role at Scale.A: My name is Sasha, and I'm a Machine Learning Engineer here at Scale. I've been at Scale for about a year and eight months. Previously, I was doing a master's program in computer science at Stanford, where my concentration was in artificial intelligence.Within my career at Scale, I've worn a lot of different hats. I first started working on the Nucleus team, in a backend software engineering role when the team was very young. It was&nbsp;Scale’s newest product at that point, and I helped build out a lot of Python SDK interfaces, and the backend schema for a lot of new features, including Autotag, which is the edge case detection feature.&nbsp;I later transitioned to building out other new products, including Scale Validate, which is the model scenario testing and validation product area.Most recently, I've recently transitioned to our Machine Learning team full time, where I'm on the Content Understanding team. I'm basically helping to support automation in labeling projects associated with content categorization and curation type labeling projects.Q: Could you give us a little bit of detail about your educational background, and how that led you to join Scale?A: My undergrad degree was in applied mathematics. Entering college, I didn't think of myself as a coding person at all. But during the years I was at Stanford, computer science was something that a lot of the undergraduate body talked about, and the entry level computer science classes were really fun. During my junior and senior years in college, I started taking some entry level machine learning classes. The first thing I ever learned about was logistic regression, and how you assign importance or attribution to different input features&nbsp;to answer social science questions. Like, is it age, or education, that contributes more to this downstream factor?I thought that was really interesting, and really wanted to double down on that. So, that's what led me to stay for a fifth year master's, and specialize in AI. I worked as a research assistant briefly, in a sociology lab, where the objective of the sociology lab was to develop a dataset, documenting urban change. My job as a research assistant was to label data on AWS, Mechanical Turk. We had a lot of difficulty controlling quality, and getting annotations that were detailed enough to model the downstream tasks of trash detection and vandalism detection.&nbsp;So, I knew that data collection was a hair on fire problem for a lot of application-driven teams. When I interviewed at Scale, I felt like the mission, and the problems that the company was trying to solve, were very urgent and impactful.Q: What attracted you to working at a hyper growth startup?A: When I was in college, I think a lot of my peers were talking about going to work at a big tech company; that was the prestigious choice. It was safe; it was a known outcome. At first, I was really focused on that, and I did a lot of internships at big companies.&nbsp;And then I got this career advice my sophomore junior year of college, from a woman at Grace Hopper, and her advice was get on a rocket ship. Basically, the idea was, if you could just put yourself in a place where there were interesting problems to solve, some of those were going to come your way.I really took that to heart, and my impression of myself started to shift toward somebody who really wanted to tackle big challenges and wear a lot of hats in their job. So, I joined Scale under the hypothesis that I was going to have an opportunity to wear a lot more hats and serve a lot of different functions at a very lean company.&nbsp;As a result of having the opportunity to take on difficult challenges, I've grown my skills at a very quick rate. And I think I gained seniority and experience, at a rate that far exceeds any previous work opportunity I had in the past.&nbsp;Q: What's your advice to students, or other people, who are interested in careers in machine learning?A: First of all, I think it's an excellent career to be in, in that there are a lot of opportunities in the space right now. And the number of positions for machine learning engineers and machine learning-adjacent engineers, is growing really rapidly. I think there are also a lot of exciting developments, both on the research and applications side.One thing that fascinates me about machine learning is how closely it can relate to real-world problems. If you’re interested in sociology, there's probably a place for that type of modeling for you. Or, if you're interested in online advertising, there are a lot of different application areas to concentrate on.The other thing that I would recommend is really reading a lot. There are a lot of great resources online for practicing building deep learning models, whether it be through Kaggle challenges, or what have you. Also, a lot of research papers and academic work are publicly available.One thing that I felt intimidated about when I started in the field was, how much there was to know, and how little of it I did know, as somebody that was early in their career. And I think making a habit of reading papers, and reading about approaches or problems that I didn't work on directly on a day-to-day basis, really increased my confidence.Q: Has working at Scale shifted your perspective on AI or ML applications at all?A: Definitely. When I first started at Scale, what I was really interested in was machine learning modeling work, and the idea of experimentation–building different neural network architectures, and being very scientific about, whether A works better than B.&nbsp;One thing that's become really apparent, both working at Scale, and seeing the types of problems that our engineering teams face, but also working with Scale customers, is the importance of systems design within large scale machine learning systems. I’ve seen how difficult the problems of scalability, observability and overall reliability of machine learning systems are.&nbsp;I really think that that's where a lot of innovation in the next 10 years is going to come from; not necessarily the modeling techniques themselves, but more, how you might integrate those modeling techniques into a larger scale operational framework.Q: What do you like to do outside of work?A:&nbsp;I've recently gotten into cycling. And so, I spend a lot of my time outside, whether it's running or riding my bike in the Bay Area. I also love wheel pottery. So on the weekends, you'll find me at the pottery studio.

Sasha Harrison details her experience as a Machine Learning Engineer.

Life at Scale

Document AI is powered by machine learning (ML) models that process documents at high accuracy and speed. When these models are fine-tuned for a customer, they become even more accurate and regularly achieve near-perfect accuracy levels and do not require any human intervention. Scale engineers continuously work to improve these models by creating new tools to make sure Document AI is highly reliable for the most critical industries we serve. This blog will highlight one of those tools, a technique called “linting”, and how it helps make Document AI’s models more accurate.<h2>What is Linting?</h2>At Scale, linting is the practice of creating rules or developing machine learning models that automatically detect errors in training data used to create models. In the case of Document AI, we leverage two types of linters: simple, quick linters and more complex ML-based linters.&nbsp;Here is an example of a simple linter comparing field types and their responses within a document.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fb8b843651dad4ba48f5831c07b48c6f2" width="755">In the example above, the letter ‘O’ should not be in a numerical, monetary amount field and our field-type linter would flag this example for correction.Another example of the simple quick linters is a linter that enforces known business rules. For example, a linter can ensure an invoice subtotal is equal to the sum of all line item totals.<h3><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F9e0a34e10d464195911c07b617ee27c7" width="574"></h3>In the example above, the subtotal on the right ($16) is inaccurate and our linter would flag this example for correction.These simple linters are helpful where straightforward rules can easily be applied. Simple linters, while useful, can be implemented by most. What’s more unique about Scale Document AI is how we go beyond the mainstream methods and tools in the document industry to achieve high-quality.&nbsp;Scenarios that require a higher level of capability or understanding are better suited to machine learning (ML)-powered linters instead of rule-based linters, which is Scale’s specialty. Our specialized ML linters are trained to perform specific tasks, such as learning to compare differences in a transcribed text and the visual representation of the text as it appears on the document’s page. In this example, when the linter finds a meaningful discrepancy, it automatically corrects it or flags the document for review by highly experienced and skilled reviewers. This human-in-the-loop review is optionally available based on customers’ needs and preferences.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F626b6d1309fa4dc28c28d6d75279abb6">In the example shown above, we trained an ML-based linter to understand the context to know that “Account #” on the bottom left refers to a bank account number as it is in a section with ACH instructions while “Account Number” on the top right is referring to a business account number. If “Account Number” in the example on the top right had been identified as a bank account number, the linter would flag this for correction.ML linters are inherently more difficult to implement than rule-based linters as they have the same challenges as developing any other machine learning models. Developing an ML model is a statistical process prone to variation. That variation can manifest as unreliability or as overfitting. For example, unreliability can lead to an ML model producing wildly different predictions on two slightly different documents. With overfitting, the model learns the detail and noise in the training data and fails to perform on new data. In the case of linting, the ML models are trained on data with errors to help identify and flag other errors. The data used for training linters is inherently not of the highest quality; therefore it’s common that while the ML linter can still make decent suggestions, the model itself is not the most performant. This requires our ML team to place more restrictions and guardrails, like ensuring the confidence threshold is set high to only flag errors and make suggestions on things the model is fairly certain about. This prevents the model from producing too many unreliable linter suggestions that lead to human reviewers simply ignoring or not trusting these suggestions.&nbsp;Developing and deploying ML linters is also an iterative process with a feedback loop. As we deploy ML linters for our document processing models, the document processing models improve upon being retrained with higher quality data. As the document processing models improve, it gives better first results as well as learning other errors to surface to the human reviewers.<h2>How Do Linters Improve Models?</h2>The power of machine learning to recognize patterns between input and output data is its greatest strength as well as its Achilles’ heel. If the training data contains errors, the model will learn the wrong task. Having the right linters in place increases the accuracy of machine learning models by ensuring the data used for training is of the highest quality.&nbsp;For Document AI, we began by designing simple linters to check field types such as monetary amounts, quantities, and ZIP codes. We then expanded beyond these simple linters and implement ML-based linters to check for complex fields such as addresses with varying formats that we would need to parse and enforce.&nbsp;To make things more concrete, let’s think about the use case of invoices. At first glance, it seems straightforward to recognize an invoice: they all list the products sold or the services rendered, the price of each, and how payment should be made. However, invoice layouts, details, and content vary widely from business to business. Some invoices are particularly formal, containing everything from company bank account numbers and line item descriptions to account manager names and contact information. On the other end of the spectrum, an informal email between longtime business partners can also serve as an invoice. The unstructured and ever-changing nature of invoices make it challenging to model comprehensively and accurately.When building Document AI’s machine learning model for processing invoices, using linters resulted in a 16.3% relative decrease in transcription errors in the training data. This relative decrease in transcription errors is crucial, as the misspelled text makes it impossible for our document parsing models to extract the right information. This linter also helped remove the need for an additional layer of human quality assurance (QA) review while still ensuring that training data is pristine.&nbsp;Training our models on the higher quality training data led to a 14.3% relative increase in our document processing model’s accuracy. This improvement translates to increased model generalizability and robustness, and more accurate models mean fewer manual reviews, providing additional value for our customers.&nbsp;Linters make Document AI more accurate and hence deliver higher quality models to our customers who rely on high accuracy document processing.<h2>Parting Thoughts</h2>Scale Document AI distills insights from your messy, changing document data and delivers fast, accurate models tailored to your business without the need for templates. If you’re interested in learning more, take a look at our <a href="https://www.youtube.com/watch?v=jWCwVcQCA10" rel="noopener noreferrer" target="_blank">product video</a> or visit our <a href="https://scale.com/document-ai" rel="noopener noreferrer" target="_blank">product page</a>.Acknowledgments:We thank Malcolm Greaves from Scale’s Document AI Machine Learning team for his experiments that laid the groundwork for this post and for frequent consultations throughout the writing of this blog.

Document AI's linters improve model quality by ensuring common and hard-to-spot errors never make it into the training data.

Scale Document AI leverages ML Linting to Improve Document Processing

How ML Linting Meaningfully Improves Adaptive Document Processing at Scale

Willow leads the Customer Operations team at Scale. She is a former director of logistics at Amazon.com, a graduate of MIT’s Leaders for Global Operations program, and a former captain in the U.S. Army’s Military Police Corps. I am a transgender pioneer within the military and corporate world, with a career spanning 12 years in the U.S. Army, MIT, Amazon, and Scale. Here are some of the things I learned along the way. Why I decided to change my gender I had decided to change my gender while in a concrete shelter from rocket attacks right before returning from deployment in Afghanistan. Rockets in this context were not particularly threatening, but sitting in a concrete shelter at night waiting for the sirens to go off can naturally lead to introspection. I had no intention of being a pioneer. It was a simple search for my own personal happiness. I felt like any other choice would be the wrong one. It was too much of a big and scary decision to think at all about how my career would shape out. There were deliberate risks I took and risks that I didn’t expect and needed help with. Back in the states, I was the first officer in my military unit to be out as bisexual after the repeal of “Don’t Ask, Don’t Tell” (DADT). This was a deliberate risk: my personal and professional lives didn’t intersect, but I thought that someone should be out as an example if other soldiers wanted to come out. I still felt like I had a vulnerable secret, as being transgender was still prohibited, and my plans to transition after departure were carefully guarded even as I changed my legal name and started living as female while on leave. My commanding officer was overtly hostile towards the repeal of DADT. After one staff call in which he used “faggoty” as a term of derision, I grew concerned about whether or not I would be able to leave the Army on my own terms. I need not have been concerned, however. Senior members of the staff met with the commander over his comments, and he decided to leave the Army over the pace of change. Like me, but for different reasons, he was voting with his feet. Returning to graduate school, I was the first openly transgender person in my program. I was one of two transgender students to tell my story to the school’s leaders in an ultimately successful bid to change insurance coverage policies for all students. Not all was respect and smooth sailing; in one emotionally crushing case, two teachers complained to my program admin about me using the restrooms that matched my gender identity (it seems strange to type this - much has changed in eight years). My friends calmed me down, and school administrators used the complaint as a teaching moment for the faculty. The only [other] in the room? As my civilian career advanced at Amazon and at Scale, I became more aware of how being transgender would shape my approach to work. No matter the realities of my professional role, I always feel like a bit of an outsider. It has been argued by folks more articulate and demonstrated in studies that diverse perspectives lead to better decisions, and clearer identification of risks. This is a good thing. The downside to feeling like an outsider is always feeling the need to push myself harder, to prove myself better, and to hold myself to an extremely high internal standard - and to always fear slipping. This approach to work has helped me advance quickly, and made me bold, but it is exhausting at times and ultimately rooted in fear. One of the factors in my recent decision to join Scale was that many company leaders identify with the LGBTQ community. I want to help build the future without worrying so much about how I fit in it. Taking a role at a smaller, fast-moving startup with less formal structure has been awesome. There’s a sense of purpose and mission and a rapid flow of ideas that I feel I can really be part of. Transgender people have always existed in one way or another, before society’s own peculiarities and social constructs started calling it a “new thing.” I wrote earlier that I was the first openly transgender applicant at my grad school program. It is highly likely that there were transgender people there before, and it is likely that some of my classmates were transgender and simply hadn’t decided to transition yet. It is important to remember that an individual story cannot represent a community, especially with big variation in opportunity and privilege shaping peoples’ lives. Advice for folks feeling like outsiders There are a few things that I wish I’d known earlier in my career. One piece of advice I got from a friend has stuck with me: “You will always think about yourself a lot more than others think about you.” That feeling of awkwardness, that one thing you did - most people simply won’t notice or remember. Our memories are inherently selective. Learn from your mistakes but choose to remember the good parts. The other thing I wish I’d understood better earlier is to not be afraid to pick a mission with purpose. Early on, I picked assignments that seemed like the best value for the company’s ability to save money or expand programs and to advance my career. I left jobs and teams that I enjoyed to seek out the external trappings of success. Many people do this, but the feeling of validation seemed so necessary when I was uncertain of my place in society. Finding a team and a voice that makes me feel at home and a mission that serves a purpose I believe in has ultimately been more gratifying. Now, what the people around me think matters more than what the world thinks. Scale believes there is tremendous potential for AI technology to drive a revolution in true efficiency and productivity, across industries - this is a change for which I want a front row seat. I am glad I am able to take that seat feeling like I am no longer an outsider.

Scale Operations leader Willow Primack shares her personal experience for International Transgender Day of Visibility

Willow Primack shares her experience for International Transgender Day of Visibility

Trans Day of Visibility: My Experience at Scale

<h1>Project Ideas and Inspiration</h1> Scale recently sponsored Treehacks, Stanford University’s annual hackathon. During this fully virtual hackathon, nearly 1300 active hackers and 114 teams submitted projects for judging. Out of all teams, 21 submitted projects using Scale’s API. The winning project for Best ML Hack was <a href="https://devpost.com/software/fine-tuned-heads" rel="noopener noreferrer" target="_blank">Fine Tuned Heads</a>, which proposed a new workflow to assist with the implementation of Computer Vision. Rather than training a single model from scratch, the encoding and fine-tuning aspects of a task were separated. The Fine Tuned Heads team created an automatic data collection pipeline using Flask to automatically execute scripts to grab .m3u8 URLs from Youtube, then used Scale Rapid to automate uploading and generating batches of ground truth labels for these images. According to the Fine Tuned Heads team, “this approach can be used to improve the efficiency of inference in ML tasks, and by engineering our self-supervised model with an efficient architecture and deploying on efficient hardware, we can reduce the carbon footprint by over 100x.'' An additional winner of the API Challenge was <a href="https://devpost.com/software/accecity" rel="noopener noreferrer" target="_blank">AcceCity</a>, which provides an ML-powered mapping platform that enables cities, urban planners, neighborhood associations, disability activists, and more to identify key areas to prioritize investment in. For their project, the AcceCity team first used the Google Maps API to send images of street views to their backend. They looked for handicapped parking, sidewalks, disability ramps, and crosswalks and used computer vision, by custom-fitting a zero shot learning model called CLIP from OpenAI, to automatically detect those objects from the images. They then tested the model using labeled data from the Scale Rapid API. Other interesting projects include using Scale Rapid to label map data to promote sustainable living, and classify patient medical image data. Beyond the examples from the Hackathon, one can use Scale Rapid for a variety of ML projects, including to <a href="https://scale.com/blog/rapid-cyclegan" rel="noopener noreferrer" target="_blank">anime-fy images</a> and to train a <a href="https://scale.com/blog/imdb-rapid" rel="noopener noreferrer" target="_blank">sentiment analysis model for movie reviews</a>. Use our <a href="https://scale.com/open-datasets" rel="noopener noreferrer" target="_blank">open datasets</a> to draw further inspiration for your projects.<h1> </h1><h1> How can you use Scale to build a Successful ML Hackathon Project?</h1> One of the most challenging problems preventing teams from creating novel ML projects in a short timespan is the “Cold Start” problem, which is when an ML model cannot draw inferences for users or items about which it has not yet gathered sufficient information. Especially during the compressed timespan of a hackathon, obtaining high-quality training data for models is nearly impossible. Scale Rapid is the fastest way to obtain production-level quality labels, with no data minimums. Using Scale Rapid, your hackathon team can experiment quickly by:<ol><li> </li><li> Setting up projects in minutes and receiving initial data within hours</li><li> </li><li> </li><li> Iterating over edge cases and instructions by getting real-time feedback</li><li> </li><li>Scale to production-level pipelines with precision quality control</li><li> </li></ol> To get started, sign up for an account with your university email on our <a href="https://scale.com/rapid/university" rel="noopener noreferrer" target="_blank">University Program Page</a> to receive $250 in Rapid Credits, or check out our <a href="https://colab.research.google.com/drive/1XZjbVtA2oGOdMp0-v0E80ujMB8tbeh7x?usp=sharing" rel="noopener noreferrer" target="_blank">quick start python notebook</a>.Rapid currently supports the following annotation types: <ul><li>General Image Annotation</li><li>2D Semantic Segmentation</li><li>Text Collection/Categorization</li><li>Document Transcription</li><li>Named Entity Recognition</li><li>Video Playback Annotation</li><li>Lidar Annotation</li><li> </li></ul><img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2ac2bcbb-1e7d-4268-2c77-bf6d2623c400/public" alt="Annotation Types Supported in Scale Rapid"> If you have your own unlabeled data, add it to your project using local file or csv upload, import it from previous projects, or upload it from the cloud (S3, GCP, and Azure). Otherwise, check out our <a href="https://scale.com/open-datasets" rel="noopener noreferrer" target="_blank">open datasets</a>. Once you receive task responses, visualize your data in Scale’s dataset management platform, Scale Nucleus, by navigating to the data tab under your project and clicking “Visualize in Nucleus”. Nucleus helps you understand the strengths and weaknesses in your dataset, analyze the long tail of your dataset, collaboratively edit labels, and explore metrics, failure cases, and confusion matrices linked to your training data. <img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/899fda4e-d4ab-4ed4-7ef4-5fa12e396200/public" alt="Scale Nucleus <> Scale Rapid Integration"> Quickly find similar images using the “Autotag” feature in Nucleus, and further refine your set of images by providing a few examples of positive and negative examples. After identifying this subset of your dataset, you can send it back for labeling through Scale Rapid. Scale AI’s products help enable and strengthen a team’s ML development cycle immensely by delivering high quality data for any project. As you or your team continues to maintain and improve your project post-hackathon, use Scale AI for consistent, reliable data in order to create, validate, and maintain production for your high-performing ML models.

Scale AI recently sponsored Treehacks, Stanford University’s annual hackathon. Explore how to use Scale Rapid for a successful ML hackathon.

Using Scale Rapid for Hackathon Projects

How to Use Scale for Your ML Hackathon Project

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

customers

Display order on the page /customers. Higher values show first

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

basecamp-agenda

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

research

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Blog