image.jpg

Yuri Maruyama

As we close out Women's History Month, we reflect on the incredible contributions of women throughout history and their importance in shaping our world today. In particular, we want to highlight the vital role that women have played and continue to play in STEM (Science, Technology, Engineering, and Math) fields.
Despite facing numerous challenges and barriers, women have made significant strides in STEM. From <a href="https://www.computerhistory.org/babbage/adalovelace/">Ada Lovelace</a>, the world's first computer programmer, to <a href="https://www.nasa.gov/content/katherine-johnson-biography">Katherine Johnson,</a> who helped put the first human on the moon, women have been pioneers in fields traditionally dominated by men. However, there is still much work to be done to address the gender gap in STEM, particularly in terms of representation and pay equity.
Throughout this month, Scale&rsquo;s Women ERG spoke to a few of its members to showcase the perspectives and experiences of women working in AI. We heard from Scaliens who have consistently broken barriers and achieved success in their fields, as well as those who have faced challenges and obstacles along the way.
We hope that by sharing these stories, we can help amplify the importance of diversity and inclusion in STEM and inspire the next generation of female leaders in the field. So, join us as we celebrate Women's History Month and recognize the incredible contributions of women in STEM. Together, let's continue to work towards a more equitable and inclusive future for all.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/503f7e9b-f8aa-4580-a096-2cba1e2402d6">
&ldquo;Having it all&rdquo; as a woman means different things to different women. I think this is important to acknowledge as I discuss what it means to me.
Last year, I began pursuing my MBA on top of my full-time job. When I first started, I quickly discovered that my definition of &ldquo;having it all&rdquo; was exhausting and unsustainable in my new season of life. It was time to change that definition.
Oftentimes women tend to take on more responsibilities both at work and at home. This was exactly what I was doing in every aspect of my life. It forced me to prioritize the limited free time I have. I had to set boundaries in my work and personal life to make room for school. I highly recommend The Book of Boundaries: Setting the Limits that Set You Free by Melissa Hartwig Urban as it was immensely helpful for me.
&ldquo;Having it all&rdquo; not only means different things to different women. It also means different things in different seasons of life. In my current season of life, &ldquo;having it all&rdquo; means shifting priorities and setting boundaries that help me succeed where I am today. When I finish my MBA, I&rsquo;ll enter a new season of life where &ldquo;having it all&rdquo; is going to evolve. And I can&rsquo;t wait to see what that looks like.&nbsp;
Yuri Maruyama
Head of Product Marketing
I joined Scale four and a half years ago when Scale was a forty-person company. At the time, I didn&rsquo;t think it would be possible for me to start a family while working at a hyper-growth, early-stage startup. It took a few years but I welcomed my first baby last summer. I was able to take and enjoy my full parental leave without getting pulled into any work or feeling any guilt.&nbsp;
Once I came back, I worried about how I would balance being back to work full-time while also being present for my family. I still often fall short of my own expectations, but to manage this, I&rsquo;ve blocked off my calendar in the mornings until 8:30am and in the evening between 5-8pm to spend with my family. Blocking this time and holding this boundary has helped to make me more present and productive at work during the 8:30am-5pm time when I am 100% focused on work. At the same time, I&rsquo;ve been able to be present for my family in the mornings and evenings without feeling guilty or like I need to check slack or email.
Bihan Jiang
Product Manager
Balancing work and life is both a challenge and a necessity. One way that I&rsquo;ve found balance is by scheduling time for activities that I enjoy, such as exploring coffee shops (I drink 3+ cups a day!), traveling, watching dramas, and playing board games like Catan and Tractor.
It&rsquo;s also critical to find passion in your work. As a product manager on the Spellbook team, I&rsquo;m responsible for driving the development of our product, collaborating with cross-functional teams, and staying on top of industry trends and customer needs. It can be challenging, but it&rsquo;s also incredibly exciting to be building in the generative AI space, which is evolving every day.
Achieving a balance between work and life is a continuous effort that requires prioritization. By making sure to allocate time for my hobbies and interests, as well as having a support system in place, I am able to maintain a strong sense of self while excelling in my career.
Karena Zhang
Account Executive
Passionate about living as a true &ldquo;Corporate Athlete,&rdquo; I find a myriad of ways to stay active and challenge myself in the office and beyond. When I first began my career, I was a Reformer Pilates instructor part-time while working full-time in tech. Now that I have since retired from being a fitness instructor, I have taken on a new challenge of traveling the world by way of sliding down mountains on wood planks (skiing). I was also a competitive college tennis player at Carnegie Mellon, so I am not new to navigating the stress of work and play, but I know how important it is to have a strong sense of self outside of work. I believe that no matter how stressful things get at work, I still have to live my life. Calendar blocks plus a passion for planning allow me to balance working hard, having fun, and making history.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/f383b465-2282-480b-bdb2-246cb1198506">As a female engineer, I have experienced being overlooked or underestimated many times in my career. Most of the time, there's nobody to blame--these biases are just that deeply ingrained in everyone. Over time though, I have learned to use this to my advantage. When my coworkers had low expectations of me, as long as I still set ambitious goals for myself, I could make sure I blew their expectations out of the water. I constantly remind myself daily to be confident in myself and to not let my goals and abilities be dictated by the male-dominated environment I'm surrounded by. I also keep in mind that as a minority in the workplace, I can bring fresh perspectives to the table. Finding success in engineering definitely feels like an uphill battle sometimes, but that also makes success feel much more rewarding.
Iris Guo
Strategic Product Manager
Despite lots of excitement and eagerness, I didn&rsquo;t know what to expect coming into the role of a strategic product manager as a new grad. In the past, I had more traditional product manager internships at big tech companies where I was building software/ hardware features and focused on the user experiences. I haven&rsquo;t done operations before and this was my first time getting into the weeds of running a project as an STO. As a woman with high degrees of agreeableness and conscientiousness, I found it difficult to be the one calling the shots and not asking for approvals before execution. At first, I thought I had to come up with the solution for every problem, so I would put a lot of pressure on myself to do everything right and I wasn&rsquo;t used to asking for help since I didn&rsquo;t want to bother other people. I tend to be detail-oriented and diligent in my tasks, but sometimes I would miss out on the bigger picture for strategic thinking. When I was stressed out at work, I would lean into my passion for dancing as a creative channel for me to destress. Dancing helped me express my full sense of self and relieved my obsessive thoughts from work to stay present. I performed ballroom dancing growing up and I started learning k-pop/ jazz funk/ contemporary/ bachata/ salsa when I moved to SF six months ago. Every group dance required lots of teamwork to help each other to learn the harder part of the dance, and I learned to ask for help from people with more experiences, which I applied to problems at work. Through dancing, I also learned to experiment and iterate fast, because we would have different versions of the same dance and improve from trials and errors. Dancing enlightened me to see the bigger pictures and let go of the unimportant details with bold movements, which served as a constant reminder to me at work. I felt extremely grateful for having a healthy and mutually beneficial relationship between my hobby and my job, and I believe that everyone can find the right balance between work and life.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/5ac7d1f9-b882-43ac-9da9-8511d5062f95">I&rsquo;m thrilled to be a woman and working with other talented men and women here at Scale as a part of the AI revolution. As a society, we have a ways to go in terms of representation and equity across all STEM fields, and think that women have an especially important role to play in bringing diverse perspectives shaping AI products, and ensuring that the future of AI is just, equitable, and serves all of humanity better.

We hope that by sharing these stories, we can help amplify the importance of diversity and inclusion in STEM and inspire the next generation of female leaders in the field.

Throughout this month, Scale’s Women ERG spoke to a few of its members to showcase the perspectives and experiences of women working in AI.

Women’s History Month at Scale

Document AI is powered by machine learning (ML) models that process documents at high accuracy and speed. When these models are fine-tuned for a customer, they become even more accurate and regularly achieve near-perfect accuracy levels and do not require any human intervention. Scale engineers continuously work to improve these models by creating new tools to make sure Document AI is highly reliable for the most critical industries we serve. This blog will highlight one of those tools, a technique called “linting”, and how it helps make Document AI’s models more accurate.<h2>What is Linting?</h2>At Scale, linting is the practice of creating rules or developing machine learning models that automatically detect errors in training data used to create models. In the case of Document AI, we leverage two types of linters: simple, quick linters and more complex ML-based linters.&nbsp;Here is an example of a simple linter comparing field types and their responses within a document.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fb8b843651dad4ba48f5831c07b48c6f2" width="755">In the example above, the letter ‘O’ should not be in a numerical, monetary amount field and our field-type linter would flag this example for correction.Another example of the simple quick linters is a linter that enforces known business rules. For example, a linter can ensure an invoice subtotal is equal to the sum of all line item totals.<h3><img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F9e0a34e10d464195911c07b617ee27c7" width="574"></h3>In the example above, the subtotal on the right ($16) is inaccurate and our linter would flag this example for correction.These simple linters are helpful where straightforward rules can easily be applied. Simple linters, while useful, can be implemented by most. What’s more unique about Scale Document AI is how we go beyond the mainstream methods and tools in the document industry to achieve high-quality.&nbsp;Scenarios that require a higher level of capability or understanding are better suited to machine learning (ML)-powered linters instead of rule-based linters, which is Scale’s specialty. Our specialized ML linters are trained to perform specific tasks, such as learning to compare differences in a transcribed text and the visual representation of the text as it appears on the document’s page. In this example, when the linter finds a meaningful discrepancy, it automatically corrects it or flags the document for review by highly experienced and skilled reviewers. This human-in-the-loop review is optionally available based on customers’ needs and preferences.<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F626b6d1309fa4dc28c28d6d75279abb6">In the example shown above, we trained an ML-based linter to understand the context to know that “Account #” on the bottom left refers to a bank account number as it is in a section with ACH instructions while “Account Number” on the top right is referring to a business account number. If “Account Number” in the example on the top right had been identified as a bank account number, the linter would flag this for correction.ML linters are inherently more difficult to implement than rule-based linters as they have the same challenges as developing any other machine learning models. Developing an ML model is a statistical process prone to variation. That variation can manifest as unreliability or as overfitting. For example, unreliability can lead to an ML model producing wildly different predictions on two slightly different documents. With overfitting, the model learns the detail and noise in the training data and fails to perform on new data. In the case of linting, the ML models are trained on data with errors to help identify and flag other errors. The data used for training linters is inherently not of the highest quality; therefore it’s common that while the ML linter can still make decent suggestions, the model itself is not the most performant. This requires our ML team to place more restrictions and guardrails, like ensuring the confidence threshold is set high to only flag errors and make suggestions on things the model is fairly certain about. This prevents the model from producing too many unreliable linter suggestions that lead to human reviewers simply ignoring or not trusting these suggestions.&nbsp;Developing and deploying ML linters is also an iterative process with a feedback loop. As we deploy ML linters for our document processing models, the document processing models improve upon being retrained with higher quality data. As the document processing models improve, it gives better first results as well as learning other errors to surface to the human reviewers.<h2>How Do Linters Improve Models?</h2>The power of machine learning to recognize patterns between input and output data is its greatest strength as well as its Achilles’ heel. If the training data contains errors, the model will learn the wrong task. Having the right linters in place increases the accuracy of machine learning models by ensuring the data used for training is of the highest quality.&nbsp;For Document AI, we began by designing simple linters to check field types such as monetary amounts, quantities, and ZIP codes. We then expanded beyond these simple linters and implement ML-based linters to check for complex fields such as addresses with varying formats that we would need to parse and enforce.&nbsp;To make things more concrete, let’s think about the use case of invoices. At first glance, it seems straightforward to recognize an invoice: they all list the products sold or the services rendered, the price of each, and how payment should be made. However, invoice layouts, details, and content vary widely from business to business. Some invoices are particularly formal, containing everything from company bank account numbers and line item descriptions to account manager names and contact information. On the other end of the spectrum, an informal email between longtime business partners can also serve as an invoice. The unstructured and ever-changing nature of invoices make it challenging to model comprehensively and accurately.When building Document AI’s machine learning model for processing invoices, using linters resulted in a 16.3% relative decrease in transcription errors in the training data. This relative decrease in transcription errors is crucial, as the misspelled text makes it impossible for our document parsing models to extract the right information. This linter also helped remove the need for an additional layer of human quality assurance (QA) review while still ensuring that training data is pristine.&nbsp;Training our models on the higher quality training data led to a 14.3% relative increase in our document processing model’s accuracy. This improvement translates to increased model generalizability and robustness, and more accurate models mean fewer manual reviews, providing additional value for our customers.&nbsp;Linters make Document AI more accurate and hence deliver higher quality models to our customers who rely on high accuracy document processing.<h2>Parting Thoughts</h2>Scale Document AI distills insights from your messy, changing document data and delivers fast, accurate models tailored to your business without the need for templates. If you’re interested in learning more, take a look at our <a href="https://www.youtube.com/watch?v=jWCwVcQCA10" rel="noopener noreferrer" target="_blank">product video</a> or visit our <a href="https://scale.com/document-ai" rel="noopener noreferrer" target="_blank">product page</a>.Acknowledgments:We thank Malcolm Greaves from Scale’s Document AI Machine Learning team for his experiments that laid the groundwork for this post and for frequent consultations throughout the writing of this blog.

Document AI's linters improve model quality by ensuring common and hard-to-spot errors never make it into the training data.

Scale Document AI leverages ML Linting to Improve Document Processing

How ML Linting Meaningfully Improves Adaptive Document Processing at Scale

Scale AI was founded on the belief that better data → better AI. There has recently been a renewed focus on “good” data and its impact on the quality of models in the industry. Andrew Ng, who spoke at our first Transform conference earlier this year, emphasized this point: “We spend a lot of time sourcing and preparing high-quality data before we train the model, just like a chef would spend a lot of time to source and prepare high-quality ingredients before they cook a meal. I feel like a lot of the emphasis of the work in AI should shift to systematic data preparation.” While we have intuitively understood the importance of data quality, the Google Research Team paper <a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/0d556e45afc54afeb2eb6b51a9bc1827b9961ff4.pdf" rel="noopener noreferrer" target="_blank">"Everyone wants to do the model work, not the data work’: Data Cascades in</a><a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/0d556e45afc54afeb2eb6b51a9bc1827b9961ff4.pdf" rel="noopener noreferrer" target="_blank"> High-Stakes AI"</a> published earlier this year was the first we’ve come across that attempts to quantify and provide empirical evidence of “bad” data’s impacts.<h3>Overview of the Paper</h3> AI models increasingly are being applied in a broad range of domains from healthcare to climate change. Based on interviews conducted with 53 global AI practitioners working on a variety of applications, the paper’s researchers define, identify causes of, and present empirical evidence on the problem of data cascades. Data cascades are compounding events caused by data issues that lead to negative, downstream effects, resulting in the accumulation of tech debt over time. For more details on the paper’s findings, we encourage you to read the full research <a href="https://storage.googleapis.com/pub-tools-public-publication-data/pdf/0d556e45afc54afeb2eb6b51a9bc1827b9961ff4.pdf" rel="noopener noreferrer" target="_blank">here</a>.<h3>Key Takeaways</h3> The paper notes that 92% of AI practitioners interviewed experienced at least one data cascade or adverse outcomes caused by problems in a data processing pipeline. The paper also cites several causes of cascades, but the number one cause — cited by 54.7% of the practitioners interviewed — was “interacting with physical world bitterness.” In other words, when AI systems move from their well-defined, static development environments and are deployed in the physical world, the systems are challenged by the dynamic changes inherent in the real world.<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/5e7113c0-9478-4cf4-54d3-839083523d00/public" alt="Data Cascades Causes"> Table 1: Prevalence and distribution of data cascades, triggers, and impacts. IN is short for India, EA &amp; WA for East African and West African countries respectively, and US is for the United States. Many ML researchers and engineers today are taught to develop models using pristine datasets in academic settings. In real-world applications, however, the data ML engineers must work with is messy or even non-existent. While the paper’s focus is on high-stakes domains with limited resources, even in high-resource settings, ML engineers or data scientists will hack together a data pipeline leveraging whatever tools and services they can find. The lack of unified infrastructure to support comprehensive data operations creates structural failures that ultimately cause data cascades and their commensurate tech debt. Exacerbating the issue, as noted by <a href="https://papers.nips.cc/paper/2015/file/86df7dcfd896fcaf2674f757a2463eba-Paper.pdf" rel="noopener noreferrer" target="_blank">Sculley et al., 2015</a>, complex models erode abstraction boundaries for ML systems. Developing and maintaining traditional code already results in tech debt. But that is significantly compounded by ML-specific issues — including data dependencies, analysis debt, and configuration debt — that are much harder to detect, diagnose, and fix. The longer tech debt accumulates, the more pervasive it becomes. The more pervasive the tech debt, the more time and effort ML teams ultimately must spend to resolve issues, resulting in productivity loss and slower deployments. <h1> </h1><h1> Where Scale Comes In: Minimizing Data Cascades by Focusing on Data Work</h1><h1> </h1> Scale was founded on the belief that for AI and ML, data is the new code. By providing a data-centric, end-to-end solution to manage the entire ML lifecycle, we bridge that skill gap, so ML engineers don’t have to change how they work. By focusing on the data, we enable our customers to focus on the model work, not the data work, minimizing the risk of data cascades and making AI systems safer and more robust. <h2>Data Annotation</h2> We started this journey by focusing on data annotation. When we first began providing data annotation solutions to our autonomous vehicle customers, our quality assurance (QA) systems were highly manual. Human QA Agents or Queue Managers would audit and re-check sample tasks and manually assign scores to individual Taskers. Over the years, as we expanded into new use cases and new industries, we have built out a variety of internal tools to automate many of these QA processes including: <h4>Automated Benchmarks</h4> With benchmarks, we programmatically take an annotation task we know is correct, create an error in that task (usually a common mistake we’ve seen Taskers make), and serve them as regular tasks to our Taskers. By leveraging benchmark tasks, we can automatically give Taskers scores based on whether they fixed the error before submitting. This then allows us to understand how well each Tasker is performing without manually scoring each individual on a variety of tasks. <h4>Linters</h4> Linters are tools designed to catch errors and flag anomalies. At Scale, we leverage a variety of types of linters. For example, a response-based linter looks at a submitted task and checks for possible errors such as annotations that should not exist, like duplicate bounding boxes for the same object. Over time, we have layered response-based linters into the UI to flag issues for our Taskers as they annotate -- for example, warnings when the size of a bounding box appears too large. Moving beyond our core expertise in data annotation, our team has been hard at work building products to support the entire ML lifecycle. These include: <h2>Data Collection</h2> Collecting diverse and representative data, which is the first step in developing a machine learning model, can be a challenge, especially for content and language use cases where there is much more subjectivity. Leveraging our existing global Tasker base, our text and audio collection workflows are API supported and seamlessly integrate with our data labeling pipeline for seamless end-to-end dataset creation. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/uq2zsk4cy0"></iframe> <h2>Data Management</h2> While more training data is generally better than less data, once data is collected, ML teams need to select what data to annotate. Today, most ML teams do this either by randomly selecting data to label, or manually parsing through their data to identify valuable examples. Scale Nucleus uses active learning and advanced querying to help ML teams visualize their data, understand over- or under-represented labels and attributes to help mitigate bias, and identify edge cases to address the long-tail challenges of machine learning development. Nucleus also enables model-assisted QA, using model predictions (model errors in particular) to identify errors in labeled data and to improve data quality. In the video below, we show how active learning powers the Autotag feature of Nucleus to allow users to automatically search for and tag similar images — in this case flowers. <iframe class="ql-video" frameborder="0" allowfullscreen="true" src="https://scaleai.wistia.com/embed/medias/yocc96733p"></iframe> <h2>Data Generation</h2> The ability to understand how ML models will react to rare or dangerous real-world scenarios is crucial for safety-critical computer vision applications. However, it’s not always feasible to collect enough examples of edge cases in the real world. Currently in alpha, Scale customers can augment ground-truth training data or generate scenarios from scratch to simulate a variety of cameras, sensors, or environmental conditions such as fog or night-time lighting. With data generation, ML teams can synthesize specific, troublesome scenarios (often pre-labeled) to expose their models to more edge cases and under-represented scenarios than they could otherwise collect. <h1>Real-World Impact</h1> As we noted in our <a href="https://scale.com/blog/series-e" rel="noopener noreferrer" target="_blank">Series E Announcement</a>, we’ve already seen the impact that the right data infrastructure can bring to our customers. From 10X’ing throughput for Toyota to improving States Title’s document processing model to more than 95%, we have freed up many of our customers’ ML teams to focus on the model work instead of the data work. We’re proud to be trusted by the world’s most ambitious AI teams to build the data foundation that will set up their organizations for success for decades to come. In the coming weeks, we’ll also dive deeper into some of the topics — like how to solve for the long-tail distributions of data with better data management — on our blog. In the meantime, if you or your team are looking for an end-to-end solution for your datasets, <a href="https://scale.com/sales" rel="noopener noreferrer" target="_blank">reach out to us</a>. We’d love to hear from you.

Better Data = Better AI. Understanding the downstream impact of "bad" data and how to mitigate these issues.

Yuri Maruyama

Women’s History Month at Scale

How ML Linting Meaningfully Improves Adaptive Document Processing at Scale

Doing the Data Work to Mitigate Data Cascades