Aerin Kim

Aerin is an engineering manager at Scale AI, leading a Catalog ML team. Below are Aerin's stories about what she does and why she loves working at Scale.
<h2>What brought you to Scale?</h2>
Before I joined Scale, I worked on a project called NL2SQL which stands for "Natural Language to Structured Query Language" at Microsoft. NL2SQL is a translation problem that automatically converts a user's spoken-language query into SQL.
Just a few months after my team had started working on it, we were beating the SOTA (state-of-the-art) performance on research benchmarks. We achieved this by evaluating different ML model architectures and improving them. In other words, we somewhat solved the NL2SQL problem in research after only a few months of work.
However, in production, we weren&rsquo;t breaking the state of the art. The model still made a few mistakes on the user's input. And it became clear to me that changing the model architectures would only get us so far, and that the most crucial thing we could do to improve the performance of NL2SQL was to improve the training data. This was done by generating edge cases synthetically and therefore increasing the training data coverage.
So, we started synthetically generating training data (NL-SQL pairs) by following syntax requirements, snowflake schema, etc. However, the data that we have artificially created couldn't possibly cover every possible scenario or edge case. Because the true edge cases are ones that you can't predict or plan for, like the questions that people have typed into our search query bar.&nbsp;Another way to increase the coverage of the training data was to label the user inputs and feed them back to the training data. Taking care of that lifecycle, also called an auto-labeling pipeline, was quite an operation. Even though I was given a lot of resources to manage the auto labeling pipeline, I couldn't undertake any other engineering work because it required so much overhead.
So I started looking for the best third-party training data provider in the market, and that's how I met Scale, and the rest is history.
<h2>What do you enjoy most about working at Scale?</h2>
I like that I get to work on problems that are important and challenging enough to customers that they are willing to pay to have them solved. This means that the problems are real.
I also like that Scale's work is always about data, which is my favorite subject. For every business, I believe data is the most valuable and private asset. Say if you're developing a machine learning model, the one and only input that determines how well your model will perform is your data. The fact that customers trust us with their data and allow us to manage it energizes me, and I feel a great deal of commitment and responsibility. Because the data we provide with them is being used as a bloodline for their business, this relationship just can't work without trust. Having a close relationship with customers is also a rewarding part of my job.
Another highlight of my time here at Scale is the everyday connection with my team. When the engineering team works well, it feels like a well-oiled machine, and you will notice tangible results from that. We've done some seemingly impossible jobs together, and it feels amazing to see the whole team pulling together to complete a challenging project. When working on projects, you learn a great deal by openly debating and choosing the best technical solution with your team. If you're an engineer, I believe that seeing and interacting with other engineers is one of the best ways to learn and grow. That's also something I enjoy at Scale.
<h2>What makes working for a startup unique from other jobs?</h2>
I've worked at both a big tech company and a startup, and they are indeed quite different, and each one is unique in its own way.
At a startup, leading a project feels like driving a sports car. You're close to the ground and can feel the movements you make. You can turn quickly, and you can observe fast motion while you are steering. You can see the results of your team's work fairly quickly and clearly. For example, you complete a pilot project successfully, and the customer signs a large contract. There is also a good chance of crashing - for example, you fail to deliver a project and lose a customer. At a startup, you will see the results of what your team does right away.
At a big company, running a project is more like flying a spaceship (I can only imagine this). You&rsquo;re farther from the ground, it's harder to make quick decisions because you are not the only stakeholder, and the course has been set long in advance. There are also very well-built manuals and processes to guide you along the way.
So far, I've enjoyed seeing the immediate, direct effects of my team's work at Scale.&nbsp;
<h2>What's your favorite part about being an engineering manager at Scale?</h2>
One of the things I like about being an engineering manager is that I get to work on large-scale projects with a strong, cohesive team, which results in a scalable solution and a greater overall team output. Apart from the collaborative and multiplier nature of the job, I also enjoy problem solving at Scale.
People think that a software engineer's job is to write code, but I don't think that's a complete picture of an engineer. A more accurate job description would be "a problem solver", since our main goal is to build things that help people solve problems. You won't be able to solve problems if you blindly write code, especially from the wrong angle.
I believe the most important thing to do in order to solve problems is to accurately diagnose them. So the ability to diagnose problems accurately is, in my opinion, the most important skill for an engineer to have. If your diagnosis is wrong, you will rarely be effective in solving actual problems. And you can only make the right diagnosis if you have a full, concrete understanding of the problem, the data, and the use cases.&nbsp;
While surgeons can rely on other professionals like physicians, x-ray technicians, to aid with diagnosis, ML/SW engineers have to diagnose problems on their own. We have to carefully examine the available data, debug the code thoroughly, then choose the best solution and also implement. The ML/SWE profession comes with a significant amount of both power and responsibility.
If you are someone who enjoys doing things like this (proper diagnosis and implementation), you will do well in the field of engineering.
<h2>Tell me about interesting projects that you ran at Scale?</h2>
Every project at Scale is interesting and unique in its own way, as each project has different data modality (3D Lidar, 2D imagery, NLP, tabular, etc.) I enjoy that I get to work on diverse problems and apply the peculiarities of each data modality to ML model development. For example, 3D data often comes with the new axes - yaw, pitch, roll, unlike 2D data and the sensor's effect on the LIDAR data is significant. NLP also has its own quirks, such as different types of tokenizers, vector representations, etc.
These days, my team is working on multi-modal (Language + Vision) problems. For example, given an image of a model wearing clothes, can we describe what she&rsquo;s wearing in detail (e.g. short sleeve, polka-dots, long boots, etc.) in natural language? To this end, one popular research/exploratory approach is to train a multi-task (e.g., Visual Question Answering, Image Captioning, Text-to-Image Generation, Visual Grounding, etc.) model and use it for attribute prediction.
Although I can't talk about the details of the projects I work on for clients, I do make an intentional effort to make my team's work more universally applicable, scientifically relevant, and therefore publishable. The attribute prediction work (link) that was just accepted at CVPR 2022 is a great example. One of the most common requests we get from clients is to assign visual characteristics such as color, shape, or motion to detected objects. But there aren't many researchers working on this topic publicly (though there are a lot of practitioners working on it in industry), and there aren't many benchmark datasets for attribute prediction. So we did in-depth research on this topic, presented our findings at CVPR, and received overwhelmingly positive feedback from the ML community.
The Fitzpatrick 17k project with MIT is also a good example. The Fitzpatrick scale is a 6-point scale that tells how dark your skin tone is. And this project&rsquo;s motives are quite fascinating. Because medical imaging doesn't have much data on darker skin, ML models trained on these datasets have trouble recognizing dermatological illnesses on darker skin.&nbsp;We wanted to bring attention to this problem and encourage more inclusion of images with darker skin tones in datasets. So we annotated 17k clinical images in DermaAmin and Atlas Dermatologico, the two major dermatology atlases, using the Fitzpatrick Scale.&nbsp;
We trained a few models on this new dataset and then looked at how well they work on images with different skin tones. And yes, adding more diverse data to the training set was an effective way to reduce bias.&nbsp;
Because of this work, researchers and practitioners&nbsp;are now a little more likely to take into account and examine biases in training datasets before developing ML models in the healthcare sector. Ultimately, we want to make sure that ML models trained on imbalanced datasets don't make healthcare inequalities worse by accident. With this study, we received an honorable mention award at the CVPR skin imaging workshop.
&hellip;&hellip;..
If you are interested in any of the topics that I have said above or in working to solve intriguing ML/SWE challenges, email your resume at <a href="mailto:aerin.kim@scale.com" target="_blank" rel="noopener noreferrer">aerin.kim@scale.com</a>!

Read Aerin's stories about what she does and why she loves working at Scale

Aerin is an engineering manager at Scale AI, leading a Catalog ML team

How Nick Beighton transformed ASOS into a 4 billion dollar business by investing in product catalog data

Using Rapid and Nucleus together will get you better results than using Rapid alone.

Use Scale Nucleus with Rapid to Uncover Errors Smartly

Peer-Reviewed Doesn’t Mean Perfect Data

Innovations in the field of Machine Learning (ML) continue to be driven by a dynamic research community. The ML team at Scale is proud to contribute to the field through its own research, dataset partnerships with leading Universities such as Oxford (<a href="https://scale.com/open-datasets/oxford" rel="noopener noreferrer" target="_blank">DEBAGREEMENT: Reddit 50K Dataset</a>) and the University of Waterloo and the University of Toronto (<a href="https://scale.com/open-datasets/oxford" rel="noopener noreferrer" target="_blank">CADC Dataset</a>), and collaborations with Stanford University, the University of California, Berkeley, and the University of Washington. As <a href="https://neurips.cc/" rel="noopener noreferrer" target="_blank">NeurIPS 2021</a>, gets underway, we are pleased to share details on three papers the ML team have published as part of the conference: <ol><li> </li><li>DEBAGREEMENT: A Comment-Reply Dataset for (Dis)agreement Detection in Online Debates</li><li>Authors: John Pougué-Biyong, Valentina Semenova, Alexandre Matton, Rachel Han, Aerin Kim, Renaud Lambiotte, Doyne Farmer</li><li> </li><li>Natural Adversarial Objects</li><li>Authors: Felix Lau, Sasha Harrison, Nishant Subramani, Aerin Kim, Elliot Branson, Rosanne Liu</li><li> </li><li> </li><li>Are We Learning Yet? A Meta-Review of Evaluation Failures Across Machine Learning</li><li>Authors: Thomas Liao, Rohan Taori, Inioluwa Deborah Raji, Ludwig Schmidt</li><li> </li></ol><h2><a href="https://openreview.net/forum?id=udVUN__gFO" rel="noopener noreferrer" target="_blank">DEBAGREEMENT: A Comment-Reply Dataset for (Dis)agreement Detection in Online Debates</a></h2> <h3>Paper Summary:</h3> DEBAGREEMENT is a dataset including 42,894 comment-reply pairings from the popular debate website Reddit, each of which has been labeled with agree, neutral, or disagree. We gathered interactions from five different forums: r/BlackLivesMatter, r/Brexit, r/Climate, r/Democrats, and r/Republican. Comment pairings for each forum were chosen in such a way that they generate a user interaction graph when taken as a whole. DEBAGREEMENT offers a challenge for Natural Language Processing (NLP) systems because it includes slang, irony, and topic-specific humor, all of which are common in online discussions. We compared the performance of state-of-the-art language models on a (dis)agreement detection task, and looked at the usage of contextual information that is accessible to the models during training (graph, authorship, and temporal information). DEBAGREEMENT provides novel opportunities for combining graph-based and text-based machine learning techniques to detect agreements as well as disagreements online, in light of recent research showing that context, such as social context or knowledge graph information, enables language models to perform better on downstream NLP tasks. <h3>Key Takeaways:</h3> <ol><li> </li><li>Language models trained on existing datasets underperform when run on real world data. This discrepancy is exposed when evaluating the models using the new DEBAGREEMENT dataset.</li><li> </li><li> </li><li>DEBAGREEMENT presents new opportunities for modeling diverse online interactions with text and context (authorship, graph, temporal information). Its graph structure allows for the combination of text-based machine learning (ML) with graph representation learning (GRL) approaches.</li><li> </li><li> </li><li>By modeling online discussion forums as graphs of user interactions, researchers can: 1) transform the agreement/disagreement detection problem into a sign link prediction task, and 2) use existing signed graph embedding techniques evaluated on publicly accessible signed graphs such as Epinions and Slashdot.</li><li>The sentiment and polarization of topics is not static in time and for the first time, this dataset exposes that and shows how realistic topic sentiment evolves over time.</li><li> </li></ol><h2><a href="https://arxiv.org/abs/2111.04204" rel="noopener noreferrer" target="_blank">Natural Adversarial Objects</a></h2> <h3>Paper Summary:</h3> Although cutting-edge object identification approaches have shown impressive performance, models are often vulnerable to adversarial assaults and out-of-distribution data. Natural Adversarial Objects (NAO) is a novel dataset for assessing the robustness of object identification techniques. NAO comprises 7,934 images and 9,943 objects that are unaltered and depict real-world circumstances, yet cause cutting-edge detection methods to misclassify with high confidence. When compared to the conventional MS-COCO validation set, the mean average precision (mAP) of EfficientDet-D7 lowers by 74.5 percent when tested on NAO. Furthermore, when evaluating a range of object detection architectures, we discovered that improved performance on the MS-COCO validation set does not always transfer to better performance on the NAO, implying that robustness cannot be gained merely by training a more accurate model. We look at why NAO cases are tough to identify and classify. Experiments using image patch shuffles demonstrate that models are too sensitive to local texture. Furthermore, when predicting class labels with integrated gradients and background replacement, we discovered that the detection model is dependent on pixel information inside the bounding box and insensitive to background context. <h3>Key Takeaways:</h3> <ol><li> </li><li>The NAO dataset presents a difficult robustness assessment challenge for detection models trained on MS-COCO. As part of our evaluation, we tested seven state-of-the-art detection models from a variety of families, and found that they consistently failed to perform accurately on NAO when compared to the MS-COCO validation set, which included both in-distribution and out-of-distribution objects, among other things.</li><li> </li><li> </li><li>We also described the process of building such a dataset yourself, which may be used to create similar datasets for any target dataset or pre-trained model.</li><li> </li><li> </li><li>The popular MS-COCO dataset has "blind-spots" that make it challenging to appropriately categorize these naturally adversarial objects. </li><li> </li><li> </li><li>Additionally, we show that detection models are overly sensitive to local texture but insensitive to background change, which makes them vulnerable to natural adversarial objects.</li><li> </li></ol><h2><a href="https://openreview.net/pdf?id=mPducS1MsEK" rel="noopener noreferrer" target="_blank">Are We Learning Yet? A Meta-Review of Evaluation Failures Across Machine Learning</a></h2> <h3>Paper Summary:</h3> Evaluation issues often undermine the validity of results in machine learning research. In collaboration with researchers from Stanford University, the University of California, Berkeley, and the University of Washington, we conducted a meta-review of 100+ survey papers to identify common benchmark evaluation problems across subfields. In some cases, several years’ worth of progress in certain fields may be misstated. Our meta-review surveys evaluation papers reporting on a broad range of subfields, ranging from computer vision to deep reinforcement learning, to recommender systems and natural language processing, and more. We found a consistent set of failure modes, which we organized into a systematic taxonomy. This taxonomy of failure modes helps researchers understand where different types of failures are common and provides a framework for future research. Our taxonomy distinguishes between “internal” and “external” validity failures. Internal validity failures occur within the context of a single benchmark and raise questions about the consistency or reliability of measurements on that specific benchmark. For example, running baseline methods with less hyperparameter tuning than a newly proposed method can underestimate the performance of existing methods and hence lead to overly optimistic claims of performance improvements. External validity failures, in contrast, describe failures that occur when dealing with multiple benchmarks. External validity failures occur when results from one benchmark do not generalize to other settings. One example is dataset misalignment: if data collected for a benchmark is not representative of the setting where models will be used, then results from that benchmark may not reliably predict performance in the real world. <h3>Key Takeaways:</h3> <ol><li> </li><li>Evaluation failures pose a risk to the validity of research findings in machine learning. Our meta-review of 100+ survey papers observed common failure modes across a wide range of subfields. </li><li> </li><li> </li><li>While some internal validity failures can be identified in the context of a single benchmark, other external validity failures only appear when looking at multiple datasets.</li><li> </li></ol><h2>Conclusion</h2> As we look ahead to 2022, we look forward to supporting more university research teams and driving more research initiatives within Scale. To learn more about our work and the work of many of our partners, our team hosts regular TechTalks on <a href="https://exchange.scale.com/public/events" rel="noopener noreferrer" target="_blank">Scale Exchange</a>, our community of ML practitioners. Join the community to learn more and register for upcoming talks. If you’re interested in joining our growing ML team, take a look at our <a href="https://scale.com/careers" rel="noopener noreferrer" target="_blank">careers page</a> for open positions.

An Overview of ML Research Papers Accepted At NeurIPS 2021

Aerin Kim

How Nick Beighton transformed ASOS into a 4 billion dollar business by investing in product catalog data

Peer-Reviewed Doesn’t Mean Perfect Data

Machine Learning Papers at NeurIPS 2021