Bihan Jiang

Product Manager

On July 15th, Scale AI hosted its second Generative AI Hackathon, bringing together talented developers, data scientists, and AI enthusiasts from around the world. The hackathon fostered creative projects, featured cute puppies, and showcased the potential of Generative AI in solving real-world challenges. Hackers created projects that leveraged the power of AI to address various domains, from code generation and bioinformatics to natural language processing and immersive experiences.​​<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/EroCZrhWWegZRf0lGRpNgoFXHyzf6LRth1PfBNUntmQHLRAe1aoLRmCC-l1DLLrA9diZC7R7vnMfjz9fwF-SroDQD4Q5L0FdGB1RzZU-4uN_5Xvy6RixAwof1KY8UPAagdu-TxYDq0XfskoedOmQffY" width="624" height="468">
<h2 dir="ltr">Winning Projects</h2>
There were a multitude of remarkable projects but the standout projects from the hackathon were:
<h3 dir="ltr">First place: YOBO (You Only Build Once)</h3>
Winners: Baladhurgesh Paramasivan, Joshua Dai, Shreyas Agarwal, and Raul Molina
YOBO, or You Only Build Once, took first place. YOBO presented an online tool that streamlines the process of converting ideas into reality. The tool requires no setup and offers a visually intuitive depiction of the code architecture, making it easy to comprehend for users. Furthermore, YOBO generates initial code snippets, enabling developers to kickstart their projects efficiently. The tool can be explored&nbsp;<a href="https://yobo-scale.onrender.com/">here</a>.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/72229cc7-f5f9-42e8-a320-941e480698c0" width="658" height="323">
<h3 dir="ltr">Second place, Best use of Chroma: Protex</h3>
Winners: Shawn Dimantha, Alex Beatson, Rohit Dilip, and Ishaan Javali
Protex, took second place as well as the best use of Chroma award. Protex wowed the judges with its search engine for proteins. Researchers could describe the protein they sought, obtain CLIP-like embeddings, and locate the closest matches. This breakthrough technology can significantly reduce the time and effort required in pharmaceutical research, potentially leading to groundbreaking medical discoveries.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/mzrSe1-PIY12igOJpomB9Y1cZTp7Pa2lg3nTrxuv7obUk2fGk09YMHSbxmpCBZlASN_PJylaIpDNHeQkELp6fIqX4zbtzdhszKUGXkU4M0BotzmHXQSmfKFZHOWviYK9oiz4Mp3Mktf7U9M7ut4mCp4" width="624" height="832">
<h3 dir="ltr">Third place, Best use of Modal: Kaboom</h3>
Winners: Divya Amirtharaj, Chloe Loughridge, Jothi Ramaswamy, Moritz Stephan, and Joshua Fang
Kaboom unleashed creativity by turning direct messages (DMs) into engaging comic strips and won third place and the Best Use of Modal award. Using Generative AI, this project transformed ordinary conversations into visually captivating narratives, offering an innovative way to communicate and share stories.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/M1CH135Q9GQxotvgzINIHZUxg77BTdwvzDl4dWy4sIHQr0bAGJlKo0AQRkhZI_CS8fcbT4s4T1LYbxkIw4RuPUzKJX0qWWtDUJ8YEUbab-5GOSlpq1Xcp6aILiR83Fnaba7j5lDdP_6BrjY9Lx1R5uE" width="624" height="416">
<h3 dir="ltr">​​Best Use of OpenAI and Audience Choice Award: Enable</h3>
Winners: Jayaditya Sethi and Ethan Jagoda
The Best Use of OpenAI prize was awarded to Enable. Enable focused on transforming natural language requests into fully functional, modular prosthetic devices to enhance accessibility and independence. The seamless integration of OpenAI's technology into this project showcased the potential of AI to create positive and transformative solutions for real-world problems. Enable also won the Audience Choice award.
<img src="https://lh4.googleusercontent.com/CGazcH_plwFscABEWZqvWR2X-koQCzwidO99OVmiosAE03-H4xjjSIXrrKmdoIvrY_YSQNATfn6-R4ikL1BiNnr2JaVN8vVxfEP4ZFs_diFq8QKj-xc692_xSO6omKJfCRH0wwBwkLy2JVkvdBEI0_A" width="624" height="367">
<img src="https://lh5.googleusercontent.com/OFw1nT-WYJTlF9RKvtfMcQxZsn6CtRp81vVpGx4FRkA3ljmH74wYCE38ol4ZOsUeR6UBWWBDLGrwmrLmsDeIPdpSdNvmx9PLje8BSPuXo44H9hOjeM5Ivd39IVKCMWNa1Y4SAw0yn6taFfOOtd_FvrY" width="624" height="416">
<h3 dir="ltr">Best Productivity Hack: Bob the Cloud Builder</h3>
Winners: Mihir Chintawar and Kiran Illindala&nbsp;
The Best Productivity Hack prize was awarded to Bob the Cloud Builder. This project revolved around Agentic Cloud deployment, where the system learns common patterns and combines them to optimize cloud-based operations. This creative use of generative AI opens up new possibilities for streamlining cloud deployment processes, saving time, and boosting overall productivity. Bob the Cloud Builder's impressive implementation stood out for its practicality and efficiency, making it a well-deserved winner in the hackathon.
<h3 dir="ltr">Best Frontend (Vercel): Agent83</h3>
Winners: Kai Hayden, Ryan Xu, and Shahan Nedadahandeh
Agent83 won the Best Frontend prize sponsored by Vercel. Agent83 is an AI-powered search engine that utilizes natural language queries to perform keyword-based searches on an LLM-generated search index. The intuitive frontend developed on Vercel showcased the team's design prowess, making the user experience engaging and effortless.&nbsp;
<h3 dir="ltr">Best Use of Spellbook: Dream with Vision Pro</h3>
Winners: Molly Cantillon, Sigil Wen, Rahul Chalamala, and Jonathan Lei
Dream with Vision Pro secured the Best Use of Spellbook prize. Dream with Vision Pro was an inspiring lucid text-to-3D tool built with the Apple VisionOS SDK. Harnessing the capabilities of multiple AI technologies, including Scale&rsquo;s Spellbook platform, OpenAI's GPT-4, and Shap-E, Modal, Replicate, and the Meta Quest 2, this project allows users to transform their imaginations into stunning immersive experiences. The clever integration of Spellbook demonstrated how AI could be leveraged to unlock new levels of creativity and enable users to bring their dreams to life in vibrant 3D environments.
<h2 dir="ltr">Prompt Engineering Part 2 with Riley Goodside</h2>
Riley Goodside, Scale&rsquo;s Staff Prompt Engineer and a leading figure in the AI and prompt engineering domain delivered an insightful session on Prompt Engineering. His presentation dove deep into advanced techniques to fine-tune AI models effectively. Attendees learned about cutting-edge strategies for prompt design and optimization, gaining valuable insights that they could apply to their own AI projects.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/ll0dNu6l8oZ2IV1uivcfoKPJjpG2YX05cmR5-L7k2XlgU6K5PiIBzdebWs5N9FQJOGa_xBnZhG8gSGKldZd8YFvCnSbPv0fsPegNsGO75GamyNCnVFEzR442bzdDg4Irj2M_ia61WzSPTaLN75vNMt4" width="624" height="416">
<h2 dir="ltr">Conclusion</h2>
Our Second Generative AI Hackathon showcased the potential impact of Generative AI across diverse fields, including prosthetics, biotech, and infrastructure. The winners, YOBO, Protex, Kaboom, and other remarkable projects, demonstrated the power of AI-driven hacks in simplifying tasks, accelerating research, and creating captivating experiences. As the AI landscape continues to evolve, Scale will continue to host events like this to bring together builders and inspire the next generation of AI enthusiasts to push the boundaries of what is possible. The Hackathon was generously sponsored by our partners Index Ventures, Chroma, Vercel, Modal, and Cometeer. Congratulations to all the participants, winners, volunteers, and sponsors for making the Second Generative AI Hackathon unforgettable!&nbsp;

In July, Scale AI hosted its second Generative AI Hackathon. This blog provides an overview of winning projects and a recap of the event. 

Scale's Second Generative AI Hackathon | Blog

Scale's Second Generative AI Hackathon

As we close out Women's History Month, we reflect on the incredible contributions of women throughout history and their importance in shaping our world today. In particular, we want to highlight the vital role that women have played and continue to play in STEM (Science, Technology, Engineering, and Math) fields.
Despite facing numerous challenges and barriers, women have made significant strides in STEM. From <a href="https://www.computerhistory.org/babbage/adalovelace/">Ada Lovelace</a>, the world's first computer programmer, to <a href="https://www.nasa.gov/content/katherine-johnson-biography">Katherine Johnson,</a> who helped put the first human on the moon, women have been pioneers in fields traditionally dominated by men. However, there is still much work to be done to address the gender gap in STEM, particularly in terms of representation and pay equity.
Throughout this month, Scale&rsquo;s Women ERG spoke to a few of its members to showcase the perspectives and experiences of women working in AI. We heard from Scaliens who have consistently broken barriers and achieved success in their fields, as well as those who have faced challenges and obstacles along the way.
We hope that by sharing these stories, we can help amplify the importance of diversity and inclusion in STEM and inspire the next generation of female leaders in the field. So, join us as we celebrate Women's History Month and recognize the incredible contributions of women in STEM. Together, let's continue to work towards a more equitable and inclusive future for all.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/503f7e9b-f8aa-4580-a096-2cba1e2402d6">
&ldquo;Having it all&rdquo; as a woman means different things to different women. I think this is important to acknowledge as I discuss what it means to me.
Last year, I began pursuing my MBA on top of my full-time job. When I first started, I quickly discovered that my definition of &ldquo;having it all&rdquo; was exhausting and unsustainable in my new season of life. It was time to change that definition.
Oftentimes women tend to take on more responsibilities both at work and at home. This was exactly what I was doing in every aspect of my life. It forced me to prioritize the limited free time I have. I had to set boundaries in my work and personal life to make room for school. I highly recommend The Book of Boundaries: Setting the Limits that Set You Free by Melissa Hartwig Urban as it was immensely helpful for me.
&ldquo;Having it all&rdquo; not only means different things to different women. It also means different things in different seasons of life. In my current season of life, &ldquo;having it all&rdquo; means shifting priorities and setting boundaries that help me succeed where I am today. When I finish my MBA, I&rsquo;ll enter a new season of life where &ldquo;having it all&rdquo; is going to evolve. And I can&rsquo;t wait to see what that looks like.&nbsp;
Yuri Maruyama
Head of Product Marketing
I joined Scale four and a half years ago when Scale was a forty-person company. At the time, I didn&rsquo;t think it would be possible for me to start a family while working at a hyper-growth, early-stage startup. It took a few years but I welcomed my first baby last summer. I was able to take and enjoy my full parental leave without getting pulled into any work or feeling any guilt.&nbsp;
Once I came back, I worried about how I would balance being back to work full-time while also being present for my family. I still often fall short of my own expectations, but to manage this, I&rsquo;ve blocked off my calendar in the mornings until 8:30am and in the evening between 5-8pm to spend with my family. Blocking this time and holding this boundary has helped to make me more present and productive at work during the 8:30am-5pm time when I am 100% focused on work. At the same time, I&rsquo;ve been able to be present for my family in the mornings and evenings without feeling guilty or like I need to check slack or email.
Bihan Jiang
Product Manager
Balancing work and life is both a challenge and a necessity. One way that I&rsquo;ve found balance is by scheduling time for activities that I enjoy, such as exploring coffee shops (I drink 3+ cups a day!), traveling, watching dramas, and playing board games like Catan and Tractor.
It&rsquo;s also critical to find passion in your work. As a product manager on the Spellbook team, I&rsquo;m responsible for driving the development of our product, collaborating with cross-functional teams, and staying on top of industry trends and customer needs. It can be challenging, but it&rsquo;s also incredibly exciting to be building in the generative AI space, which is evolving every day.
Achieving a balance between work and life is a continuous effort that requires prioritization. By making sure to allocate time for my hobbies and interests, as well as having a support system in place, I am able to maintain a strong sense of self while excelling in my career.
Karena Zhang
Account Executive
Passionate about living as a true &ldquo;Corporate Athlete,&rdquo; I find a myriad of ways to stay active and challenge myself in the office and beyond. When I first began my career, I was a Reformer Pilates instructor part-time while working full-time in tech. Now that I have since retired from being a fitness instructor, I have taken on a new challenge of traveling the world by way of sliding down mountains on wood planks (skiing). I was also a competitive college tennis player at Carnegie Mellon, so I am not new to navigating the stress of work and play, but I know how important it is to have a strong sense of self outside of work. I believe that no matter how stressful things get at work, I still have to live my life. Calendar blocks plus a passion for planning allow me to balance working hard, having fun, and making history.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/f383b465-2282-480b-bdb2-246cb1198506">As a female engineer, I have experienced being overlooked or underestimated many times in my career. Most of the time, there's nobody to blame--these biases are just that deeply ingrained in everyone. Over time though, I have learned to use this to my advantage. When my coworkers had low expectations of me, as long as I still set ambitious goals for myself, I could make sure I blew their expectations out of the water. I constantly remind myself daily to be confident in myself and to not let my goals and abilities be dictated by the male-dominated environment I'm surrounded by. I also keep in mind that as a minority in the workplace, I can bring fresh perspectives to the table. Finding success in engineering definitely feels like an uphill battle sometimes, but that also makes success feel much more rewarding.
Iris Guo
Strategic Product Manager
Despite lots of excitement and eagerness, I didn&rsquo;t know what to expect coming into the role of a strategic product manager as a new grad. In the past, I had more traditional product manager internships at big tech companies where I was building software/ hardware features and focused on the user experiences. I haven&rsquo;t done operations before and this was my first time getting into the weeds of running a project as an STO. As a woman with high degrees of agreeableness and conscientiousness, I found it difficult to be the one calling the shots and not asking for approvals before execution. At first, I thought I had to come up with the solution for every problem, so I would put a lot of pressure on myself to do everything right and I wasn&rsquo;t used to asking for help since I didn&rsquo;t want to bother other people. I tend to be detail-oriented and diligent in my tasks, but sometimes I would miss out on the bigger picture for strategic thinking. When I was stressed out at work, I would lean into my passion for dancing as a creative channel for me to destress. Dancing helped me express my full sense of self and relieved my obsessive thoughts from work to stay present. I performed ballroom dancing growing up and I started learning k-pop/ jazz funk/ contemporary/ bachata/ salsa when I moved to SF six months ago. Every group dance required lots of teamwork to help each other to learn the harder part of the dance, and I learned to ask for help from people with more experiences, which I applied to problems at work. Through dancing, I also learned to experiment and iterate fast, because we would have different versions of the same dance and improve from trials and errors. Dancing enlightened me to see the bigger pictures and let go of the unimportant details with bold movements, which served as a constant reminder to me at work. I felt extremely grateful for having a healthy and mutually beneficial relationship between my hobby and my job, and I believe that everyone can find the right balance between work and life.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/5ac7d1f9-b882-43ac-9da9-8511d5062f95">I&rsquo;m thrilled to be a woman and working with other talented men and women here at Scale as a part of the AI revolution. As a society, we have a ways to go in terms of representation and equity across all STEM fields, and think that women have an especially important role to play in bringing diverse perspectives shaping AI products, and ensuring that the future of AI is just, equitable, and serves all of humanity better.

We hope that by sharing these stories, we can help amplify the importance of diversity and inclusion in STEM and inspire the next generation of female leaders in the field.

Throughout this month, Scale’s Women ERG spoke to a few of its members to showcase the perspectives and experiences of women working in AI.

Women’s History Month at Scale

We&rsquo;re excited to announce that <a href="https://scale.com/spellbook">Spellbook</a> is now fully available for self-serve signups!&nbsp;
Spellbook is Scale&rsquo;s platform for large language model (LLM) apps. We believe the future of data science and machine learning is people building lightweight applications on top of large language models.&nbsp;
Speed and infrastructure quality is key&ndash;how easy is it to create a new LLM app, how fast can teams experiment, and how quickly can they get feedback on whether that experiment is better than a baseline or not? Once those experiments have signs of life, how efficiently can teams fine tune them, hone them with expert feedback, and launch them into production?&nbsp;
I&rsquo;ll walk through how you can create your own production-ready LLM app in less than 10 minutes. Follow along at <a href="https://spellbook.scale.com">https://spellbook.scale.com</a>!&nbsp;
App Creation
As an easy example, let&rsquo;s get started by creating a new marketing copy generation app that will take an event, size, and demographic, and generate an event description.&nbsp;
<figure class="image align-center"><img title="test" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/c23f338d-0dd8-492a-a458-70ff95c024fa" width="624" height="280">
<figcaption>App setup page in Spellbook</figcaption>
</figure>
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/2acb3b28-73c0-4bea-8ad3-385a5aba254a" width="580" height="423">
<figcaption>Example dataset with event, size, and demographic data.</figcaption>
</figure>
Quickly set up your prompt and see how your prompt runs on multiple rows of data.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/35ca21e5-3b43-4feb-ba8e-dd6d5215c8cb" width="3159" height="1599">
Evaluate Your LLM App
Make sure your model is outputting good generation with our human evaluation pipeline. Define evaluation criteria on how a human should determine whether your model output is good or bad. Here, we want to ensure our event description is relevant.
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/6ab0f3da-e7c8-4258-af87-b4455f71a35b" width="624" height="317">
<figcaption>Define human evaluation criteria for your model output.</figcaption>
</figure>
You&rsquo;ll get your results back in the form of a &ldquo;Hit rate&rdquo; along with downloadable results you can use to fine tune your model!
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/d9584b8a-0898-41a9-ac5c-70c343a7faff" width="624" height="319">
<figcaption>Receive evaluation results back after humans have finished reviewing.</figcaption>
</figure>
Deploying Your App
Once you have a model you&rsquo;re satisfied with, you can deploy your model to production in one click through our deployments support.&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/6d34a534-3432-4925-a3de-51dfef56ef71" width="624" height="315">
<figcaption>Create a new deployment for an App Variant.</figcaption>
</figure>
You&rsquo;ll receive an API endpoint that you can integrate directly into production, with built-in monitoring!&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/e8a2dfe8-b11c-4ecf-9d2d-30bbcc04ae57" width="624" height="315">
<figcaption>Integrate your LLM app into production using our code snippets, and monitor your requests and latency.</figcaption>
</figure>
As you improve your LLM apps, you can keep swapping them out in your deployments tab by selecting the most preferable variant. This ensures that once you&rsquo;ve gotten an endpoint integrated, you don&rsquo;t have to update your code again.&nbsp;
<figure class="image"><img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/ded46a41-1762-41f0-b7a2-732f2f2e420c" width="624" height="335">
<figcaption>Swap out your deployments with updated variants as you improve the quality of your LLM app.</figcaption>
</figure>
<h4 dir="ltr">Try Spellbook</h4>
As LLMs become more powerful, the ability to quickly deploy enterprise quality LLM apps to production and quickly iterate based on user feedback is crucial. Try <a href="http://spellbook.scale.com">Spellbook</a> today to create your first apps!&nbsp;
&nbsp;

We believe the future of data science and machine learning is people building lightweight applications on top of large language models. 

 Try Spellbook today to create your first apps! 

Create And Deploy LLM Apps in Less Than 10 Minutes

<h3 dir="ltr">Introduction</h3>
OpenAI just released their new model that powers ChatGPT, GPT-3.5 Turbo. The best part &mdash; it's 10x cheaper than their previous best model.
OpenAI has found that GPT-3.5 Turbo is actually their best model for many non-chat use cases. Some early testers found it super easy to switch from the Davinci-003 model to GPT-3.5 Turbo with just a little bit of adjustment needed to their prompts. So, it's a much more cost-effective option for a lot of non-chat use cases.
Now that there is this new model available, it can be tough to know which one to use for your LLM apps in production. Spellbook makes it easy to build, compare, and deploy large language model apps, so you can evaluate GPT-3.5 Turbo against other models like Davinci-003 and FLAN to see which one performs best for your specific use case. In this blog post, we'll dive into the details of how these models differ on single-turn completions and what their strengths and weaknesses are, and share some real-world results from tests run on Spellbook- <a href="https://scale.com/spellbook">try it yourself today</a>!
<h3 dir="ltr">Classification&nbsp;</h3>
From our previous <a href="https://scale.com/blog/gpt-3-davinci-003-comparison">experiments</a>, we concluded that Davinci-003 performed much better than Davinci-002 with 0 shot prompts (92% vs 69.91%), and on par or slightly worse than Davinci-002 on few shot prompts (87.2% vs 91.6%). We observe a similar case with GPT-3.5 Turbo and Davinci-003.
GPT-3.5 Turbo performs better on 0 shot classification- on an insider trading classification example it achieves 91.72% accuracy, versus 82.02% for Davinci-003.
<figure class="image"><img src="https://lh4.googleusercontent.com/-qWbBYjvANJhwOpm25BihanYgq7goa1pw4QDINq3s6ZZedbwsHzuAOguit4S-2ug0TyVVx2DXgt7ep2J1MBiO_quaJrAWRlb4TRJJj-Te4zo5V3XHE58gNCbw_Rni-gFLx5gRxf0Aq0-bFM-aKUwN4c" width="624" height="349">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a 0-shot insider trading classification use case.</figcaption>
</figure>
However, Davinci-003 performs slightly better than GPT-3.5 Turbo with k-shot examples at 90.47% accuracy versus 88.37% for Davinci-003, but the quality is still lower than the GPT-3.5 Turbo 0 shot example.&nbsp;
<figure class="image"><img src="https://lh4.googleusercontent.com/iIefySacpNH98k4jEBgnUTSaRb5FCQiodLKt98KyvuI4-Z_iQp3lkH7bkQeKc-k-Sz2jnulr4wXT93pfU9visP46mOy0cfNXc3WVqr_b4M_meiDognTpgAL8vaHiB3cCW5eOCuhG_ZfEQPkSh6OWpYc" width="624" height="357">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on an insider trading classification use case with k-shot examples.</figcaption>
</figure>
The use of k-shot examples will likely not be very common with GPT-3.5 Turbo- including them in chat prompts can take up a significant amount of context, which is not ideal for a chatbot that receives a high volume of API requests. In cases where you&rsquo;re using GPT-3.5 Turbo for single-turn completion, this will be pretty much equivalent to adding k-shots for Davinci-003, but for multi-turn use cases, this can lead to inefficient resource usage if k-shots are included in every prompt.&nbsp;
On a more nuanced classification example, like sentiment analysis, we also see that GPT-3.5 Turbo outperforms Davinci-003 on 0 shot experiments (84.26% vs 78.57%).
<figure class="image"><img src="https://lh6.googleusercontent.com/ygoPuU_tuQoheEQ2N5Ut-7KG0dZjEap2zu6D-p4gijZ87fIfDz-zqnTY2oYaQIqWZhSMBrbrsxlGjZdFzogzJ1CZM5lgHmlsLnJkOpqSll81yjRYcioGG5BBK1xUraifOOiLojBPqj8t1HRZ7nBUwEk" width="624" height="360">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a 0-shot sentiment analysis use case.</figcaption>
</figure>
<h3 dir="ltr">Text Generation</h3>
If you're looking for an AI language model that can generate long, detailed answers to complex questions, GPT-3.5 Turbo might seem like the obvious choice since it's been trained on a massive dataset of human language and produces coherent, contextually-appropriate responses.
But what if you're looking for a model that can provide clear, concise responses? In that case, you might want to consider using Davinci-003 instead.
In our experiment, we provided both GPT-3.5 Turbo and Davinci-003 with a set of 30 questions, and asked each model to provide 30 answers. What we found was that GPT-3.5 Turbo&rsquo;s responses tended to be much longer than Davinci-003's, with many of them exceeding 100 words. In contrast, Davinci-003's responses were much more concise, often consisting of just a few sentences. Given a max tokens of 500, the average response from GPT-3.5 Turbo was 156 words (~208 tokens) nearly twice the amount of Davinci-003, whose responses were an average of 83 words (~111 tokens). 
<figure class="image"><img src="https://lh3.googleusercontent.com/XWVxJwRIeuE8s5I686NmO8WlR9DJPekjQ_TLSimXu8xm52eKrxOD0kJSyPcDqZj_mpvJ1aeA1soQ4DG8CYWsd3DqMnN4mE-SYj6Wqw7s5bu6XgdMYNQZjblaQcctic_hfJvUma8nqCEeitps7GeSucU" width="624" height="351">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a question answer use case.</figcaption>
</figure>
<h3 dir="ltr">Math&nbsp;</h3>
Notably, GPT-3.5 Turbo is significantly better than Davinci-003 at math. When evaluating by exact match with an answer key, GPT-3.5 Turbo gets the correct numerical answer 75% of the time, while Davinci-003 only gets 61% correct.
<figure class="image"><img src="https://lh6.googleusercontent.com/IG9V-yRgXR9vLTW0Myve1LV8Q8w1FpFPTs7t-YAaFXHAZQzRMUXZVvc64lb-2P6ERE79u_f_FY4eB1Y4dyK4kK59WtuEuMm-hNxQJ2C5m8J4ZuTxt1FE-C0ENR-iYK_8zXWId-SS7gy-qNIGeJ1ZGiY" width="624" height="313">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a math use case.</figcaption>
</figure>
We also selected a subset of the questions for expert human reviewers to evaluate the models correctness based on both the numerical correctness and the logical reasoning. Interestingly, upon asking both models to explain their answers, it became clear that the models would sometimes hallucinate reasoning for how they achieved the answer, even if it was numerically correct.&nbsp;
<figure class="image"><img src="https://lh4.googleusercontent.com/xmFdfVGuo3p81w5agMlTCOEuBLDkDdYHo6oniV8MYtIxc5IuqcFVti6vof3b3bb3dO-1QyOfyJ5ryVRleV6nSrPYpJc8w55vcRtnMY_KXhh7wB7HvNJ6ZB6bht9S8ijLJdgDviWDN6fJzXTrWCPQIJ8" width="624" height="315">
<figcaption>Evaluation configuration for human evaluation of math explanations.</figcaption>
</figure>
<figure class="image"><img src="https://lh4.googleusercontent.com/JRmEQBPW5OjYMxQWGH1ZyT7hVaeQ8dAUGx2Zy8qprMyCCwh1rrg1woFZzb1eSNS1sqkkgdI95cytMCljkoYVugA4tA3shO8SvP43hVQnOTW28Bkobru0sTHNqN6k1QXCjgwy9uSIcaPMIR4QxRdcBSg" width="624" height="229">
<figcaption>Quality drill down on human evaluations of math explanations from Davinci-003</figcaption>
</figure>
From the quality drill down summary from the completed tasks, Davinci-003 only got 44% of the questions correct. Repeating the same workflow for GPT-3.5 Turbo yielded 68% accuracy.&nbsp;
A couple interesting math lessons GPT-3.5 Turbo picked up that Davinci-003 struggled with:
<ul>
<li dir="ltr" aria-level="1">
Solving questions using the order of operations
</li>
</ul>
<figure class="image"><img src="https://lh5.googleusercontent.com/TLCynrXwIToi9u-tS5vFU-oBMcuKupJtA9foUG4ryegt3xwZCaGEhoPWmZpiteoUvTt8BzoRkVHqMY5L_xKOjwB2LLceaLxCNAHV0IYHJVLkVs9c7d8Cb6PnpVIMewge0AreLBYPPkBCRGqUL2MyMKg" width="624" height="276">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about order of operations.</figcaption>
</figure>
<ul>
<li dir="ltr" aria-level="1">
Plugging in variables into equations
</li>
</ul>
<figure class="image"><img src="https://lh3.googleusercontent.com/j63fVykVKFmDpt1qMIW3XPkntWiApDSvocH4oImJLdOr7quH0ESBTqDZwFSQpICgO1Idyja9iz_9UJn2cEDdOqB2z2Lv1UrRERGmb9qm9ojwBTuUwDhGBrvuZjsjeDpHYeFV2EM0sauHr3B_8shZNB0" width="624" height="147">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about plugging in variables into equations.</figcaption>
</figure>
<ul>
<li dir="ltr" aria-level="1">
The negative exponent rule
</li>
</ul>
<figure class="image"><img src="https://lh3.googleusercontent.com/sxzxXRaidO3EmcVfJE_fHvgUBwRfRkdpZ2RVhAtEO6R0tjjTdDLtueT0VtrPqDouqax6beTzb-d4m4Z3t8VwZaGopCOVj67sxPVCKzWKsloIgnx_PHftT6YsF0JEY2y3qfCn1bqp-vAILRwurbozbvc" width="624" height="212">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about negative exponents.</figcaption>
</figure>
<h3 dir="ltr">Base model swapping</h3>
It&rsquo;s worth noting that OpenAI has recommended the replacement of Davinci-003 implementations with the chat API in various instances, given its significantly lower cost of only $0.002 per 1,000 tokens. The ChatAPI can be used in a manner similar to the traditional Davinci-003 prompt format, where the system prompt is treated as the prompt body, the user message is treated as the input, and the assistant message serves as the completion.
For users who have LLM apps integrated in production using Spellbook, it&rsquo;s super easy to fork a variant, swap out the base model to GPT-3.5 Turbo and save it as a new one, and swap in the new variant, without having to write a single line of code.&nbsp;
<figure class="image"><img src="https://lh6.googleusercontent.com/Wp3bvT0IPiS0olJFhr-qiPieXViAuVBcq5q4ZAQOAcKJK8FFg2tOZp_nIsGcCdVZ0A8wdtdoAWj4CCoxMexDGD0IcUNyDPA-kFKvq0GuzHC3J_JWv4-T2oakkCKddvz13tS4OtmYzec0j-SXmLKOIws" width="624" height="325">
<figcaption>Quickly swap out your deployment&rsquo;s underlying variant to use GPT-3.5 Turbo without writing a single line of code.</figcaption>
</figure>
<h3 dir="ltr">Takeaways</h3>
OpenAI's new GPT-3.5 Turbo model offers a cost-effective option for many non-chat use cases. While GPT-3.5 Turbo performs well in 0-shot classification and math, Davinci-003 performs slightly better in k-shot classification and may be a better option for those looking for clear, concise responses that get straight to the point.
Spellbook makes it easy to build, compare, and deploy large language model apps, so you can evaluate GPT-3.5 Turbo against other models like Davinci-003 and FLAN to see which one performs best for your specific use case. With the ability to fork a variant and swap out the base model to GPT-3.5 Turbo, users can seamlessly integrate the new model into their LLM apps without writing any additional code.
Overall, GPT-3.5 Turbo performs better than Davinci-003 on most tasks at 10% of the cost - but is much more rambly. You can test out whether GPT-3.5 Turbo is better for your use case through <a href="https://scale.com/spellbook">Spellbook - sign up today</a>.

GPT-3.5 Turbo performs better than Davinci-003 on most tasks at 10% of the cost - but is much more rambly. You can test out whether GPT-3.5 Turbo is better for your use case through Spellbook

Compare the API behind ChatGPT to existing Davinci models using Spellbook

ChatGPT API: Cheaper, Faster, Wordier

Scale hosted its first ever generative AI hackathon at our San Francisco headquarters on January 21st! We welcomed over 300 developers, engineers, and hackers from across the country to come in and create something awesome.&nbsp;
From side projects to chatbots, from work projects to generative art pieces &mdash; hackers submitted over 80 projects. The Hackathon was also buzzing with workshops where attendees could learn about topics like Advanced Prompt Engineering with&nbsp;<a href="https://twitter.com/goodside">Riley Goodside</a>, who helped attendees incorporate the art of large language model prompting into their projects.
Besides just meeting people and hacking, hackers could walk away with cool prizes, like win Napa tours, skydiving, drones and more.
<h2 dir="ltr" style="text-align: center;">Winners Feature&nbsp;</h2>
<h4 dir="ltr">First Place: GPT is All You Need for the Backend</h4>
Taking first place was a project called &ldquo;GPT is all you need for the backend&rdquo;. This innovative project automates backend processes with a Large Language Model (LLM). It can infer business logic based on the name of the API call and maintain state in the form of a JSON (lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate) blob. With this new tech, developers are now able to create an entire backend and database with no manual logic programming. Following the instructions, they can define their application and proceed to jump straight into making API calls.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/ck-TTZe5zkMITFldY26SHLQaImb5xCNsw4kury6by8yGjLBOfiLI7ct8VW5hjrfdjxIlrUPpyEvJaDeG_ww_rtbN1XECH501cxwgth3DyGWLYGLE1aEWtyZWWoKRRAycJMkXEja4mwIJCWwxbU7f5Jg" width="340" height="372">
<h4 dir="ltr">Second Place: Chempilot</h4>
&ldquo;Chempilot&rdquo;, the AI-enabled SMARTS (SMiles ARbitrary Target Specification - a chemical language used to describe molecular structures and substructures) analysis project, earned second place at the hackathon. Chempilot makes it easy to search chemical databases and guide generative models, by allowing users to generate SMARTS queries from natural language descriptions of molecules. This groundbreaking project was built by leveraging GPT3 and <a href="https://scale.com/spellbook">Spellbook</a>, while combining that with open-source datasets and libraries to build the dataset. Chempilot is incredibly useful for drug discovery, making it faster and easier than ever to find and analyze the right molecule. It&rsquo;s a unique combination of AI, chemistry, and cheminformatics expertise, making it an invaluable tool for any lab.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/P6PJGzsX8v9SKF7TTTRuUofrsaeXFPrIc_wZMnlVOELEE4IO8TNUFeE_Dw6uZfFAISPOw7qyQWG3LjwTqbLzANWcTlmEubMIwTbK9a-jHXFTyQwVL7k0DSTT8ol0bc9xEkpyN3dwRklNzn2wJ06ij_s" width="338" height="442">
The team behind the 'Rap Battles' project made waves with their use of generative AI to create rap battles, and won third place, as well as the Audience Choice award. Through the use of GPT-3 for lyrics, wavenet for vocals, and stable diffusion for the avatars, users can input the names of any two people and generate a rap battle between them. The team even took it one step further and launched the website on the day of the hackathon, providing a live demo of George Washington vs. Andrej Karpathy!
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/9XsTs_8nut6Kyi8GeRz_YcxhGgHSKyzbOy7Iz7K4K62zB128aEFl12sfkXLVZnDrlMdgoNOchwk26-PRp1UkhHkiExItA3y7dU_RZzubBN2oX-Rhmj_RjE1Y0NHeuI8PLnO255d0DVhueKc7zZdex18" width="470" height="385">
The project &ldquo;Node Fixer&rdquo;, won the Best Use of Spellbook award, and is the perfect tool for automating corrective action for syntax and runtime errors in JavaScript programs. With a combination of a VSCode extension, a rust binary, and Spellbook, Node Fixer allows developers to quickly identify and repair errors that cause their program to crash. When an error is detected, Node Fixer sends the error and the code to a Spellbook app, which then analyzes and produces a fix. The fixed code is written back into the editor, ensuring the developer can keep their program running smoothly and quickly. With Node Fixer, there is no need to manually fix trivial errors in your code.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/TjZZYOgCKD_RECp4Giu5cuB7TPqLWKdG22HokGYb1ovBnsfIi_UaWJG4zVasm8Q07HqkDWHI0BMsVrtPyj6RNXxwlJeLPQprqy4OvMBlSyXq3cVDjoy8L4zOkPEy2gSCyQvmcEHRG4DpspRFSDFpOnU" width="690" height="422">
<h3 dir="ltr" style="text-align: center;">Advanced Prompt Engineering with Riley Goodside&nbsp;</h3>
Riley Goodside also hosted a workshop on Advanced Prompt Engineering. Prompt engineering is the magic behind top-notch language model responses. A well-written prompt is crucial for avoiding off-topic, inconsistent, or offensive output. It's a challenging task that requires a nuanced understanding of how to talk to the models- there are research conferences and papers written on the best prompting techniques. The workshop garnered massive interest, and hackers filled up the office to learn the most innovative prompting strategies!
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/nbSgB9Mk6zXljYm7Rv4Y1i326SZzI3MTwQZUVFFUY0hD4Ifn6hLim_XqYbjTua-Vhoy35iAxvc54kv1TI4bXsiifFhrIEsP_L67WXo7AbecvahqxzvvMtRomljV2PdlNU0RuC4aC4E6D9TAQyMAVWQM" width="624" height="468">
Our hackathon was more than just a platform for hacking on cutting-edge projects, it was a community building event for passionate builders. You could satisfy your sweet tooth with Smitten's liquid nitrogen ice cream, bond with some furry friends at the puppy cuddling station, bring your own personal style to life with live screen printing on hoodies and tees, and even fuel up with a shot of espresso from the coffee bar. With so much to do and so many amazing people to meet, the hackathon was a prime opportunity for hackers to dive into generative AI and meet legends in the field.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/lx3H3DoC3k97GX7gLEe0Vsywhp5XR1c_TfRhIFQmzQgC2sqiyM98SP06fptcbPU_-iN7yr1gizlfWJq_kfz9BEZx3XbZJcPnJqvqLKd4ejyvdCfUhJVs08DtIfs3p-9PCDCbAOq6r9MSLV2FlGEpQ3Q" width="624" height="416"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh5.googleusercontent.com/UNaO30_derXRDPyGA2uFOjtJKkx9sVEb0J443_9eCnpu0L8_jXGQQ2S95dWAipTQ6wR6jJwPQMzUqF5vg2SqEFw8NWXTdVG407KhzsMk1GyqpZgHk8pcKGsvNF2TR6_0rfbyAjJw4OMbnEXBQyiKsgA" width="624" height="416">
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/SNojvekmg8mkVfpiAQPN7xpcQY7m6RQ0PEMmHTjueXISSRbPxs6AJ1RA6dE0el_6il7xa_3zy28GpfsQm8MvUV-g5VDxpMsb0m9b6LpjaiJgL4D8cYBk6EtV6hN8bXbXg3TpR2qMaI7StG8sLXJFxv8" width="624" height="416">
P.S. some of our favorite tweets about the hackathon :)&nbsp;
&nbsp;
<img src="https://lh4.googleusercontent.com/BNVKBDmYBj9GqY9w8ouGl2VFyr3SrK9dGf3w4I7L4WqZLQfAqiTDY_V_XB4BsBQrtauIE2lhnQ_idvzMIBIEG3eLOn4CEyXOhFUvHs7TLUHWp_PO36mWXeHXIfz5n6nGGoAlmwvSSVUa563UtBhkPcE" width="293" height="370"><img src="https://lh6.googleusercontent.com/BNTQxrxaDMWA94V4U_x5TqIfxvuQ_Gi4ZoF2EQndeFJ6dmun4V6NCr0MzCLWx6WdqTF07ylssd8M3Plr_1z34EaIuewbFVPiqI_bYODLxf7lklcOiiw42smrx_ihG5vp77VRpHrrGaZRz1okFRMZIlg" width="283" height="370">
<img src="https://lh4.googleusercontent.com/0ZcXCryx9vrkw9oyYu6zjT-wHLjFqM06sI11qIfgVAT7goE4tuw7qJZ6bY6sMbhgQUOxaFOvAT9QpDfM-4A4cF2Oiy_NMVSgGC85mtNK_Csq0sNny9l_WXCeIsjGUgML4rtYPgZYSb7QaKHzD_cjq-k" width="308" height="370"><img src="https://lh3.googleusercontent.com/qFWKetCapC-7aM4QbvTcu6lYU3_IWfgeo0RGYURuUO_7TONrtWEwbN21rtRcEDpN6tQzrLPr6UDkbACRlU1iQfZPKHcSKjH60zBiifS8csL47ULDQmqgOCraalVWh53zuhikF_WpJ5sMUww1MSsDNKI" width="256" height="370">
<img src="https://lh4.googleusercontent.com/XN5UpVoL4ue6mXgJozDI_6zX_zmAGKaMcZJKZDyjzd3jZQbU8ESShOMm3QTrIHQ-x6R4Fpcshpo-iwe43pjejP-QtE6uyj6DAcKKjm1yRMRpNQLFVzoQYsckpFR9pAG40scdDib3YeV8fZljGCyiMcg" width="301" height="370"><img src="https://lh3.googleusercontent.com/JgCQ71IhQOuqldXpvwLDY39GK95sgJJ1dNoaiCt107BCYSGk6TBa6Yg-ulJxXCD61J0B-CFlMvFbMYlhd0u7FdOU0UkT-4mpcjL7PfogbAVg2ZMyal7oMNzW4mkNTbqDczLJF1wQYRelKzIeyX5LJFU" width="283" height="370"><img src="https://lh4.googleusercontent.com/xUuNd3LblslS9aYrV8r-UWRNmvwbS28b7sMi-wgKiP5lGVWpHKEumgjARwwS05k9LZlOl7MQ2pjFopyGhCEqlZZrwlpAUgKnjzOowlpZb6XQUHtJnjdZh0Lj1SeelrXpBylEya3ZN4WXsB_DqOjIkgw" width="292" height="266"><img src="https://lh3.googleusercontent.com/crVzJnUXiAndcc-IcQ5kCFhMFHtuSIKEpQDFSMYGJFl_11XRI2Paw2EyAnIT5zQ9bnGyWyB1cVypEW-VEpQdLy6XqAgtDMQbs2GeRU1YRiVf6d9X-gvmjeFO9kEuVLyr3Qg45VxrbgTvQjwfERhDTIs" width="299" height="266"><img src="https://lh3.googleusercontent.com/snJGc791L74WwKlDcAVtlj4mYw7XdA82nRWRViQLSDaNuzLNPp75sqJ6bBuCvX0JBEmPXmm5AT8lQtiyJeMBZ7TzDV5_Q4EF5OPak51gQUqZfHvlkfite3JIAggOrnmnNl2L8SFmUUyTe0sY8TYFL3c" width="292" height="249"><img src="https://lh4.googleusercontent.com/De-fqScHljBxUGhJJSoEVwZTHLO0qojNg0MfvKDPQH2Qe8Hzr7KB9o4D3Xr6-296aZOsQ_PhC2TO_6tSqLcdUsJRRCfVdIMHMiwk_l269wxsNttxpeC7eObiU0JUt782CzMLgiuZa1xgt24p8a-XjGM" width="297" height="249">
&nbsp;
&nbsp;
&nbsp;

The hackathon was more than just a platform for hacking on cutting-edge projects, it was a community building event for passionate builders.

Scale hosted it's first ever generative AI hackathon

Scale's Generative AI Hackathon

Large-scale datasets are at the center of model development and evaluation. Quality of training data is the dominant factor for model performance. Anyone can create an excellent model based on great data, but even the best ML engineers can’t design good models with bad data.&nbsp;If you’ve ever used popular public datasets such as ImageNet or MNIST, you would assume that the class and attribute <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">labels</a> are correct. Shockingly, ImageNet, the most widely used dataset for popular image recognition systems, has a 6% error rate[6]. When models are fed bad data, such as incorrect classes, incorrect features, or missing <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a>, they often have severe classification problems, which can destabilize ML benchmarks and contribute to representational harms in datasets.&nbsp;Examples of representational harms[1] in datasets include under-representation of darker skinned subjects compared with lighter-skinned subjects within prominent facial analysis datasets[2,3], and millions of images of people in ImageNet labeled with offensive categories, including racial slurs and derogatory phrases[5].&nbsp;Even in leading applied ML research groups, approaches for characterizing and fixing labeling errors in massive datasets are limited.&nbsp;One approach to combat noise in a dataset is Confident Learning (CL), which is based on pruning noisy data, counting with probabilistic thresholds to estimate noise, and ranking examples to train with confidence. However, CL entirely ignores the case where one or more true labels are missing. This limitation occurs because CL relies on building a pairwise class-conditional distribution between the given and true labels[6]. Other techniques include sorting by model loss, which surfaces the examples that are most ambiguous and difficult to learn, and sorting by confidence where model and ground truth disagree (quite effective!).What is the most principled option after amassing a collection of bad labels? In practice, it would be fixing your dataset by going through the images and fixing the <a href="https://scale.com/guides/data-labeling-annotation-guide" rel="noopener noreferrer" target="_blank">annotations</a> one by one.&nbsp;This is both laborious and impractical.That’s why (1.), a way to visualize, triage, and organize your data, and (2.) a way to fix your annotations, are critical to your data annotation project.Scale provides an integrated dataset and workforce management platform that solves these problems.Let’s walk through an example of how you can use two of Scale’s products: Scale Nucleus and Scale Studio to identify and fix bad labels leveraging model predictions. For this example, we’ll be using the public <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg" rel="noopener noreferrer" target="_blank">Berkeley Deep Drive dataset</a>.&nbsp;<img src="https://lh3.googleusercontent.com/hYW3_C_Yz4vuaZKQppVBRJB273v9Nz2b9bgHBDbl-WUrdKjfHg31Ic6E2j34WJZSuXSufVkBgfWf0Q7ThfnZ-l7CxScpOViGECQXXHBhk3_VkMqYT0t-GpgGL_1J_O97Vg7wmCjTPh-PSodX-Zm7o-c">Starting on the <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg?explorer_display=image&amp;display=insights&amp;faceted_open=1&amp;filtered_model_run=run_bwmaprbx3nhg040ghndg" rel="noopener noreferrer" target="_blank">“Charts” page in Scale Nucleus</a>, a simple click into the confusion matrix shows 2,277 cases in which models predicted “person” but the ground truth label is “car”.<img src="https://lh5.googleusercontent.com/e-ElAIxlS6X1koxM2vOVAHv-MUG_IC_trcCX6K1YXyvDDPZbO4RnNOXBXRUgUaK6Apf6c9AM5-NHq67tounEbnyEqoQbMxKoxnRndm3FFaMh4CNkTbrbR7qNg_W1p5JWwvD9DuNViLj2BUt1NfahoZg">After sorting the results by descending model confidence, a brief scan of the corresponding objects makes it clear that these annotations are wrong.This may be surprising given that the <a href="https://dashboard.scale.com/nucleus/ds_bwkezj6g5c4g05gqp1eg" rel="noopener noreferrer" target="_blank">Berkeley Deep Drive dataset</a> is being used by thousands of researchers and models, but almost all datasets have these kinds of issues.<img src="https://lh4.googleusercontent.com/iPxyYxuBrw-c2817ZQiLrCxCU_xVEyC4aHDq5044UO30_qAEeXOXc3H9PnsXxiaD9BLRI4reFCGERZ1eI4KTjVRmoCifT41hOh8mGIvOOBMwwMJhG8oQuEE2aE5iiExfkkoRRu2LLbFxiDO3OkoWYeo">Now that you have discovered some errors with the data labels, you might want to correct them.With one click, you can send the bad labels over to Scale Studio, our self-service annotation platform, where your workforce can review and fix the errors. The annotation results will be automatically imported back into Scale Nucleus, overwriting the previous, flawed set of labels.<img src="https://lh3.googleusercontent.com/JWVK3_yCVI-Z1udBltlndb0T3qWfKboWTrHFaW2IFYnIoggE2PojskkIvJfQgBecKtkJtzf5f2ti7QeYrS7Cog2mPdnhHItx2-e31Uh3ZV3HM3fqcBTBQSpSrUTcCH9cTqvZphpSoO2ulpwyOLhB-o4"><h2>What’s next:</h2>At Scale, we are perpetually focused on both data and label quality. That’s why we build tools, products, and platforms to ensure that you’re getting the best results you possibly can from your ML models. But building high-performance models starts with error-free datasets. As you are <a href="https://dashboard.scale.com/nucleus" rel="noopener noreferrer" target="_blank">exploring your dataset in Nucleus</a>, verifying label accuracy, identifying edge cases, and collecting mismatches between predictions and ground truth, you may find that you and your teammates want to collaborate on adjusting or improving your ground truth labels. If you want to make these edits in house, in the most efficient way possible, <a href="https://dashboard.scale.com/signup?product=studio" rel="noopener noreferrer" target="_blank">sign up for Scale Studio today</a>.<h2>References</h2><ol><li>Crawford, Kate. "The trouble with bias." Conference on Neural Information Processing Systems, invited speaker. 2017.</li><li>Buolamwini, Joy, and Timnit Gebru. "Gender shades: Intersectional accuracy disparities in commercial gender classification." Conference on fairness, accountability and transparency. PMLR, 2018.</li><li>Raji, Inioluwa Deborah, and Genevieve Fried. "About face: A survey of facial recognition evaluation." arXiv preprint arXiv:2102.00813 (2021).</li><li>De Vries, Terrance, et al. "Does object recognition work for everyone?." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. 2019.</li><li>Crawford, Kate, Trevor Paglen, and A. I. Excavating. "The politics of images in machine learning training sets." Excavating AI.</li><li>Northcutt, Curtis, Lu Jiang, and Isaac Chuang. "Confident learning: Estimating uncertainty in dataset labels." Journal of Artificial Intelligence Research 70 (2021): 1373-1411.</li></ol>

Nucleus helps you curate data to train better models; Studio helps you use internal resources to improve your dataset in ways that Nucleus flags.

Sometimes it's helpful to bring your own labelers domain expertise to assist in both data and model quality.

Building Great Models with Better Data on Nucleus and Studio

<h1>Project Ideas and Inspiration</h1>
&nbsp;
Scale recently sponsored Treehacks, Stanford University&rsquo;s annual hackathon.
During this fully virtual hackathon, nearly 1300 active hackers and 114 teams
submitted projects for judging. Out of all teams, 21 submitted projects using
Scale&rsquo;s API.
&nbsp;
The winning project for Best ML Hack was
<a href="https://devpost.com/software/fine-tuned-heads" target="_blank" rel="noopener noreferrer">Fine Tuned Heads</a>,
which proposed a new workflow to assist with the implementation of Computer
Vision. Rather than training a single model from scratch, the encoding and
fine-tuning aspects of a task were separated. The Fine Tuned Heads team
created an automatic data collection pipeline using Flask to automatically
execute scripts to grab .m3u8 URLs from Youtube, then used Scale Rapid to
automate uploading and generating batches of ground truth labels for these
images. According to the Fine Tuned Heads team, &ldquo;this approach can be used to
improve the efficiency of inference in ML tasks, and by engineering our
self-supervised model with an efficient architecture and deploying on
efficient hardware, we can reduce the carbon footprint by over 100x.''
&nbsp;
An additional winner of the API Challenge was
<a href="https://devpost.com/software/accecity" target="_blank" rel="noopener noreferrer">AcceCity</a>, which provides
an ML-powered mapping platform that enables cities, urban planners,
neighborhood associations, disability activists, and more to identify key
areas to prioritize investment in. For their project, the AcceCity team first
used the Google Maps API to send images of street views to their backend. They
looked for handicapped parking, sidewalks, disability ramps, and crosswalks
and used computer vision, by custom-fitting a zero shot learning model called
CLIP from OpenAI, to automatically detect those objects from the images. They
then tested the model using labeled data from the Scale Rapid API.
&nbsp;
Other interesting projects include using Scale Rapid to label map data to
promote sustainable living, and classify patient medical image data.
&nbsp;
Beyond the examples from the Hackathon, one can use Scale Rapid for a variety
of ML projects, including to
<a href="https://scale.com/blog/rapid-cyclegan" target="_blank" rel="noopener noreferrer">anime-fy images</a> and to
train a
sentiment analysis model for movie reviews. Use our <a href="https://scale.com/open-datasets" target="_blank" rel="noopener noreferrer">open datasets</a> to draw
further inspiration for your projects.
<h1>&nbsp;</h1>
<h1>How can you use Scale to build a Successful ML Hackathon Project?</h1>
&nbsp;
One of the most challenging problems preventing teams from creating novel ML
projects in a short timespan is the &ldquo;Cold Start&rdquo; problem, which is when an ML
model cannot draw inferences for users or items about which it has not yet
gathered sufficient information. Especially during the compressed timespan of
a hackathon, obtaining high-quality training data for models is nearly
impossible.
&nbsp;
Scale Rapid is the fastest way to obtain production-level quality labels, with
no data minimums. Using Scale Rapid, your hackathon team can experiment
quickly by:
<ol>
<li>Setting up projects in minutes and receiving initial data within hours</li>
<li>Iterating over edge cases and instructions by getting real-time feedback</li>
<li>Scale to production-level pipelines with precision quality control</li>
</ol>
&nbsp;
To get started, sign up for an account with your university email on our
University Program Page to
receive $250 in Rapid Credits, or check out our
<a href="https://colab.research.google.com/drive/1XZjbVtA2oGOdMp0-v0E80ujMB8tbeh7x?usp=sharing" target="_blank" rel="noopener noreferrer">quick start python notebook</a>.
Rapid currently supports the following annotation types:
<ul>
<li>General Image Annotation</li>
<li>2D Semantic Segmentation</li>
<li>Text Collection/Categorization</li>
<li>Document Transcription</li>
<li>Named Entity Recognition</li>
<li>Video Playback Annotation</li>
<li>Lidar Annotation</li>
</ul>
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/2ac2bcbb-1e7d-4268-2c77-bf6d2623c400/public" alt="Annotation Types Supported in Scale Rapid">
&nbsp;
&nbsp;
If you have your own unlabeled data, add it to your project using local file
or csv upload, import it from previous projects, or upload it from the cloud
(S3, GCP, and Azure). Otherwise, check out our
<a href="https://scale.com/open-datasets" target="_blank" rel="noopener noreferrer">open datasets</a>.
&nbsp;
&nbsp;
Once you receive task responses, visualize your data in Scale&rsquo;s dataset
management platform, Scale Nucleus, by navigating to the data tab under your
project and clicking &ldquo;Visualize in Nucleus&rdquo;. Nucleus helps you understand
the strengths and weaknesses in your dataset, analyze the long tail of your
dataset, collaboratively edit labels, and explore metrics, failure cases,
and confusion matrices linked to your training data.
&nbsp;
&nbsp;
<img src="https://imagedelivery.net/wLbZE4_NzVVdgHc15St55g/899fda4e-d4ab-4ed4-7ef4-5fa12e396200/public" alt="Scale Nucleus &lt;&gt; Scale Rapid Integration">
&nbsp;
&nbsp;
Quickly find similar images using the &ldquo;Autotag&rdquo; feature in Nucleus, and
further refine your set of images by providing a few examples of positive
and negative examples. After identifying this subset of your dataset, you
can send it back for labeling through Scale Rapid.
&nbsp;
&nbsp;
Scale AI&rsquo;s products help enable and strengthen a team&rsquo;s ML development cycle
immensely by delivering high quality data for any project. As you or your
team continues to maintain and improve your project post-hackathon, use
Scale AI for consistent, reliable data in order to create, validate, and
maintain production for your high-performing ML models.
&nbsp;

Scale AI sponsored Treehacks: Learn how Scale Rapid powers successful ML hackathons for fast, efficient development.