experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

plasmic-graphcms

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

react-aria

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

General

People

Engineering

As we close out Women's History Month, we reflect on the incredible contributions of women throughout history and their importance in shaping our world today. In particular, we want to highlight the vital role that women have played and continue to play in STEM (Science, Technology, Engineering, and Math) fields.
Despite facing numerous challenges and barriers, women have made significant strides in STEM. From <a href="https://www.computerhistory.org/babbage/adalovelace/">Ada Lovelace</a>, the world's first computer programmer, to <a href="https://www.nasa.gov/content/katherine-johnson-biography">Katherine Johnson,</a> who helped put the first human on the moon, women have been pioneers in fields traditionally dominated by men. However, there is still much work to be done to address the gender gap in STEM, particularly in terms of representation and pay equity.
Throughout this month, Scale&rsquo;s Women ERG spoke to a few of its members to showcase the perspectives and experiences of women working in AI. We heard from Scaliens who have consistently broken barriers and achieved success in their fields, as well as those who have faced challenges and obstacles along the way.
We hope that by sharing these stories, we can help amplify the importance of diversity and inclusion in STEM and inspire the next generation of female leaders in the field. So, join us as we celebrate Women's History Month and recognize the incredible contributions of women in STEM. Together, let's continue to work towards a more equitable and inclusive future for all.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/503f7e9b-f8aa-4580-a096-2cba1e2402d6">
&ldquo;Having it all&rdquo; as a woman means different things to different women. I think this is important to acknowledge as I discuss what it means to me.
Last year, I began pursuing my MBA on top of my full-time job. When I first started, I quickly discovered that my definition of &ldquo;having it all&rdquo; was exhausting and unsustainable in my new season of life. It was time to change that definition.
Oftentimes women tend to take on more responsibilities both at work and at home. This was exactly what I was doing in every aspect of my life. It forced me to prioritize the limited free time I have. I had to set boundaries in my work and personal life to make room for school. I highly recommend The Book of Boundaries: Setting the Limits that Set You Free by Melissa Hartwig Urban as it was immensely helpful for me.
&ldquo;Having it all&rdquo; not only means different things to different women. It also means different things in different seasons of life. In my current season of life, &ldquo;having it all&rdquo; means shifting priorities and setting boundaries that help me succeed where I am today. When I finish my MBA, I&rsquo;ll enter a new season of life where &ldquo;having it all&rdquo; is going to evolve. And I can&rsquo;t wait to see what that looks like.&nbsp;
Yuri Maruyama
Head of Product Marketing
I joined Scale four and a half years ago when Scale was a forty-person company. At the time, I didn&rsquo;t think it would be possible for me to start a family while working at a hyper-growth, early-stage startup. It took a few years but I welcomed my first baby last summer. I was able to take and enjoy my full parental leave without getting pulled into any work or feeling any guilt.&nbsp;
Once I came back, I worried about how I would balance being back to work full-time while also being present for my family. I still often fall short of my own expectations, but to manage this, I&rsquo;ve blocked off my calendar in the mornings until 8:30am and in the evening between 5-8pm to spend with my family. Blocking this time and holding this boundary has helped to make me more present and productive at work during the 8:30am-5pm time when I am 100% focused on work. At the same time, I&rsquo;ve been able to be present for my family in the mornings and evenings without feeling guilty or like I need to check slack or email.
Bihan Jiang
Product Manager
Balancing work and life is both a challenge and a necessity. One way that I&rsquo;ve found balance is by scheduling time for activities that I enjoy, such as exploring coffee shops (I drink 3+ cups a day!), traveling, watching dramas, and playing board games like Catan and Tractor.
It&rsquo;s also critical to find passion in your work. As a product manager on the Spellbook team, I&rsquo;m responsible for driving the development of our product, collaborating with cross-functional teams, and staying on top of industry trends and customer needs. It can be challenging, but it&rsquo;s also incredibly exciting to be building in the generative AI space, which is evolving every day.
Achieving a balance between work and life is a continuous effort that requires prioritization. By making sure to allocate time for my hobbies and interests, as well as having a support system in place, I am able to maintain a strong sense of self while excelling in my career.
Karena Zhang
Account Executive
Passionate about living as a true &ldquo;Corporate Athlete,&rdquo; I find a myriad of ways to stay active and challenge myself in the office and beyond. When I first began my career, I was a Reformer Pilates instructor part-time while working full-time in tech. Now that I have since retired from being a fitness instructor, I have taken on a new challenge of traveling the world by way of sliding down mountains on wood planks (skiing). I was also a competitive college tennis player at Carnegie Mellon, so I am not new to navigating the stress of work and play, but I know how important it is to have a strong sense of self outside of work. I believe that no matter how stressful things get at work, I still have to live my life. Calendar blocks plus a passion for planning allow me to balance working hard, having fun, and making history.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/f383b465-2282-480b-bdb2-246cb1198506">As a female engineer, I have experienced being overlooked or underestimated many times in my career. Most of the time, there's nobody to blame--these biases are just that deeply ingrained in everyone. Over time though, I have learned to use this to my advantage. When my coworkers had low expectations of me, as long as I still set ambitious goals for myself, I could make sure I blew their expectations out of the water. I constantly remind myself daily to be confident in myself and to not let my goals and abilities be dictated by the male-dominated environment I'm surrounded by. I also keep in mind that as a minority in the workplace, I can bring fresh perspectives to the table. Finding success in engineering definitely feels like an uphill battle sometimes, but that also makes success feel much more rewarding.
Iris Guo
Strategic Product Manager
Despite lots of excitement and eagerness, I didn&rsquo;t know what to expect coming into the role of a strategic product manager as a new grad. In the past, I had more traditional product manager internships at big tech companies where I was building software/ hardware features and focused on the user experiences. I haven&rsquo;t done operations before and this was my first time getting into the weeds of running a project as an STO. As a woman with high degrees of agreeableness and conscientiousness, I found it difficult to be the one calling the shots and not asking for approvals before execution. At first, I thought I had to come up with the solution for every problem, so I would put a lot of pressure on myself to do everything right and I wasn&rsquo;t used to asking for help since I didn&rsquo;t want to bother other people. I tend to be detail-oriented and diligent in my tasks, but sometimes I would miss out on the bigger picture for strategic thinking. When I was stressed out at work, I would lean into my passion for dancing as a creative channel for me to destress. Dancing helped me express my full sense of self and relieved my obsessive thoughts from work to stay present. I performed ballroom dancing growing up and I started learning k-pop/ jazz funk/ contemporary/ bachata/ salsa when I moved to SF six months ago. Every group dance required lots of teamwork to help each other to learn the harder part of the dance, and I learned to ask for help from people with more experiences, which I applied to problems at work. Through dancing, I also learned to experiment and iterate fast, because we would have different versions of the same dance and improve from trials and errors. Dancing enlightened me to see the bigger pictures and let go of the unimportant details with bold movements, which served as a constant reminder to me at work. I felt extremely grateful for having a healthy and mutually beneficial relationship between my hobby and my job, and I believe that everyone can find the right balance between work and life.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/5ac7d1f9-b882-43ac-9da9-8511d5062f95">I&rsquo;m thrilled to be a woman and working with other talented men and women here at Scale as a part of the AI revolution. As a society, we have a ways to go in terms of representation and equity across all STEM fields, and think that women have an especially important role to play in bringing diverse perspectives shaping AI products, and ensuring that the future of AI is just, equitable, and serves all of humanity better.

image.jpg

We hope that by sharing these stories, we can help amplify the importance of diversity and inclusion in STEM and inspire the next generation of female leaders in the field.

Throughout this month, Scale’s Women ERG spoke to a few of its members to showcase the perspectives and experiences of women working in AI.

Women’s History Month at Scale

Artificial Intelligence (AI) has been a game-changer for everyone. Together with data, AI has become a key component of a robust digital core&mdash;the main source of competitive advantage for companies today. The advent of&nbsp;<a href="https://www.accenture.com/us-en/insights/technology/generative-ai" data-analytics-link-name="generative ai" data-analytics-content-type="cta">generative AI</a>&nbsp;marks the beginning of a new era, one where businesses&nbsp;<a href="https://www.accenture.com/us-en/insights/consulting/total-enterprise-reinvention" data-analytics-link-name="reinvent" data-analytics-content-type="cta">reinvent</a>&nbsp;how they work. Just search for ChatGPT if you haven&rsquo;t already, and you&rsquo;ll get an idea of the possibilities. ChatGPT is derived from a large language model (LLM)&mdash;GPT4&mdash;developed by Open AI, which is a foundation model capable of generating human-like language and carrying out complex tasks across a number of functions.
We are entering an era of AI industrialization, where entire industries stand to be transformed by new, innovative applications of generative AI. One of the highest barriers to unlocking business value using generative AI and LLMs, however, is customizing and fine-tuning the pre-trained foundation models for their unique needs, using their own data. That&rsquo;s why Accenture is partnering with Scale AI.&nbsp;
<a title="Scale AI &amp; Accenture Partnership" href="https://scaleai.wistia.com/medias/2fjp89kejj">video</a>
&nbsp;
<h3 class="p1">Generative AI partner spotlight: Scale AI</h3>
Accenture&rsquo;s collaboration with&nbsp;<a href="https://scale.com/" target="_blank" rel="noopener" data-analytics-link-name="scale ai" data-analytics-content-type="cta">Scale AI</a>&nbsp;focuses on helping enterprises use their own data on general purpose models, customizing them to suit their business needs. According to CEO and founder Alexandr Wang, Scale AI has been &ldquo;quietly powering this generative AI wave, providing the data and infrastructure for many of the top companies to actually build large-scale foundation models, including Open AI.&rdquo;
Accenture and Scale AI are collaborating to provide an enterprise LLM platform for enterprise and government clients to fine-tune generative AI models that can learn from human input through a process called Reinforcement Learning from Human Feedback (RLHF). The result is generative models optimized for human interaction that are able to improve how people live and work.
For example, Scale AI was able to help retail clients gain higher shopping conversion and sales by using AI-generated product images for their newest products. And with social media clients, AI-generated content led to more than 100 million impressions.
Could humans have done this? Absolutely &ndash; but at what cost to the business?
This is business augmentation at the intersection of human and machine. By freeing up employees from automatable, routine tasks, we can redirect energy and focus to more strategic creative work that requires human intuition, decision-making and problem-solving skills - work that generative AI cannot do&mdash;at least not yet. The combination of human plus machine results in more efficient processes, improved decision-making and greater innovation.
Success with generative AI depends as much on people and training as it does on technology. It&rsquo;s up to business leaders to take the lead in upskilling employees and reshaping jobs and tasks to get the most out of technologies like generative AI.
Thanks to partnerships like this one with Scale AI, Accenture helps leaders create business value with generative AI technology quickly and at scale.

Scale AI + Accenture: Enhancing Business Value with Human Ingenuity

We&rsquo;re excited to announce that <a href="https://scale.com/spellbook">Spellbook</a> is now fully available for self-serve signups!&nbsp;
Spellbook is Scale&rsquo;s platform for large language model (LLM) apps. We believe the future of data science and machine learning is people building lightweight applications on top of large language models.&nbsp;
Speed and infrastructure quality is key&ndash;how easy is it to create a new LLM app, how fast can teams experiment, and how quickly can they get feedback on whether that experiment is better than a baseline or not? Once those experiments have signs of life, how efficiently can teams fine tune them, hone them with expert feedback, and launch them into production?&nbsp;
I&rsquo;ll walk through how you can create your own production-ready LLM app in less than 10 minutes. Follow along at <a href="https://spellbook.scale.com">https://spellbook.scale.com</a>!&nbsp;
App Creation
As an easy example, let&rsquo;s get started by creating a new marketing copy generation app that will take an event, size, and demographic, and generate an event description.&nbsp;
<figure class="image align-center"><img title="test" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/c23f338d-0dd8-492a-a458-70ff95c024fa" width="624" height="280">
<figcaption>App setup page in Spellbook</figcaption>
</figure>
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/2acb3b28-73c0-4bea-8ad3-385a5aba254a" width="580" height="423">
<figcaption>Example dataset with event, size, and demographic data.</figcaption>
</figure>
Quickly set up your prompt and see how your prompt runs on multiple rows of data.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/35ca21e5-3b43-4feb-ba8e-dd6d5215c8cb" width="3159" height="1599">
Evaluate Your LLM App
Make sure your model is outputting good generation with our human evaluation pipeline. Define evaluation criteria on how a human should determine whether your model output is good or bad. Here, we want to ensure our event description is relevant.
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/6ab0f3da-e7c8-4258-af87-b4455f71a35b" width="624" height="317">
<figcaption>Define human evaluation criteria for your model output.</figcaption>
</figure>
You&rsquo;ll get your results back in the form of a &ldquo;Hit rate&rdquo; along with downloadable results you can use to fine tune your model!
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/d9584b8a-0898-41a9-ac5c-70c343a7faff" width="624" height="319">
<figcaption>Receive evaluation results back after humans have finished reviewing.</figcaption>
</figure>
Deploying Your App
Once you have a model you&rsquo;re satisfied with, you can deploy your model to production in one click through our deployments support.&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/6d34a534-3432-4925-a3de-51dfef56ef71" width="624" height="315">
<figcaption>Create a new deployment for an App Variant.</figcaption>
</figure>
You&rsquo;ll receive an API endpoint that you can integrate directly into production, with built-in monitoring!&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/e8a2dfe8-b11c-4ecf-9d2d-30bbcc04ae57" width="624" height="315">
<figcaption>Integrate your LLM app into production using our code snippets, and monitor your requests and latency.</figcaption>
</figure>
As you improve your LLM apps, you can keep swapping them out in your deployments tab by selecting the most preferable variant. This ensures that once you&rsquo;ve gotten an endpoint integrated, you don&rsquo;t have to update your code again.&nbsp;
<figure class="image"><img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/ded46a41-1762-41f0-b7a2-732f2f2e420c" width="624" height="335">
<figcaption>Swap out your deployments with updated variants as you improve the quality of your LLM app.</figcaption>
</figure>
<h4 dir="ltr">Try Spellbook</h4>
As LLMs become more powerful, the ability to quickly deploy enterprise quality LLM apps to production and quickly iterate based on user feedback is crucial. Try <a href="http://spellbook.scale.com">Spellbook</a> today to create your first apps!&nbsp;
&nbsp;

We believe the future of data science and machine learning is people building lightweight applications on top of large language models. 

 Try Spellbook today to create your first apps! 

Create And Deploy LLM Apps in Less Than 10 Minutes

Getting data to train and evaluate your large language models has never been more Rapid! Introducing&hellip;Scale&rsquo;s Large Language Model Toolkit!&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/6b85a60e-339a-46fd-bcb7-ce485df871de" width="624" height="391">
We&rsquo;ve developed expedited workflows to help you set up projects within Rapid, Scale&rsquo;s self-serve data labeling tool. <a href="https://scale.com/rapid">Sign up</a> today to try it out and get $20 in free credits.
After signing up, you&rsquo;ll see all of our available project templates and our &ldquo;LLM Training Toolkit&rdquo; front and center. We&rsquo;ve divided our LLM toolkit into three buckets according to different workflows:&nbsp;
<ol>
<li dir="ltr" aria-level="1">
Demonstration Data: Collect human-generated text over various domains and use cases
</li>
<li dir="ltr" aria-level="1">
Comparison Data: Evaluate your model outputs with human feedback
</li>
<li dir="ltr" aria-level="1">
Trust &amp; Safety: Align your model to produce factually correct and safe content&nbsp;
</li>
</ol>
Demonstration Data&nbsp;
We have two subcategories of project types under the &ldquo;Demonstration Data&rdquo; bucket.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/09a3aa9a-f570-4ee7-93f8-42d2f47a9795" width="624" height="319">
(1) InstructGPT-style presets: Each of these comes with expert-written instructions, a preset taxonomy with the ability to add a customization section for your specific use case, the ability to upload your own prompts or have Scale create them for you, and a pre-trained workforce that has significant experience in these types of prompts. Check out Section 3.4 of the&nbsp;<a href="https://arxiv.org/pdf/2203.02155.pdf">InstructGPT paper</a> for yourself! 😉
(2) Demonstration corrections: If you have demonstrations or model outputs that need corrections, we can also assist you. Just upload your data or hook up your model endpoint and setup your taxonomy accordingly so that our labelers can get that done for you quickly.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/ef708a53-23de-49b1-ac2c-f0b744870c68">
Comparison Data&nbsp;
With our workflows, you can easily customize comparison annotation projects and quickly scale to production. You can ask a labeler to rank, rate on a Likert scale, and classify model outputs. You can either upload these model outputs or configure a model endpoint for labelers to interact with as they are making the comparisons.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/7eeab0fb-3f99-4b29-821e-5dcff793cfdb">
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/d27084b0-472c-417d-89e0-ea1364ef2e9f">
Trust &amp; Safety&nbsp;
Aligning your models with Rapid speed has never been easier. With our expertise and specialized workforce, we can work with you to identify problematic model output.&nbsp;
We support several use cases, including Grounded Generation, Red Teaming, Completion Citation, and Research-Based Correction.&nbsp;
<ul>
<li dir="ltr" aria-level="1">
Grounded Generation: Ask a labeler to research or read certain sources to verify uploaded model output.
</li>
<li dir="ltr" aria-level="1">
Red Teaming*: Provide an endpoint for labelers to chat with and elicit unsafe model outputs.&nbsp;
</li>
<li dir="ltr" aria-level="1">
Completion Citation*: Ask a labeler to attribute various phrases to sources that the information came from.&nbsp;
</li>
<li dir="ltr" aria-level="1">
Research-Based Correction*: Upload model output for labelers to find better sources for, edit/rewrite completions given any linked sources, add/edit/remove missing citations, etc.&nbsp;&nbsp;&nbsp;
</li>
</ul>
*Note: Use cases marked with * are currently not available via our self-serve platform. If you are interested, please contact sales to learn more.&nbsp;
For our Grounded Generation use case, we offer two self-serve, popular verification methods.
(1) Multi-class Classification: This involves selecting all error classes that pertain to the text via confidence-based consensus.&nbsp;
(2) Named Entity Recognition:&nbsp; This involves highlighting spans of erroneous text and indicating the relevant error types for each span.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/a35f53ff-4fae-48e1-b8f1-674fbcf3f752">
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/079d3f20-4345-4797-a4e3-31ba66781b0b"> And this is just the beginning! Our toolkit is constantly expanding, and we are excited to continuously be solving some of the world&rsquo;s hardest data annotation problems. Not seeing what you&rsquo;re looking for in the toolkit today or have other questions? Email us at: scale_rapid@scale.com.

Learn how to collect human-generated text, evaluate your model outputs with human feedback, and align your model to produce factually correct and safe content. 

Getting data to train and evaluate large language models (LLMs) has never been easier

Introducing: Scale’s Large Language Model Toolkit

<h3 dir="ltr">Introduction</h3>
OpenAI just released their new model that powers ChatGPT, GPT-3.5 Turbo. The best part &mdash; it's 10x cheaper than their previous best model.
OpenAI has found that GPT-3.5 Turbo is actually their best model for many non-chat use cases. Some early testers found it super easy to switch from the Davinci-003 model to GPT-3.5 Turbo with just a little bit of adjustment needed to their prompts. So, it's a much more cost-effective option for a lot of non-chat use cases.
Now that there is this new model available, it can be tough to know which one to use for your LLM apps in production. Spellbook makes it easy to build, compare, and deploy large language model apps, so you can evaluate GPT-3.5 Turbo against other models like Davinci-003 and FLAN to see which one performs best for your specific use case. In this blog post, we'll dive into the details of how these models differ on single-turn completions and what their strengths and weaknesses are, and share some real-world results from tests run on Spellbook- <a href="https://scale.com/spellbook">try it yourself today</a>!
<h3 dir="ltr">Classification&nbsp;</h3>
From our previous <a href="https://scale.com/blog/gpt-3-davinci-003-comparison">experiments</a>, we concluded that Davinci-003 performed much better than Davinci-002 with 0 shot prompts (92% vs 69.91%), and on par or slightly worse than Davinci-002 on few shot prompts (87.2% vs 91.6%). We observe a similar case with GPT-3.5 Turbo and Davinci-003.
GPT-3.5 Turbo performs better on 0 shot classification- on an insider trading classification example it achieves 91.72% accuracy, versus 82.02% for Davinci-003.
<figure class="image"><img src="https://lh4.googleusercontent.com/-qWbBYjvANJhwOpm25BihanYgq7goa1pw4QDINq3s6ZZedbwsHzuAOguit4S-2ug0TyVVx2DXgt7ep2J1MBiO_quaJrAWRlb4TRJJj-Te4zo5V3XHE58gNCbw_Rni-gFLx5gRxf0Aq0-bFM-aKUwN4c" width="624" height="349">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a 0-shot insider trading classification use case.</figcaption>
</figure>
However, Davinci-003 performs slightly better than GPT-3.5 Turbo with k-shot examples at 90.47% accuracy versus 88.37% for Davinci-003, but the quality is still lower than the GPT-3.5 Turbo 0 shot example.&nbsp;
<figure class="image"><img src="https://lh4.googleusercontent.com/iIefySacpNH98k4jEBgnUTSaRb5FCQiodLKt98KyvuI4-Z_iQp3lkH7bkQeKc-k-Sz2jnulr4wXT93pfU9visP46mOy0cfNXc3WVqr_b4M_meiDognTpgAL8vaHiB3cCW5eOCuhG_ZfEQPkSh6OWpYc" width="624" height="357">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on an insider trading classification use case with k-shot examples.</figcaption>
</figure>
The use of k-shot examples will likely not be very common with GPT-3.5 Turbo- including them in chat prompts can take up a significant amount of context, which is not ideal for a chatbot that receives a high volume of API requests. In cases where you&rsquo;re using GPT-3.5 Turbo for single-turn completion, this will be pretty much equivalent to adding k-shots for Davinci-003, but for multi-turn use cases, this can lead to inefficient resource usage if k-shots are included in every prompt.&nbsp;
On a more nuanced classification example, like sentiment analysis, we also see that GPT-3.5 Turbo outperforms Davinci-003 on 0 shot experiments (84.26% vs 78.57%).
<figure class="image"><img src="https://lh6.googleusercontent.com/ygoPuU_tuQoheEQ2N5Ut-7KG0dZjEap2zu6D-p4gijZ87fIfDz-zqnTY2oYaQIqWZhSMBrbrsxlGjZdFzogzJ1CZM5lgHmlsLnJkOpqSll81yjRYcioGG5BBK1xUraifOOiLojBPqj8t1HRZ7nBUwEk" width="624" height="360">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a 0-shot sentiment analysis use case.</figcaption>
</figure>
<h3 dir="ltr">Text Generation</h3>
If you're looking for an AI language model that can generate long, detailed answers to complex questions, GPT-3.5 Turbo might seem like the obvious choice since it's been trained on a massive dataset of human language and produces coherent, contextually-appropriate responses.
But what if you're looking for a model that can provide clear, concise responses? In that case, you might want to consider using Davinci-003 instead.
In our experiment, we provided both GPT-3.5 Turbo and Davinci-003 with a set of 30 questions, and asked each model to provide 30 answers. What we found was that GPT-3.5 Turbo&rsquo;s responses tended to be much longer than Davinci-003's, with many of them exceeding 100 words. In contrast, Davinci-003's responses were much more concise, often consisting of just a few sentences. Given a max tokens of 500, the average response from GPT-3.5 Turbo was 156 words (~208 tokens) nearly twice the amount of Davinci-003, whose responses were an average of 83 words (~111 tokens). 
<figure class="image"><img src="https://lh3.googleusercontent.com/XWVxJwRIeuE8s5I686NmO8WlR9DJPekjQ_TLSimXu8xm52eKrxOD0kJSyPcDqZj_mpvJ1aeA1soQ4DG8CYWsd3DqMnN4mE-SYj6Wqw7s5bu6XgdMYNQZjblaQcctic_hfJvUma8nqCEeitps7GeSucU" width="624" height="351">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a question answer use case.</figcaption>
</figure>
<h3 dir="ltr">Math&nbsp;</h3>
Notably, GPT-3.5 Turbo is significantly better than Davinci-003 at math. When evaluating by exact match with an answer key, GPT-3.5 Turbo gets the correct numerical answer 75% of the time, while Davinci-003 only gets 61% correct.
<figure class="image"><img src="https://lh6.googleusercontent.com/IG9V-yRgXR9vLTW0Myve1LV8Q8w1FpFPTs7t-YAaFXHAZQzRMUXZVvc64lb-2P6ERE79u_f_FY4eB1Y4dyK4kK59WtuEuMm-hNxQJ2C5m8J4ZuTxt1FE-C0ENR-iYK_8zXWId-SS7gy-qNIGeJ1ZGiY" width="624" height="313">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a math use case.</figcaption>
</figure>
We also selected a subset of the questions for expert human reviewers to evaluate the models correctness based on both the numerical correctness and the logical reasoning. Interestingly, upon asking both models to explain their answers, it became clear that the models would sometimes hallucinate reasoning for how they achieved the answer, even if it was numerically correct.&nbsp;
<figure class="image"><img src="https://lh4.googleusercontent.com/xmFdfVGuo3p81w5agMlTCOEuBLDkDdYHo6oniV8MYtIxc5IuqcFVti6vof3b3bb3dO-1QyOfyJ5ryVRleV6nSrPYpJc8w55vcRtnMY_KXhh7wB7HvNJ6ZB6bht9S8ijLJdgDviWDN6fJzXTrWCPQIJ8" width="624" height="315">
<figcaption>Evaluation configuration for human evaluation of math explanations.</figcaption>
</figure>
<figure class="image"><img src="https://lh4.googleusercontent.com/JRmEQBPW5OjYMxQWGH1ZyT7hVaeQ8dAUGx2Zy8qprMyCCwh1rrg1woFZzb1eSNS1sqkkgdI95cytMCljkoYVugA4tA3shO8SvP43hVQnOTW28Bkobru0sTHNqN6k1QXCjgwy9uSIcaPMIR4QxRdcBSg" width="624" height="229">
<figcaption>Quality drill down on human evaluations of math explanations from Davinci-003</figcaption>
</figure>
From the quality drill down summary from the completed tasks, Davinci-003 only got 44% of the questions correct. Repeating the same workflow for GPT-3.5 Turbo yielded 68% accuracy.&nbsp;
A couple interesting math lessons GPT-3.5 Turbo picked up that Davinci-003 struggled with:
<ul>
<li dir="ltr" aria-level="1">
Solving questions using the order of operations
</li>
</ul>
<figure class="image"><img src="https://lh5.googleusercontent.com/TLCynrXwIToi9u-tS5vFU-oBMcuKupJtA9foUG4ryegt3xwZCaGEhoPWmZpiteoUvTt8BzoRkVHqMY5L_xKOjwB2LLceaLxCNAHV0IYHJVLkVs9c7d8Cb6PnpVIMewge0AreLBYPPkBCRGqUL2MyMKg" width="624" height="276">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about order of operations.</figcaption>
</figure>
<ul>
<li dir="ltr" aria-level="1">
Plugging in variables into equations
</li>
</ul>
<figure class="image"><img src="https://lh3.googleusercontent.com/j63fVykVKFmDpt1qMIW3XPkntWiApDSvocH4oImJLdOr7quH0ESBTqDZwFSQpICgO1Idyja9iz_9UJn2cEDdOqB2z2Lv1UrRERGmb9qm9ojwBTuUwDhGBrvuZjsjeDpHYeFV2EM0sauHr3B_8shZNB0" width="624" height="147">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about plugging in variables into equations.</figcaption>
</figure>
<ul>
<li dir="ltr" aria-level="1">
The negative exponent rule
</li>
</ul>
<figure class="image"><img src="https://lh3.googleusercontent.com/sxzxXRaidO3EmcVfJE_fHvgUBwRfRkdpZ2RVhAtEO6R0tjjTdDLtueT0VtrPqDouqax6beTzb-d4m4Z3t8VwZaGopCOVj67sxPVCKzWKsloIgnx_PHftT6YsF0JEY2y3qfCn1bqp-vAILRwurbozbvc" width="624" height="212">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about negative exponents.</figcaption>
</figure>
<h3 dir="ltr">Base model swapping</h3>
It&rsquo;s worth noting that OpenAI has recommended the replacement of Davinci-003 implementations with the chat API in various instances, given its significantly lower cost of only $0.002 per 1,000 tokens. The ChatAPI can be used in a manner similar to the traditional Davinci-003 prompt format, where the system prompt is treated as the prompt body, the user message is treated as the input, and the assistant message serves as the completion.
For users who have LLM apps integrated in production using Spellbook, it&rsquo;s super easy to fork a variant, swap out the base model to GPT-3.5 Turbo and save it as a new one, and swap in the new variant, without having to write a single line of code.&nbsp;
<figure class="image"><img src="https://lh6.googleusercontent.com/Wp3bvT0IPiS0olJFhr-qiPieXViAuVBcq5q4ZAQOAcKJK8FFg2tOZp_nIsGcCdVZ0A8wdtdoAWj4CCoxMexDGD0IcUNyDPA-kFKvq0GuzHC3J_JWv4-T2oakkCKddvz13tS4OtmYzec0j-SXmLKOIws" width="624" height="325">
<figcaption>Quickly swap out your deployment&rsquo;s underlying variant to use GPT-3.5 Turbo without writing a single line of code.</figcaption>
</figure>
<h3 dir="ltr">Takeaways</h3>
OpenAI's new GPT-3.5 Turbo model offers a cost-effective option for many non-chat use cases. While GPT-3.5 Turbo performs well in 0-shot classification and math, Davinci-003 performs slightly better in k-shot classification and may be a better option for those looking for clear, concise responses that get straight to the point.
Spellbook makes it easy to build, compare, and deploy large language model apps, so you can evaluate GPT-3.5 Turbo against other models like Davinci-003 and FLAN to see which one performs best for your specific use case. With the ability to fork a variant and swap out the base model to GPT-3.5 Turbo, users can seamlessly integrate the new model into their LLM apps without writing any additional code.
Overall, GPT-3.5 Turbo performs better than Davinci-003 on most tasks at 10% of the cost - but is much more rambly. You can test out whether GPT-3.5 Turbo is better for your use case through <a href="https://scale.com/spellbook">Spellbook - sign up today</a>.

GPT-3.5 Turbo performs better than Davinci-003 on most tasks at 10% of the cost - but is much more rambly. You can test out whether GPT-3.5 Turbo is better for your use case through Spellbook

Compare the API behind ChatGPT to existing Davinci models using Spellbook

ChatGPT API: Cheaper, Faster, Wordier

Have you ever listened to the song, &ldquo;Umi Says&rdquo; by Mos Def? It&rsquo;s a fantastic song and in the chorus, Mos Def sings, &ldquo;My Umi said, shine your light on the world. Shine your light for the world to see.&rdquo; My mother would say the same thing to me when I was younger and she still says it to this day! But what once started as rhythmic words of encouragement, now has begun to transform into a deeper, more raw sense of self. It has taught me that we can brandish one&rsquo;s individuality by sharing experiences, growing from adversities, and passing lessons learned onto my peers, and, to me, that is what I believe Black History Month is truly about. I never saw it as a design to cast blame or seek apologies; but rather a time to appreciate both Black culture and Black contributions. It is a time to develop a profound understanding of our current generation and those that came before us. Because from listening, comes understanding.&nbsp;&nbsp;
My family educated me on what being a Black person in America was like for them and what it may be like for me. They taught me how to navigate the waters and pushed me to hear others&rsquo; experiences when given the chance. And then in 2020, the George Floyd movement sparked discussion across the country and a societal shift began taking place. Various groups began broadcasting their desire for institutional change. Police training, workplace diversity, legislation geared towards discrimination; every facet that could touch bias or mistreatment within the Black community was under a microscope. Echoing a similar sentiment, different organizations and initiatives approached the problem in the form of open conversation. One of which was The Commit to C.A.R.E (Care About Racial Equity) Now initiative that was launched as a response to the consistent injustices, stereotypes, and negative attitudes about Black men in America. The initiative partnered with Dove Men+Care and the NBPA (National Basketball Player&rsquo;s Association) to create a portfolio focusing on &ldquo;celebrating the contributions of Black men in communities across the country and telling their stories as a part of our commitment to change the way Black men are seen and treated in our society.&rdquo; This initiative brought to light the perspectives and stories from CEOs, comedians, NBA players, authors, and actors alike to capture them in a series of videos and photo shoots. I was fortunate enough to participate in this collaboration and got to share my own experience on what it is like to be a Black man in America. These conversations moved the needle forward in the larger conversation about racial inequity and emphasized the need for tangible benchmarks for measuring this change.&nbsp;
&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh5.googleusercontent.com/MC-c6NkZ9xHbl5KiZg1PIjgUigQuZ3nlGGzPuxy1SvnmZ8T_EZtqqwpS6cRDQIDXOe6RFzqm4jupcz2sx0DBYtTKDBM-fiymx9WAqC2StoQbeYGfZFGUmDBdiuH1YoqRq4jXkqSXPAUr" width="235" height="235">
<figcaption>Here I am posing for the The Commit to C.A.R.E Now portfolio in 2020. This initiative partnered with Dove Men+Care and the National Basketball Players Association.</figcaption>
</figure>
Participating in this program and listening to the other men who joined me in it, taught me what it means to persevere and implement change. I heard their struggles, their hopes, and their own thoughts of what could be done to improve our culture&rsquo;s place in this world. Their views reaffirmed my belief: that when there is representation in any environment, avenues for change become achievable. At a granular level, having a diverse group of leaders can create networking opportunities for all communities through continuous involvement and influence. What were once one time donations, turns into institutionalized funding. Higher quality education becomes accessible rather than the faraway dream it used to be. These are examples of how equipping Black communities with the right tools can snowball into something that my community continually strives for: generational wealth in the form of capital, knowledge, and opportunity. Access to these tools allows for all generations to stand a chance against cronyism and social stratification.&nbsp;
While learning lessons from our current generation is powerful, I feel that it&rsquo;s important to also spotlight and commemorate the feats that Black Americans have accomplished in the past. Despite the history of racism and oppression, the Black community has paved the way for people like me. This community has been fighting for equality and fair opportunities so that Black people today may experience life in a way that they could not. They have demonstrated constant efforts to leave the world a better place than when they entered it and these very efforts have percolated from leaders like Martin Luther King Jr. down into the communities he served. As a result, these communities place a great deal of importance on sharing the impact that these historical endeavors have had on their lives by passing their history down to the next generation. So when the National Museum of African American History and Culture opened in 2016, my grandparents jumped at the opportunity to donate an artifact to the collection. Donating this artifact meant the world to my grandmother in particular because behind it, lies the history of her father.&nbsp;
Percival Leon Burrows was born in Barbados, and later in life, would move to Buffalo, NY and then Harlem to join the Universal Negro Improvement Association and African Communities League (UNIA-ACL) alongside Marcus Garvey. He then received a Black Star Line certificate for his service and passed it down to my grandmother who donated this very certificate to the museum. I&rsquo;ll never forget how proud my grandmother stood while looking at a small piece of my family&rsquo;s Black History on display in the nation&rsquo;s capital.&nbsp;
&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/Wlk_aR5NJZju3VJC7bnlsxxBPRXA7J1_Yw-EEctsfLTYUu65St43VRqht19euAUmP62SOC5QQ7p9tmtLlIi6_NMRc9SkF_CHmfvoxuhGF6LD09XNQZZz_DC8fe3PA8KKPoc8wTfAbrdC" width="246" height="207">
<figcaption>UNIA-ACL promoted the purchase of stock in this shipping company to encourage commercial trading among African people worldwide.</figcaption>
</figure>
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/Ke6ICNeyeAbtQLjJgR1MPS4-YxNIMJ0r7df5V3ba_ww1p8CsITwsEo_AMvEgOWXY3MzRajudXX2UhpCFCEkopzkTlh-sWUsDr8BV9ihsK-9h5HuGBulhydnXxVu3N8lyte2pN4_lWvWC" width="188" height="252">
<figcaption>Leota &ldquo;Burrows&rdquo; &amp; Edgar Henry pose in front of the Voices of Protest and Resistance exhibit at the National Museum of African American History and Culture.</figcaption>
</figure>
I am thankful for everything the Black community has done for me and I am especially grateful to my family and colleagues who have shaped the way I approach my work and my life. For without these efforts, I would not be in the position I am today. I would not have had the tools to become a Carnegie Mellon graduate. I would not have had the skills to play college ball and try out for the NBA&rsquo;s official minor league basketball organization. I would not have had the connections to intern under a U.S Senator. And I most definitely would not be a Security Engineer at Scale AI. But the efforts made by Commit to C.A.R.E Now, UNIA-ACL, and my grandmother, exhibit the lengths our community goes to progress the state of all African Americans and demonstrates what can happen when our community has a chance to shine their own light on the world. I hope that I am able to pay it forward with half the capacity that my community has done for me.&nbsp;
And Black history doesn&rsquo;t just stop in February! If you live in D.C like me and you&rsquo;re interested in learning more about the African American saga, I recommend visiting the <a href="https://nmaahc.si.edu/">National Museum of African American History and Culture</a>. It&rsquo;s packed full with extraordinary exhibits and thousands of artifacts that have been donated by people all over the world. If you aren&rsquo;t in the area but want to learn more, check out the Commit to C.A.R.E Now video (14:25)&nbsp;I was featured in! Or, if you use Instagram, checkout @blackarchives.co or @brownhistory.&nbsp;
<a href="https://www.youtube.com/watch?v=I82ioHNSEss">Commit to C.A.R.E Now video </a>
But for now, &ldquo;Alexa, play Umi Says, by Mos Def.&rdquo;
&nbsp;

Scale Celebrates 2023 Black History Month

Seth Henry shares his experiences as a Black man to celebrate Black History Month

Visions of Color

Scale hosted its first ever generative AI hackathon at our San Francisco headquarters on January 21st! We welcomed over 300 developers, engineers, and hackers from across the country to come in and create something awesome.&nbsp;
From side projects to chatbots, from work projects to generative art pieces &mdash; hackers submitted over 80 projects. The Hackathon was also buzzing with workshops where attendees could learn about topics like Advanced Prompt Engineering with&nbsp;<a href="https://twitter.com/goodside">Riley Goodside</a>, who helped attendees incorporate the art of large language model prompting into their projects.
Besides just meeting people and hacking, hackers could walk away with cool prizes, like win Napa tours, skydiving, drones and more.
<h2 dir="ltr" style="text-align: center;">Winners Feature&nbsp;</h2>
<h4 dir="ltr">First Place: GPT is All You Need for the Backend</h4>
Taking first place was a project called &ldquo;GPT is all you need for the backend&rdquo;. This innovative project automates backend processes with a Large Language Model (LLM). It can infer business logic based on the name of the API call and maintain state in the form of a JSON (lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate) blob. With this new tech, developers are now able to create an entire backend and database with no manual logic programming. Following the instructions, they can define their application and proceed to jump straight into making API calls.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/ck-TTZe5zkMITFldY26SHLQaImb5xCNsw4kury6by8yGjLBOfiLI7ct8VW5hjrfdjxIlrUPpyEvJaDeG_ww_rtbN1XECH501cxwgth3DyGWLYGLE1aEWtyZWWoKRRAycJMkXEja4mwIJCWwxbU7f5Jg" width="340" height="372">
<h4 dir="ltr">Second Place: Chempilot</h4>
&ldquo;Chempilot&rdquo;, the AI-enabled SMARTS (SMiles ARbitrary Target Specification - a chemical language used to describe molecular structures and substructures) analysis project, earned second place at the hackathon. Chempilot makes it easy to search chemical databases and guide generative models, by allowing users to generate SMARTS queries from natural language descriptions of molecules. This groundbreaking project was built by leveraging GPT3 and <a href="https://scale.com/spellbook">Spellbook</a>, while combining that with open-source datasets and libraries to build the dataset. Chempilot is incredibly useful for drug discovery, making it faster and easier than ever to find and analyze the right molecule. It&rsquo;s a unique combination of AI, chemistry, and cheminformatics expertise, making it an invaluable tool for any lab.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/P6PJGzsX8v9SKF7TTTRuUofrsaeXFPrIc_wZMnlVOELEE4IO8TNUFeE_Dw6uZfFAISPOw7qyQWG3LjwTqbLzANWcTlmEubMIwTbK9a-jHXFTyQwVL7k0DSTT8ol0bc9xEkpyN3dwRklNzn2wJ06ij_s" width="338" height="442">
The team behind the 'Rap Battles' project made waves with their use of generative AI to create rap battles, and won third place, as well as the Audience Choice award. Through the use of GPT-3 for lyrics, wavenet for vocals, and stable diffusion for the avatars, users can input the names of any two people and generate a rap battle between them. The team even took it one step further and launched the website on the day of the hackathon, providing a live demo of George Washington vs. Andrej Karpathy!
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/9XsTs_8nut6Kyi8GeRz_YcxhGgHSKyzbOy7Iz7K4K62zB128aEFl12sfkXLVZnDrlMdgoNOchwk26-PRp1UkhHkiExItA3y7dU_RZzubBN2oX-Rhmj_RjE1Y0NHeuI8PLnO255d0DVhueKc7zZdex18" width="470" height="385">
The project &ldquo;Node Fixer&rdquo;, won the Best Use of Spellbook award, and is the perfect tool for automating corrective action for syntax and runtime errors in JavaScript programs. With a combination of a VSCode extension, a rust binary, and Spellbook, Node Fixer allows developers to quickly identify and repair errors that cause their program to crash. When an error is detected, Node Fixer sends the error and the code to a Spellbook app, which then analyzes and produces a fix. The fixed code is written back into the editor, ensuring the developer can keep their program running smoothly and quickly. With Node Fixer, there is no need to manually fix trivial errors in your code.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/TjZZYOgCKD_RECp4Giu5cuB7TPqLWKdG22HokGYb1ovBnsfIi_UaWJG4zVasm8Q07HqkDWHI0BMsVrtPyj6RNXxwlJeLPQprqy4OvMBlSyXq3cVDjoy8L4zOkPEy2gSCyQvmcEHRG4DpspRFSDFpOnU" width="690" height="422">
<h3 dir="ltr" style="text-align: center;">Advanced Prompt Engineering with Riley Goodside&nbsp;</h3>
Riley Goodside also hosted a workshop on Advanced Prompt Engineering. Prompt engineering is the magic behind top-notch language model responses. A well-written prompt is crucial for avoiding off-topic, inconsistent, or offensive output. It's a challenging task that requires a nuanced understanding of how to talk to the models- there are research conferences and papers written on the best prompting techniques. The workshop garnered massive interest, and hackers filled up the office to learn the most innovative prompting strategies!
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/nbSgB9Mk6zXljYm7Rv4Y1i326SZzI3MTwQZUVFFUY0hD4Ifn6hLim_XqYbjTua-Vhoy35iAxvc54kv1TI4bXsiifFhrIEsP_L67WXo7AbecvahqxzvvMtRomljV2PdlNU0RuC4aC4E6D9TAQyMAVWQM" width="624" height="468">
Our hackathon was more than just a platform for hacking on cutting-edge projects, it was a community building event for passionate builders. You could satisfy your sweet tooth with Smitten's liquid nitrogen ice cream, bond with some furry friends at the puppy cuddling station, bring your own personal style to life with live screen printing on hoodies and tees, and even fuel up with a shot of espresso from the coffee bar. With so much to do and so many amazing people to meet, the hackathon was a prime opportunity for hackers to dive into generative AI and meet legends in the field.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/lx3H3DoC3k97GX7gLEe0Vsywhp5XR1c_TfRhIFQmzQgC2sqiyM98SP06fptcbPU_-iN7yr1gizlfWJq_kfz9BEZx3XbZJcPnJqvqLKd4ejyvdCfUhJVs08DtIfs3p-9PCDCbAOq6r9MSLV2FlGEpQ3Q" width="624" height="416"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh5.googleusercontent.com/UNaO30_derXRDPyGA2uFOjtJKkx9sVEb0J443_9eCnpu0L8_jXGQQ2S95dWAipTQ6wR6jJwPQMzUqF5vg2SqEFw8NWXTdVG407KhzsMk1GyqpZgHk8pcKGsvNF2TR6_0rfbyAjJw4OMbnEXBQyiKsgA" width="624" height="416">
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/SNojvekmg8mkVfpiAQPN7xpcQY7m6RQ0PEMmHTjueXISSRbPxs6AJ1RA6dE0el_6il7xa_3zy28GpfsQm8MvUV-g5VDxpMsb0m9b6LpjaiJgL4D8cYBk6EtV6hN8bXbXg3TpR2qMaI7StG8sLXJFxv8" width="624" height="416">
P.S. some of our favorite tweets about the hackathon :)&nbsp;
&nbsp;
<img src="https://lh4.googleusercontent.com/BNVKBDmYBj9GqY9w8ouGl2VFyr3SrK9dGf3w4I7L4WqZLQfAqiTDY_V_XB4BsBQrtauIE2lhnQ_idvzMIBIEG3eLOn4CEyXOhFUvHs7TLUHWp_PO36mWXeHXIfz5n6nGGoAlmwvSSVUa563UtBhkPcE" width="293" height="370"><img src="https://lh6.googleusercontent.com/BNTQxrxaDMWA94V4U_x5TqIfxvuQ_Gi4ZoF2EQndeFJ6dmun4V6NCr0MzCLWx6WdqTF07ylssd8M3Plr_1z34EaIuewbFVPiqI_bYODLxf7lklcOiiw42smrx_ihG5vp77VRpHrrGaZRz1okFRMZIlg" width="283" height="370">
<img src="https://lh4.googleusercontent.com/0ZcXCryx9vrkw9oyYu6zjT-wHLjFqM06sI11qIfgVAT7goE4tuw7qJZ6bY6sMbhgQUOxaFOvAT9QpDfM-4A4cF2Oiy_NMVSgGC85mtNK_Csq0sNny9l_WXCeIsjGUgML4rtYPgZYSb7QaKHzD_cjq-k" width="308" height="370"><img src="https://lh3.googleusercontent.com/qFWKetCapC-7aM4QbvTcu6lYU3_IWfgeo0RGYURuUO_7TONrtWEwbN21rtRcEDpN6tQzrLPr6UDkbACRlU1iQfZPKHcSKjH60zBiifS8csL47ULDQmqgOCraalVWh53zuhikF_WpJ5sMUww1MSsDNKI" width="256" height="370">
<img src="https://lh4.googleusercontent.com/XN5UpVoL4ue6mXgJozDI_6zX_zmAGKaMcZJKZDyjzd3jZQbU8ESShOMm3QTrIHQ-x6R4Fpcshpo-iwe43pjejP-QtE6uyj6DAcKKjm1yRMRpNQLFVzoQYsckpFR9pAG40scdDib3YeV8fZljGCyiMcg" width="301" height="370"><img src="https://lh3.googleusercontent.com/JgCQ71IhQOuqldXpvwLDY39GK95sgJJ1dNoaiCt107BCYSGk6TBa6Yg-ulJxXCD61J0B-CFlMvFbMYlhd0u7FdOU0UkT-4mpcjL7PfogbAVg2ZMyal7oMNzW4mkNTbqDczLJF1wQYRelKzIeyX5LJFU" width="283" height="370"><img src="https://lh4.googleusercontent.com/xUuNd3LblslS9aYrV8r-UWRNmvwbS28b7sMi-wgKiP5lGVWpHKEumgjARwwS05k9LZlOl7MQ2pjFopyGhCEqlZZrwlpAUgKnjzOowlpZb6XQUHtJnjdZh0Lj1SeelrXpBylEya3ZN4WXsB_DqOjIkgw" width="292" height="266"><img src="https://lh3.googleusercontent.com/crVzJnUXiAndcc-IcQ5kCFhMFHtuSIKEpQDFSMYGJFl_11XRI2Paw2EyAnIT5zQ9bnGyWyB1cVypEW-VEpQdLy6XqAgtDMQbs2GeRU1YRiVf6d9X-gvmjeFO9kEuVLyr3Qg45VxrbgTvQjwfERhDTIs" width="299" height="266"><img src="https://lh3.googleusercontent.com/snJGc791L74WwKlDcAVtlj4mYw7XdA82nRWRViQLSDaNuzLNPp75sqJ6bBuCvX0JBEmPXmm5AT8lQtiyJeMBZ7TzDV5_Q4EF5OPak51gQUqZfHvlkfite3JIAggOrnmnNl2L8SFmUUyTe0sY8TYFL3c" width="292" height="249"><img src="https://lh4.googleusercontent.com/De-fqScHljBxUGhJJSoEVwZTHLO0qojNg0MfvKDPQH2Qe8Hzr7KB9o4D3Xr6-296aZOsQ_PhC2TO_6tSqLcdUsJRRCfVdIMHMiwk_l269wxsNttxpeC7eObiU0JUt782CzMLgiuZa1xgt24p8a-XjGM" width="297" height="249">
&nbsp;
&nbsp;
&nbsp;

The hackathon was more than just a platform for hacking on cutting-edge projects, it was a community building event for passionate builders.

Scale hosted it's first ever generative AI hackathon

Scale's Generative AI Hackathon

How Large Language Models are Trained and Tuned using Reinforcement Learning with Human Feedback (RLHF).

Why is ChatGPT so good?

In order to facilitate the automation and speed of maintaining surface to air missiles (SAM) systems chain of custody, Scale developed an AI powered system to track movements at these sites using EO satellite imagery. This blog post explores some of the implementation and novel concepts around the problem.
SAM systems are guided missiles that are typically launched from a mobile launch vehicle or a stationary launcher and used to defend against air attacks. Two common stationary SAM systems are the SA-2 and SA-3, often used in former Soviet satellite states.&nbsp;
From overhead imagery, these systems are typically identified by hexagonal patterns in the ground with launchers placed around the hexagon (with or without missiles), a radar placed near the center (typically accompanied by a communication vehicle), and missile transloader vehicles dispersed throughout the site.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/1efbb734-34e8-484f-8d41-f0858d34f69a" alt="" width="1200" height="720">
<h6 dir="ltr" style="text-align: right;"><code class="c-mrkdwn__code" data-stringify-type="code">Image Source: Skyfi</code>&nbsp;</h6>
A SAM system's perceived threat level is defined by the activity over time at a given site and the objects of interest at the site. For example, a site with ground vehicles and launchers present may be considered occupied but not operationally functional and therefore less of a threat. However, if a radar were to show up at that site it could then become operational, which is a potential cause for alert.&nbsp;
<h2 dir="ltr">Machine Learning Approach</h2>
<h3 dir="ltr">Narrowing the problem set</h3>
Tracking the status of SAM systems requires maintaining custody of 4 object classes: radars, ground vehicles, launchers, and missiles. However, there exists nuance about how these objects are detected: SA-2/SA-3 sites can be located in both dense urban and rural settings. Training an object detection model on a general region with a SAM system will lead to perceived annotation errors and false positives.&nbsp;
If, for example, there is a parking lot nearby (as there is in the image above) that does not actually contain vehicles relevant to the status of the SAM site, feeding this parking lot into the neural network without any annotations will likely confuse training and lead to declined model performance. Similarly, during inference time, the network is likely to output a number of false positive predictions around the vehicles in this parking lot when we are only interested in military vehicles relevant and directly within the SAM site.&nbsp;&nbsp;
To counteract this, Scale built a general SAM site object detector which places a focused bounding box around the site itself (see algorithm A in the figure below). This allows preprocessing for cropping out the specific site itself to feed into the final object detection network (see algorithm B in the figure below). Additionally, it provides a stand alone algorithm that can be used for broad area searches of new SA-2/SA-3 sites around the world.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/0c0b67e2-15fb-4f41-a1c0-c6b0fdb0d5f4" alt="" width="1200" height="430">
<h3 dir="ltr">Including Apriori Knowledge</h3>
Custody tracking in satellite images provides a unique training advantage over a static, stand alone object detection network that intakes a raw image and outputs detection predictions. In the case of custody tracking, there is almost always a prior timepoint of that Area of Interest that has been reviewed by an intel analyst and &ldquo;labeled&rdquo; with points. The work of the intel analyst can provide incredibly valuable information to the neural network learning to detect these objects on its own. To take advantage of this, instead of just feeding in an image of a SAM site to the network, we feed in both the image of the SAM site and a heatmap of the location of the objects of interest from the prior timepoint.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/46203af2-8fdd-4bbd-b10a-7a3c07c10497" alt="" width="1200" height="888">
The addition of these prior heatmaps improved algorithm detection performance across all 4 classes of interest. Object detection performance on these classes is measured using mean average precision at 50% IoU (mAP50).
<div dir="ltr" align="center">
<table><colgroup><col width="218"><col width="277"></colgroup>
<tbody>
<tr>
<td>
Class of interest
</td>
<td>
mAP50% improvement w/ Prior Heatmaps
</td>
</tr>
<tr>
<td>
SAM specific ground vehicles
</td>
<td>
4.2%
</td>
</tr>
<tr>
<td>
Launchers
</td>
<td>
17%
</td>
</tr>
<tr>
<td>
Radars
</td>
<td>
20.9%
</td>
</tr>
<tr>
<td>
Missiles
</td>
<td>
13%
</td>
</tr>
</tbody>
</table>
</div>
<h2 dir="ltr">Conclusion</h2>
The simultaneous transition to great power competition and explosion of imagery data creates an urgent need for imagery analysts to exploit imagery faster than ever before. High performance SAM detection algorithms allow analysts to receive automated disposition of forces insights at machine-speed, freeing analysts to spend their time on more complex analyses. Because the algorithm&rsquo;s outputs are structured data, analysts can further automate downstream processes such as setting up alerts for anomalous behavior or maintaining chain of custody.&nbsp;
At Scale, we believe technology should be used to empower (not replace) the analyst in crafting their narratives to deliver decision advantage.&nbsp;
NOTE:
This material is based upon work supported by the USAF AFRL/SBRK under Contract No. FA864921P0899. Any opinions, findings and conclusions or recommendations expressed in this material are those of&nbsp; the author(s) and do not necessarily reflect the views of the USAF AFRL/SBRK.&nbsp;
&nbsp;

High performance SAM detection algorithms allow analysts to receive automated disposition of forces insights at machine-speed

Maintaining (SAM) Systems

SAM Introduction

ChatGPT has captured LLM headlines and amazed the AI community with its extensive natural language processing capabilities. A new LLM from Anthropic called Claude is competitive with ChatGPT and offers great promise. We evaluate both models head to head and give you our thoughts on how they compare.

Claude vs. ChatGPT: Let’s Think Step by Step

Meet Claude: Anthropic’s Rival to ChatGPT

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

custom-fonts

homepage-event

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

eval

datastream

summit-2023-speakers

customers

slugs must be all lowercase. ie: ‘my-slug’

Display order on the page /customers. Higher values show first

summit-2023-agenda

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

collapsible-sections

research

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

leaderboards

Text for a short description like: Deprecated (as of January 2025)

Keyword for search. Add multiple keyword with comma. Example: hle, hleto, vista, enigma

Blog

Women’s History Month at Scale

Scale AI + Accenture: Enhancing Business Value with Human Ingenuity

Create And Deploy LLM Apps in Less Than 10 Minutes

Introducing: Scale’s Large Language Model Toolkit

ChatGPT API: Cheaper, Faster, Wordier

Visions of Color

Scale's Generative AI Hackathon

Why is ChatGPT so good?

SAM Introduction

Meet Claude: Anthropic’s Rival to ChatGPT

The future of your industry starts here

Blog

Women’s History Month at Scale

Scale AI + Accenture: Enhancing Business Value with Human Ingenuity

Create And Deploy LLM Apps in Less Than 10 Minutes

Introducing: Scale’s Large Language Model Toolkit

ChatGPT API: Cheaper, Faster, Wordier

Visions of Color

Scale's Generative AI Hackathon

Why is ChatGPT so good?

SAM Introduction

Meet Claude: Anthropic’s Rival to ChatGPT

The future of your industry starts here