experiment

Header

Footer

PreFooter

SectionTitle

PaginationButton

BlogPage

Trusted by world class companies, Scale delivers high quality training data for AI applications such as self-driving cars, mapping, AR/VR, robotics, and more.

AI & ML Blog | Scale AI

NextImage

AnnouncementContextProvider

Screen

scale.com (Production)

lottie-react

plasmic-nav

react-scroll-parallax-global

plasmic-embed-css

plasmic-query

plasmic-basic-components

plasmic-tabs

plasmic-cms

react-chartjs-2

plasmic-strapi

react-slick

radix-ui

react-quill

Product

Company

General

People

Engineering

<h3 dir="ltr">Introduction</h3>
OpenAI just released their new model that powers ChatGPT, GPT-3.5 Turbo. The best part &mdash; it's 10x cheaper than their previous best model.
OpenAI has found that GPT-3.5 Turbo is actually their best model for many non-chat use cases. Some early testers found it super easy to switch from the Davinci-003 model to GPT-3.5 Turbo with just a little bit of adjustment needed to their prompts. So, it's a much more cost-effective option for a lot of non-chat use cases.
Now that there is this new model available, it can be tough to know which one to use for your LLM apps in production. Spellbook makes it easy to build, compare, and deploy large language model apps, so you can evaluate GPT-3.5 Turbo against other models like Davinci-003 and FLAN to see which one performs best for your specific use case. In this blog post, we'll dive into the details of how these models differ on single-turn completions and what their strengths and weaknesses are, and share some real-world results from tests run on Spellbook- <a href="https://scale.com/spellbook">try it yourself today</a>!
<h3 dir="ltr">Classification&nbsp;</h3>
From our previous <a href="https://scale.com/blog/gpt-3-davinci-003-comparison">experiments</a>, we concluded that Davinci-003 performed much better than Davinci-002 with 0 shot prompts (92% vs 69.91%), and on par or slightly worse than Davinci-002 on few shot prompts (87.2% vs 91.6%). We observe a similar case with GPT-3.5 Turbo and Davinci-003.
GPT-3.5 Turbo performs better on 0 shot classification- on an insider trading classification example it achieves 91.72% accuracy, versus 82.02% for Davinci-003.
<figure class="image"><img src="https://lh4.googleusercontent.com/-qWbBYjvANJhwOpm25BihanYgq7goa1pw4QDINq3s6ZZedbwsHzuAOguit4S-2ug0TyVVx2DXgt7ep2J1MBiO_quaJrAWRlb4TRJJj-Te4zo5V3XHE58gNCbw_Rni-gFLx5gRxf0Aq0-bFM-aKUwN4c" width="624" height="349">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a 0-shot insider trading classification use case.</figcaption>
</figure>
However, Davinci-003 performs slightly better than GPT-3.5 Turbo with k-shot examples at 90.47% accuracy versus 88.37% for Davinci-003, but the quality is still lower than the GPT-3.5 Turbo 0 shot example.&nbsp;
<figure class="image"><img src="https://lh4.googleusercontent.com/iIefySacpNH98k4jEBgnUTSaRb5FCQiodLKt98KyvuI4-Z_iQp3lkH7bkQeKc-k-Sz2jnulr4wXT93pfU9visP46mOy0cfNXc3WVqr_b4M_meiDognTpgAL8vaHiB3cCW5eOCuhG_ZfEQPkSh6OWpYc" width="624" height="357">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on an insider trading classification use case with k-shot examples.</figcaption>
</figure>
The use of k-shot examples will likely not be very common with GPT-3.5 Turbo- including them in chat prompts can take up a significant amount of context, which is not ideal for a chatbot that receives a high volume of API requests. In cases where you&rsquo;re using GPT-3.5 Turbo for single-turn completion, this will be pretty much equivalent to adding k-shots for Davinci-003, but for multi-turn use cases, this can lead to inefficient resource usage if k-shots are included in every prompt.&nbsp;
On a more nuanced classification example, like sentiment analysis, we also see that GPT-3.5 Turbo outperforms Davinci-003 on 0 shot experiments (84.26% vs 78.57%).
<figure class="image"><img src="https://lh6.googleusercontent.com/ygoPuU_tuQoheEQ2N5Ut-7KG0dZjEap2zu6D-p4gijZ87fIfDz-zqnTY2oYaQIqWZhSMBrbrsxlGjZdFzogzJ1CZM5lgHmlsLnJkOpqSll81yjRYcioGG5BBK1xUraifOOiLojBPqj8t1HRZ7nBUwEk" width="624" height="360">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a 0-shot sentiment analysis use case.</figcaption>
</figure>
<h3 dir="ltr">Text Generation</h3>
If you're looking for an AI language model that can generate long, detailed answers to complex questions, GPT-3.5 Turbo might seem like the obvious choice since it's been trained on a massive dataset of human language and produces coherent, contextually-appropriate responses.
But what if you're looking for a model that can provide clear, concise responses? In that case, you might want to consider using Davinci-003 instead.
In our experiment, we provided both GPT-3.5 Turbo and Davinci-003 with a set of 30 questions, and asked each model to provide 30 answers. What we found was that GPT-3.5 Turbo&rsquo;s responses tended to be much longer than Davinci-003's, with many of them exceeding 100 words. In contrast, Davinci-003's responses were much more concise, often consisting of just a few sentences. Given a max tokens of 500, the average response from GPT-3.5 Turbo was 156 words (~208 tokens) nearly twice the amount of Davinci-003, whose responses were an average of 83 words (~111 tokens). 
<figure class="image"><img src="https://lh3.googleusercontent.com/XWVxJwRIeuE8s5I686NmO8WlR9DJPekjQ_TLSimXu8xm52eKrxOD0kJSyPcDqZj_mpvJ1aeA1soQ4DG8CYWsd3DqMnN4mE-SYj6Wqw7s5bu6XgdMYNQZjblaQcctic_hfJvUma8nqCEeitps7GeSucU" width="624" height="351">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a question answer use case.</figcaption>
</figure>
<h3 dir="ltr">Math&nbsp;</h3>
Notably, GPT-3.5 Turbo is significantly better than Davinci-003 at math. When evaluating by exact match with an answer key, GPT-3.5 Turbo gets the correct numerical answer 75% of the time, while Davinci-003 only gets 61% correct.
<figure class="image"><img src="https://lh6.googleusercontent.com/IG9V-yRgXR9vLTW0Myve1LV8Q8w1FpFPTs7t-YAaFXHAZQzRMUXZVvc64lb-2P6ERE79u_f_FY4eB1Y4dyK4kK59WtuEuMm-hNxQJ2C5m8J4ZuTxt1FE-C0ENR-iYK_8zXWId-SS7gy-qNIGeJ1ZGiY" width="624" height="313">
<figcaption>Comparison of GPT-3.5 Turbo against Davinci-003 on a math use case.</figcaption>
</figure>
We also selected a subset of the questions for expert human reviewers to evaluate the models correctness based on both the numerical correctness and the logical reasoning. Interestingly, upon asking both models to explain their answers, it became clear that the models would sometimes hallucinate reasoning for how they achieved the answer, even if it was numerically correct.&nbsp;
<figure class="image"><img src="https://lh4.googleusercontent.com/xmFdfVGuo3p81w5agMlTCOEuBLDkDdYHo6oniV8MYtIxc5IuqcFVti6vof3b3bb3dO-1QyOfyJ5ryVRleV6nSrPYpJc8w55vcRtnMY_KXhh7wB7HvNJ6ZB6bht9S8ijLJdgDviWDN6fJzXTrWCPQIJ8" width="624" height="315">
<figcaption>Evaluation configuration for human evaluation of math explanations.</figcaption>
</figure>
<figure class="image"><img src="https://lh4.googleusercontent.com/JRmEQBPW5OjYMxQWGH1ZyT7hVaeQ8dAUGx2Zy8qprMyCCwh1rrg1woFZzb1eSNS1sqkkgdI95cytMCljkoYVugA4tA3shO8SvP43hVQnOTW28Bkobru0sTHNqN6k1QXCjgwy9uSIcaPMIR4QxRdcBSg" width="624" height="229">
<figcaption>Quality drill down on human evaluations of math explanations from Davinci-003</figcaption>
</figure>
From the quality drill down summary from the completed tasks, Davinci-003 only got 44% of the questions correct. Repeating the same workflow for GPT-3.5 Turbo yielded 68% accuracy.&nbsp;
A couple interesting math lessons GPT-3.5 Turbo picked up that Davinci-003 struggled with:
<ul>
<li dir="ltr" aria-level="1">
Solving questions using the order of operations
</li>
</ul>
<figure class="image"><img src="https://lh5.googleusercontent.com/TLCynrXwIToi9u-tS5vFU-oBMcuKupJtA9foUG4ryegt3xwZCaGEhoPWmZpiteoUvTt8BzoRkVHqMY5L_xKOjwB2LLceaLxCNAHV0IYHJVLkVs9c7d8Cb6PnpVIMewge0AreLBYPPkBCRGqUL2MyMKg" width="624" height="276">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about order of operations.</figcaption>
</figure>
<ul>
<li dir="ltr" aria-level="1">
Plugging in variables into equations
</li>
</ul>
<figure class="image"><img src="https://lh3.googleusercontent.com/j63fVykVKFmDpt1qMIW3XPkntWiApDSvocH4oImJLdOr7quH0ESBTqDZwFSQpICgO1Idyja9iz_9UJn2cEDdOqB2z2Lv1UrRERGmb9qm9ojwBTuUwDhGBrvuZjsjeDpHYeFV2EM0sauHr3B_8shZNB0" width="624" height="147">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about plugging in variables into equations.</figcaption>
</figure>
<ul>
<li dir="ltr" aria-level="1">
The negative exponent rule
</li>
</ul>
<figure class="image"><img src="https://lh3.googleusercontent.com/sxzxXRaidO3EmcVfJE_fHvgUBwRfRkdpZ2RVhAtEO6R0tjjTdDLtueT0VtrPqDouqax6beTzb-d4m4Z3t8VwZaGopCOVj67sxPVCKzWKsloIgnx_PHftT6YsF0JEY2y3qfCn1bqp-vAILRwurbozbvc" width="624" height="212">
<figcaption>Comparison of GPT-3.5 Turbo (left) against Davinci-003 (right) on a math question about negative exponents.</figcaption>
</figure>
<h3 dir="ltr">Base model swapping</h3>
It&rsquo;s worth noting that OpenAI has recommended the replacement of Davinci-003 implementations with the chat API in various instances, given its significantly lower cost of only $0.002 per 1,000 tokens. The ChatAPI can be used in a manner similar to the traditional Davinci-003 prompt format, where the system prompt is treated as the prompt body, the user message is treated as the input, and the assistant message serves as the completion.
For users who have LLM apps integrated in production using Spellbook, it&rsquo;s super easy to fork a variant, swap out the base model to GPT-3.5 Turbo and save it as a new one, and swap in the new variant, without having to write a single line of code.&nbsp;
<figure class="image"><img src="https://lh6.googleusercontent.com/Wp3bvT0IPiS0olJFhr-qiPieXViAuVBcq5q4ZAQOAcKJK8FFg2tOZp_nIsGcCdVZ0A8wdtdoAWj4CCoxMexDGD0IcUNyDPA-kFKvq0GuzHC3J_JWv4-T2oakkCKddvz13tS4OtmYzec0j-SXmLKOIws" width="624" height="325">
<figcaption>Quickly swap out your deployment&rsquo;s underlying variant to use GPT-3.5 Turbo without writing a single line of code.</figcaption>
</figure>
<h3 dir="ltr">Takeaways</h3>
OpenAI's new GPT-3.5 Turbo model offers a cost-effective option for many non-chat use cases. While GPT-3.5 Turbo performs well in 0-shot classification and math, Davinci-003 performs slightly better in k-shot classification and may be a better option for those looking for clear, concise responses that get straight to the point.
Spellbook makes it easy to build, compare, and deploy large language model apps, so you can evaluate GPT-3.5 Turbo against other models like Davinci-003 and FLAN to see which one performs best for your specific use case. With the ability to fork a variant and swap out the base model to GPT-3.5 Turbo, users can seamlessly integrate the new model into their LLM apps without writing any additional code.
Overall, GPT-3.5 Turbo performs better than Davinci-003 on most tasks at 10% of the cost - but is much more rambly. You can test out whether GPT-3.5 Turbo is better for your use case through <a href="https://scale.com/spellbook">Spellbook - sign up today</a>.

image.jpg

GPT-3.5 Turbo performs better than Davinci-003 on most tasks at 10% of the cost - but is much more rambly. You can test out whether GPT-3.5 Turbo is better for your use case through Spellbook

Compare the API behind ChatGPT to existing Davinci models using Spellbook

ChatGPT API: Cheaper, Faster, Wordier

Have you ever listened to the song, &ldquo;Umi Says&rdquo; by Mos Def? It&rsquo;s a fantastic song and in the chorus, Mos Def sings, &ldquo;My Umi said, shine your light on the world. Shine your light for the world to see.&rdquo; My mother would say the same thing to me when I was younger and she still says it to this day! But what once started as rhythmic words of encouragement, now has begun to transform into a deeper, more raw sense of self. It has taught me that we can brandish one&rsquo;s individuality by sharing experiences, growing from adversities, and passing lessons learned onto my peers, and, to me, that is what I believe Black History Month is truly about. I never saw it as a design to cast blame or seek apologies; but rather a time to appreciate both Black culture and Black contributions. It is a time to develop a profound understanding of our current generation and those that came before us. Because from listening, comes understanding.&nbsp;&nbsp;
My family educated me on what being a Black person in America was like for them and what it may be like for me. They taught me how to navigate the waters and pushed me to hear others&rsquo; experiences when given the chance. And then in 2020, the George Floyd movement sparked discussion across the country and a societal shift began taking place. Various groups began broadcasting their desire for institutional change. Police training, workplace diversity, legislation geared towards discrimination; every facet that could touch bias or mistreatment within the Black community was under a microscope. Echoing a similar sentiment, different organizations and initiatives approached the problem in the form of open conversation. One of which was The Commit to C.A.R.E (Care About Racial Equity) Now initiative that was launched as a response to the consistent injustices, stereotypes, and negative attitudes about Black men in America. The initiative partnered with Dove Men+Care and the NBPA (National Basketball Player&rsquo;s Association) to create a portfolio focusing on &ldquo;celebrating the contributions of Black men in communities across the country and telling their stories as a part of our commitment to change the way Black men are seen and treated in our society.&rdquo; This initiative brought to light the perspectives and stories from CEOs, comedians, NBA players, authors, and actors alike to capture them in a series of videos and photo shoots. I was fortunate enough to participate in this collaboration and got to share my own experience on what it is like to be a Black man in America. These conversations moved the needle forward in the larger conversation about racial inequity and emphasized the need for tangible benchmarks for measuring this change.&nbsp;
&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh5.googleusercontent.com/MC-c6NkZ9xHbl5KiZg1PIjgUigQuZ3nlGGzPuxy1SvnmZ8T_EZtqqwpS6cRDQIDXOe6RFzqm4jupcz2sx0DBYtTKDBM-fiymx9WAqC2StoQbeYGfZFGUmDBdiuH1YoqRq4jXkqSXPAUr" width="235" height="235">
<figcaption>Here I am posing for the The Commit to C.A.R.E Now portfolio in 2020. This initiative partnered with Dove Men+Care and the National Basketball Players Association.</figcaption>
</figure>
Participating in this program and listening to the other men who joined me in it, taught me what it means to persevere and implement change. I heard their struggles, their hopes, and their own thoughts of what could be done to improve our culture&rsquo;s place in this world. Their views reaffirmed my belief: that when there is representation in any environment, avenues for change become achievable. At a granular level, having a diverse group of leaders can create networking opportunities for all communities through continuous involvement and influence. What were once one time donations, turns into institutionalized funding. Higher quality education becomes accessible rather than the faraway dream it used to be. These are examples of how equipping Black communities with the right tools can snowball into something that my community continually strives for: generational wealth in the form of capital, knowledge, and opportunity. Access to these tools allows for all generations to stand a chance against cronyism and social stratification.&nbsp;
While learning lessons from our current generation is powerful, I feel that it&rsquo;s important to also spotlight and commemorate the feats that Black Americans have accomplished in the past. Despite the history of racism and oppression, the Black community has paved the way for people like me. This community has been fighting for equality and fair opportunities so that Black people today may experience life in a way that they could not. They have demonstrated constant efforts to leave the world a better place than when they entered it and these very efforts have percolated from leaders like Martin Luther King Jr. down into the communities he served. As a result, these communities place a great deal of importance on sharing the impact that these historical endeavors have had on their lives by passing their history down to the next generation. So when the National Museum of African American History and Culture opened in 2016, my grandparents jumped at the opportunity to donate an artifact to the collection. Donating this artifact meant the world to my grandmother in particular because behind it, lies the history of her father.&nbsp;
Percival Leon Burrows was born in Barbados, and later in life, would move to Buffalo, NY and then Harlem to join the Universal Negro Improvement Association and African Communities League (UNIA-ACL) alongside Marcus Garvey. He then received a Black Star Line certificate for his service and passed it down to my grandmother who donated this very certificate to the museum. I&rsquo;ll never forget how proud my grandmother stood while looking at a small piece of my family&rsquo;s Black History on display in the nation&rsquo;s capital.&nbsp;
&nbsp;
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/Wlk_aR5NJZju3VJC7bnlsxxBPRXA7J1_Yw-EEctsfLTYUu65St43VRqht19euAUmP62SOC5QQ7p9tmtLlIi6_NMRc9SkF_CHmfvoxuhGF6LD09XNQZZz_DC8fe3PA8KKPoc8wTfAbrdC" width="246" height="207">
<figcaption>UNIA-ACL promoted the purchase of stock in this shipping company to encourage commercial trading among African people worldwide.</figcaption>
</figure>
<figure class="image"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/Ke6ICNeyeAbtQLjJgR1MPS4-YxNIMJ0r7df5V3ba_ww1p8CsITwsEo_AMvEgOWXY3MzRajudXX2UhpCFCEkopzkTlh-sWUsDr8BV9ihsK-9h5HuGBulhydnXxVu3N8lyte2pN4_lWvWC" width="188" height="252">
<figcaption>Leota &ldquo;Burrows&rdquo; &amp; Edgar Henry pose in front of the Voices of Protest and Resistance exhibit at the National Museum of African American History and Culture.</figcaption>
</figure>
I am thankful for everything the Black community has done for me and I am especially grateful to my family and colleagues who have shaped the way I approach my work and my life. For without these efforts, I would not be in the position I am today. I would not have had the tools to become a Carnegie Mellon graduate. I would not have had the skills to play college ball and try out for the NBA&rsquo;s official minor league basketball organization. I would not have had the connections to intern under a U.S Senator. And I most definitely would not be a Security Engineer at Scale AI. But the efforts made by Commit to C.A.R.E Now, UNIA-ACL, and my grandmother, exhibit the lengths our community goes to progress the state of all African Americans and demonstrates what can happen when our community has a chance to shine their own light on the world. I hope that I am able to pay it forward with half the capacity that my community has done for me.&nbsp;
And Black history doesn&rsquo;t just stop in February! If you live in D.C like me and you&rsquo;re interested in learning more about the African American saga, I recommend visiting the <a href="https://nmaahc.si.edu/">National Museum of African American History and Culture</a>. It&rsquo;s packed full with extraordinary exhibits and thousands of artifacts that have been donated by people all over the world. If you aren&rsquo;t in the area but want to learn more, check out the Commit to C.A.R.E Now video (14:25)&nbsp;I was featured in! Or, if you use Instagram, checkout @blackarchives.co or @brownhistory.&nbsp;
<a href="https://www.youtube.com/watch?v=I82ioHNSEss">Commit to C.A.R.E Now video </a>
But for now, &ldquo;Alexa, play Umi Says, by Mos Def.&rdquo;
&nbsp;

Scale Celebrates 2023 Black History Month

Seth Henry shares his experiences as a Black man to celebrate Black History Month

Visions of Color

Scale hosted its first ever generative AI hackathon at our San Francisco headquarters on January 21st! We welcomed over 300 developers, engineers, and hackers from across the country to come in and create something awesome.&nbsp;
From side projects to chatbots, from work projects to generative art pieces &mdash; hackers submitted over 80 projects. The Hackathon was also buzzing with workshops where attendees could learn about topics like Advanced Prompt Engineering with&nbsp;<a href="https://twitter.com/goodside">Riley Goodside</a>, who helped attendees incorporate the art of large language model prompting into their projects.
Besides just meeting people and hacking, hackers could walk away with cool prizes, like win Napa tours, skydiving, drones and more.
<h2 dir="ltr" style="text-align: center;">Winners Feature&nbsp;</h2>
<h4 dir="ltr">First Place: GPT is All You Need for the Backend</h4>
Taking first place was a project called &ldquo;GPT is all you need for the backend&rdquo;. This innovative project automates backend processes with a Large Language Model (LLM). It can infer business logic based on the name of the API call and maintain state in the form of a JSON (lightweight data-interchange format that is easy for humans to read and write and easy for machines to parse and generate) blob. With this new tech, developers are now able to create an entire backend and database with no manual logic programming. Following the instructions, they can define their application and proceed to jump straight into making API calls.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/ck-TTZe5zkMITFldY26SHLQaImb5xCNsw4kury6by8yGjLBOfiLI7ct8VW5hjrfdjxIlrUPpyEvJaDeG_ww_rtbN1XECH501cxwgth3DyGWLYGLE1aEWtyZWWoKRRAycJMkXEja4mwIJCWwxbU7f5Jg" width="340" height="372">
<h4 dir="ltr">Second Place: Chempilot</h4>
&ldquo;Chempilot&rdquo;, the AI-enabled SMARTS (SMiles ARbitrary Target Specification - a chemical language used to describe molecular structures and substructures) analysis project, earned second place at the hackathon. Chempilot makes it easy to search chemical databases and guide generative models, by allowing users to generate SMARTS queries from natural language descriptions of molecules. This groundbreaking project was built by leveraging GPT3 and <a href="https://scale.com/spellbook">Spellbook</a>, while combining that with open-source datasets and libraries to build the dataset. Chempilot is incredibly useful for drug discovery, making it faster and easier than ever to find and analyze the right molecule. It&rsquo;s a unique combination of AI, chemistry, and cheminformatics expertise, making it an invaluable tool for any lab.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/P6PJGzsX8v9SKF7TTTRuUofrsaeXFPrIc_wZMnlVOELEE4IO8TNUFeE_Dw6uZfFAISPOw7qyQWG3LjwTqbLzANWcTlmEubMIwTbK9a-jHXFTyQwVL7k0DSTT8ol0bc9xEkpyN3dwRklNzn2wJ06ij_s" width="338" height="442">
The team behind the 'Rap Battles' project made waves with their use of generative AI to create rap battles, and won third place, as well as the Audience Choice award. Through the use of GPT-3 for lyrics, wavenet for vocals, and stable diffusion for the avatars, users can input the names of any two people and generate a rap battle between them. The team even took it one step further and launched the website on the day of the hackathon, providing a live demo of George Washington vs. Andrej Karpathy!
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/9XsTs_8nut6Kyi8GeRz_YcxhGgHSKyzbOy7Iz7K4K62zB128aEFl12sfkXLVZnDrlMdgoNOchwk26-PRp1UkhHkiExItA3y7dU_RZzubBN2oX-Rhmj_RjE1Y0NHeuI8PLnO255d0DVhueKc7zZdex18" width="470" height="385">
The project &ldquo;Node Fixer&rdquo;, won the Best Use of Spellbook award, and is the perfect tool for automating corrective action for syntax and runtime errors in JavaScript programs. With a combination of a VSCode extension, a rust binary, and Spellbook, Node Fixer allows developers to quickly identify and repair errors that cause their program to crash. When an error is detected, Node Fixer sends the error and the code to a Spellbook app, which then analyzes and produces a fix. The fixed code is written back into the editor, ensuring the developer can keep their program running smoothly and quickly. With Node Fixer, there is no need to manually fix trivial errors in your code.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/TjZZYOgCKD_RECp4Giu5cuB7TPqLWKdG22HokGYb1ovBnsfIi_UaWJG4zVasm8Q07HqkDWHI0BMsVrtPyj6RNXxwlJeLPQprqy4OvMBlSyXq3cVDjoy8L4zOkPEy2gSCyQvmcEHRG4DpspRFSDFpOnU" width="690" height="422">
<h3 dir="ltr" style="text-align: center;">Advanced Prompt Engineering with Riley Goodside&nbsp;</h3>
Riley Goodside also hosted a workshop on Advanced Prompt Engineering. Prompt engineering is the magic behind top-notch language model responses. A well-written prompt is crucial for avoiding off-topic, inconsistent, or offensive output. It's a challenging task that requires a nuanced understanding of how to talk to the models- there are research conferences and papers written on the best prompting techniques. The workshop garnered massive interest, and hackers filled up the office to learn the most innovative prompting strategies!
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh3.googleusercontent.com/nbSgB9Mk6zXljYm7Rv4Y1i326SZzI3MTwQZUVFFUY0hD4Ifn6hLim_XqYbjTua-Vhoy35iAxvc54kv1TI4bXsiifFhrIEsP_L67WXo7AbecvahqxzvvMtRomljV2PdlNU0RuC4aC4E6D9TAQyMAVWQM" width="624" height="468">
Our hackathon was more than just a platform for hacking on cutting-edge projects, it was a community building event for passionate builders. You could satisfy your sweet tooth with Smitten's liquid nitrogen ice cream, bond with some furry friends at the puppy cuddling station, bring your own personal style to life with live screen printing on hoodies and tees, and even fuel up with a shot of espresso from the coffee bar. With so much to do and so many amazing people to meet, the hackathon was a prime opportunity for hackers to dive into generative AI and meet legends in the field.
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh6.googleusercontent.com/lx3H3DoC3k97GX7gLEe0Vsywhp5XR1c_TfRhIFQmzQgC2sqiyM98SP06fptcbPU_-iN7yr1gizlfWJq_kfz9BEZx3XbZJcPnJqvqLKd4ejyvdCfUhJVs08DtIfs3p-9PCDCbAOq6r9MSLV2FlGEpQ3Q" width="624" height="416"><img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh5.googleusercontent.com/UNaO30_derXRDPyGA2uFOjtJKkx9sVEb0J443_9eCnpu0L8_jXGQQ2S95dWAipTQ6wR6jJwPQMzUqF5vg2SqEFw8NWXTdVG407KhzsMk1GyqpZgHk8pcKGsvNF2TR6_0rfbyAjJw4OMbnEXBQyiKsgA" width="624" height="416">
<img style="display: block; margin-left: auto; margin-right: auto;" src="https://lh4.googleusercontent.com/SNojvekmg8mkVfpiAQPN7xpcQY7m6RQ0PEMmHTjueXISSRbPxs6AJ1RA6dE0el_6il7xa_3zy28GpfsQm8MvUV-g5VDxpMsb0m9b6LpjaiJgL4D8cYBk6EtV6hN8bXbXg3TpR2qMaI7StG8sLXJFxv8" width="624" height="416">
P.S. some of our favorite tweets about the hackathon :)&nbsp;
&nbsp;
<img src="https://lh4.googleusercontent.com/BNVKBDmYBj9GqY9w8ouGl2VFyr3SrK9dGf3w4I7L4WqZLQfAqiTDY_V_XB4BsBQrtauIE2lhnQ_idvzMIBIEG3eLOn4CEyXOhFUvHs7TLUHWp_PO36mWXeHXIfz5n6nGGoAlmwvSSVUa563UtBhkPcE" width="293" height="370"><img src="https://lh6.googleusercontent.com/BNTQxrxaDMWA94V4U_x5TqIfxvuQ_Gi4ZoF2EQndeFJ6dmun4V6NCr0MzCLWx6WdqTF07ylssd8M3Plr_1z34EaIuewbFVPiqI_bYODLxf7lklcOiiw42smrx_ihG5vp77VRpHrrGaZRz1okFRMZIlg" width="283" height="370">
<img src="https://lh4.googleusercontent.com/0ZcXCryx9vrkw9oyYu6zjT-wHLjFqM06sI11qIfgVAT7goE4tuw7qJZ6bY6sMbhgQUOxaFOvAT9QpDfM-4A4cF2Oiy_NMVSgGC85mtNK_Csq0sNny9l_WXCeIsjGUgML4rtYPgZYSb7QaKHzD_cjq-k" width="308" height="370"><img src="https://lh3.googleusercontent.com/qFWKetCapC-7aM4QbvTcu6lYU3_IWfgeo0RGYURuUO_7TONrtWEwbN21rtRcEDpN6tQzrLPr6UDkbACRlU1iQfZPKHcSKjH60zBiifS8csL47ULDQmqgOCraalVWh53zuhikF_WpJ5sMUww1MSsDNKI" width="256" height="370">
<img src="https://lh4.googleusercontent.com/XN5UpVoL4ue6mXgJozDI_6zX_zmAGKaMcZJKZDyjzd3jZQbU8ESShOMm3QTrIHQ-x6R4Fpcshpo-iwe43pjejP-QtE6uyj6DAcKKjm1yRMRpNQLFVzoQYsckpFR9pAG40scdDib3YeV8fZljGCyiMcg" width="301" height="370"><img src="https://lh3.googleusercontent.com/JgCQ71IhQOuqldXpvwLDY39GK95sgJJ1dNoaiCt107BCYSGk6TBa6Yg-ulJxXCD61J0B-CFlMvFbMYlhd0u7FdOU0UkT-4mpcjL7PfogbAVg2ZMyal7oMNzW4mkNTbqDczLJF1wQYRelKzIeyX5LJFU" width="283" height="370"><img src="https://lh4.googleusercontent.com/xUuNd3LblslS9aYrV8r-UWRNmvwbS28b7sMi-wgKiP5lGVWpHKEumgjARwwS05k9LZlOl7MQ2pjFopyGhCEqlZZrwlpAUgKnjzOowlpZb6XQUHtJnjdZh0Lj1SeelrXpBylEya3ZN4WXsB_DqOjIkgw" width="292" height="266"><img src="https://lh3.googleusercontent.com/crVzJnUXiAndcc-IcQ5kCFhMFHtuSIKEpQDFSMYGJFl_11XRI2Paw2EyAnIT5zQ9bnGyWyB1cVypEW-VEpQdLy6XqAgtDMQbs2GeRU1YRiVf6d9X-gvmjeFO9kEuVLyr3Qg45VxrbgTvQjwfERhDTIs" width="299" height="266"><img src="https://lh3.googleusercontent.com/snJGc791L74WwKlDcAVtlj4mYw7XdA82nRWRViQLSDaNuzLNPp75sqJ6bBuCvX0JBEmPXmm5AT8lQtiyJeMBZ7TzDV5_Q4EF5OPak51gQUqZfHvlkfite3JIAggOrnmnNl2L8SFmUUyTe0sY8TYFL3c" width="292" height="249"><img src="https://lh4.googleusercontent.com/De-fqScHljBxUGhJJSoEVwZTHLO0qojNg0MfvKDPQH2Qe8Hzr7KB9o4D3Xr6-296aZOsQ_PhC2TO_6tSqLcdUsJRRCfVdIMHMiwk_l269wxsNttxpeC7eObiU0JUt782CzMLgiuZa1xgt24p8a-XjGM" width="297" height="249">
&nbsp;
&nbsp;
&nbsp;

The hackathon was more than just a platform for hacking on cutting-edge projects, it was a community building event for passionate builders.

Scale hosted it's first ever generative AI hackathon

Scale's Generative AI Hackathon

How Large Language Models are Trained and Tuned using Reinforcement Learning with Human Feedback (RLHF).

Why is ChatGPT so good?

In order to facilitate the automation and speed of maintaining surface to air missiles (SAM) systems chain of custody, Scale developed an AI powered system to track movements at these sites using EO satellite imagery. This blog post explores some of the implementation and novel concepts around the problem.
SAM systems are guided missiles that are typically launched from a mobile launch vehicle or a stationary launcher and used to defend against air attacks. Two common stationary SAM systems are the SA-2 and SA-3, often used in former Soviet satellite states.&nbsp;
From overhead imagery, these systems are typically identified by hexagonal patterns in the ground with launchers placed around the hexagon (with or without missiles), a radar placed near the center (typically accompanied by a communication vehicle), and missile transloader vehicles dispersed throughout the site.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/1efbb734-34e8-484f-8d41-f0858d34f69a" alt="" width="1200" height="720">
<h6 dir="ltr" style="text-align: right;"><code class="c-mrkdwn__code" data-stringify-type="code">Image Source: Skyfi</code>&nbsp;</h6>
A SAM system's perceived threat level is defined by the activity over time at a given site and the objects of interest at the site. For example, a site with ground vehicles and launchers present may be considered occupied but not operationally functional and therefore less of a threat. However, if a radar were to show up at that site it could then become operational, which is a potential cause for alert.&nbsp;
<h2 dir="ltr">Machine Learning Approach</h2>
<h3 dir="ltr">Narrowing the problem set</h3>
Tracking the status of SAM systems requires maintaining custody of 4 object classes: radars, ground vehicles, launchers, and missiles. However, there exists nuance about how these objects are detected: SA-2/SA-3 sites can be located in both dense urban and rural settings. Training an object detection model on a general region with a SAM system will lead to perceived annotation errors and false positives.&nbsp;
If, for example, there is a parking lot nearby (as there is in the image above) that does not actually contain vehicles relevant to the status of the SAM site, feeding this parking lot into the neural network without any annotations will likely confuse training and lead to declined model performance. Similarly, during inference time, the network is likely to output a number of false positive predictions around the vehicles in this parking lot when we are only interested in military vehicles relevant and directly within the SAM site.&nbsp;&nbsp;
To counteract this, Scale built a general SAM site object detector which places a focused bounding box around the site itself (see algorithm A in the figure below). This allows preprocessing for cropping out the specific site itself to feed into the final object detection network (see algorithm B in the figure below). Additionally, it provides a stand alone algorithm that can be used for broad area searches of new SA-2/SA-3 sites around the world.
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/0c0b67e2-15fb-4f41-a1c0-c6b0fdb0d5f4" alt="" width="1200" height="430">
<h3 dir="ltr">Including Apriori Knowledge</h3>
Custody tracking in satellite images provides a unique training advantage over a static, stand alone object detection network that intakes a raw image and outputs detection predictions. In the case of custody tracking, there is almost always a prior timepoint of that Area of Interest that has been reviewed by an intel analyst and &ldquo;labeled&rdquo; with points. The work of the intel analyst can provide incredibly valuable information to the neural network learning to detect these objects on its own. To take advantage of this, instead of just feeding in an image of a SAM site to the network, we feed in both the image of the SAM site and a heatmap of the location of the objects of interest from the prior timepoint.&nbsp;
<img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/46203af2-8fdd-4bbd-b10a-7a3c07c10497" alt="" width="1200" height="888">
The addition of these prior heatmaps improved algorithm detection performance across all 4 classes of interest. Object detection performance on these classes is measured using mean average precision at 50% IoU (mAP50).
<div dir="ltr" align="center">
<table><colgroup><col width="218"><col width="277"></colgroup>
<tbody>
<tr>
<td>
Class of interest
</td>
<td>
mAP50% improvement w/ Prior Heatmaps
</td>
</tr>
<tr>
<td>
SAM specific ground vehicles
</td>
<td>
4.2%
</td>
</tr>
<tr>
<td>
Launchers
</td>
<td>
17%
</td>
</tr>
<tr>
<td>
Radars
</td>
<td>
20.9%
</td>
</tr>
<tr>
<td>
Missiles
</td>
<td>
13%
</td>
</tr>
</tbody>
</table>
</div>
<h2 dir="ltr">Conclusion</h2>
The simultaneous transition to great power competition and explosion of imagery data creates an urgent need for imagery analysts to exploit imagery faster than ever before. High performance SAM detection algorithms allow analysts to receive automated disposition of forces insights at machine-speed, freeing analysts to spend their time on more complex analyses. Because the algorithm&rsquo;s outputs are structured data, analysts can further automate downstream processes such as setting up alerts for anomalous behavior or maintaining chain of custody.&nbsp;
At Scale, we believe technology should be used to empower (not replace) the analyst in crafting their narratives to deliver decision advantage.&nbsp;
NOTE:
This material is based upon work supported by the USAF AFRL/SBRK under Contract No. FA864921P0899. Any opinions, findings and conclusions or recommendations expressed in this material are those of&nbsp; the author(s) and do not necessarily reflect the views of the USAF AFRL/SBRK.&nbsp;
&nbsp;

High performance SAM detection algorithms allow analysts to receive automated disposition of forces insights at machine-speed

Maintaining (SAM) Systems

SAM Introduction

ChatGPT has captured LLM headlines and amazed the AI community with its extensive natural language processing capabilities. A new LLM from Anthropic called Claude is competitive with ChatGPT and offers great promise. We evaluate both models head to head and give you our thoughts on how they compare.

Claude vs. ChatGPT: Let’s Think Step by Step

Meet Claude: Anthropic’s Rival to ChatGPT

Scale AI, the data infrastructure for AI, has been named a Cool Vendor in the 2022 <a href="https://www.gartner.com/doc/reprints?id=1-2C7YMC4Z&amp;ct=230109&amp;st=sb">Gartner Cool Vendors&trade; in Data-Centric AI</a> by Svetlana Sicular, Arun Chandrasekaran, Sumit Agarwal, Oleksandr Matvitskyy and Anthony Mullen.
The biggest barriers for organizations looking to implement data-centric AI strategies are having the necessary resources, talent, experience and high-quality data needed to feed algorithms and leverage the power of AI. Scale provides AI solutions tailored to business use cases and machine learning tools to unlock breakthroughs like Generative AI and operationalize AI for all organizations spanning the U.S. government, researchers, startups and Fortune 500 companies across e-commerce, logistics, technology, fintech, and more industries. Organizations such as Microsoft, Open AI, SAP, Flexport and the U.S. Army partner with Scale to solve problems with data labeling and annotation, scenario-based model testing and validation, ​​content understanding and contextualization, AI catalog for asset reusability and more.
According to Gartner, &ldquo;Data challenges plague AI implementations. They are often unexpected because AI teams focus on algorithms, and data concerns come as an afterthought.&rdquo; It recommends that when evaluating data-centric AI solutions, organizations should &ldquo;work with stakeholders to determine what data characteristics are high value for AI,&rdquo; and &ldquo;add AI capabilities such as AI-specific data management, synthetic data and data labeling &ndash; to your data management toolbox.&rdquo;
&ldquo;It&rsquo;s an honor to be named a Data-Centric AI Cool Vendor by Gartner. We believe that it is a testament to Scale&rsquo;s mission to accelerate the development of AI applications and showcases the market impact of our vision,&rdquo; said Alexandr Wang, CEO and Founder of Scale AI. &ldquo;The solutions that we provide enable organizations to reap the benefits of AI, whether they are building or applying AI. This means better business outcomes, a real return on investment, and a faster time to production.&rdquo;
According to the Gartner report, &ldquo;By 2025, 90% of new data and analytics deployments will be through an established data ecosystem, causing consolidation across the data and analytics market.&rdquo; Therefore, introducing a data management approach, such as data quality, data diversity, data and analytics governance, and data literacy to AI teams now, is imperative to any organization's AI strategy.
Download the full report <a href="https://www.gartner.com/doc/reprints?id=1-2C7YMC4Z&amp;ct=230109&amp;st=sb">HERE</a>.
Gartner Disclaimer
GARTNER and COOL VENDORS are the registered trademarks of Gartner Inc., and/or its affiliates and has been used herein with permission. All rights reserved. Gartner does not endorse any vendor, product or service depicted in its research publications, and does not advise technology users to select only those vendors with the highest ratings or other designation. Gartner research publications consist of the opinions of Gartner&rsquo;s research organization and should not be construed as statements of fact. Gartner disclaims all warranties, expressed or implied, with respect to this research, including any warranties of merchantability or fitness for a particular purpose.
About Scale AI
Scale accelerates AI development by providing the data infrastructure for AI. Its data-centric platform manages the entire ML lifecycle, from data annotation and curation to model testing and evaluation, enabling any organization &ndash; from the world&rsquo;s most advanced AI teams to legacy organizations &ndash; to develop and deploy impactful AI solutions. Scale combines ML technology with skilled human insight to ensure every AI application is built on a foundation of high-quality ground truth data. Scale is trusted by the largest technology companies in the world, including Microsoft, Toyota, Nuro, Samsung, Square, PayPal, SAP, Instacart, Pinterest, Brex, Flexport and U.S. government agencies like the U.S. Army and Air Force. Founded in 2016 and valued at $7.3 billion, Scale is backed by Founders Fund, Accel, Index, and Tiger Global.

Scale AI, the data infrastructure for AI, has been named a Cool Vendor in the 2022 Gartner Cool Vendors™ in Data-Centric AI.

Scale AI Named a 2022 Cool Vendor by Gartner® | Scale AI

Scale AI Named a 2022 Cool Vendor by Gartner®

Scale AI is dedicated to helping the Department of Defense (DoD) compete, fight, and win by providing high-quality, government-owned data that can power any AI/ML and autonomy effort.&nbsp;A first step towards this goal is organizing the DoD data layer. Technologies still relatively new for the DoD - like cloud computing and data repositories - are a good start. These are insufficient however, to provide the DoD with data processing capabilities required to fuel large-scale autonomy operations.&nbsp;Scale can provide the next critical layer in the software stack which can bring autonomy efforts out of the labs, into production, and onto the battlefield.
Beyond developing, training, and deploying individual autonomous systems, taking the time to &ldquo;get the data layer right&rdquo; can have broader impact, such as enabling the creation of &ldquo;Data Annexes&rdquo; to go along with Operational Plans (OPLANs).&nbsp;These would allow DoD leaders to provide the US Army&rsquo;s autonomous vehicle fleets with key, pre-validated datasets for critical operations in theater. These would include theater-specific perception models (LiDAR, camera, radar), segmentation-mapped avenues of approach, Order-of-Battle class and type IDs for detection and recognition, EO and SAR imagery mapped to friendly schemes of maneuver, and much more. With a single update, a vehicle fleet would be prepared to move out.&nbsp;
Scale provides the data and machine learning operations (MLOps) pipelines for the world&rsquo;s most ambitious robotics and autonomy teams.&nbsp;As the DoD explores novel approaches to human-machine teaming, it will require a sophisticated, government-owned data strategy with dedicated tooling.&nbsp;This will provide the numerous vehicle platforms and programs with the highest quality data to drive production-grade autonomy. Getting this right is a precursor to mission success.&nbsp;
Scale provides an open data architecture and MLOps pipeline in alignment with the DoD Modular Open Systems Approach (MOSA), decoupled from vehicle ownership.&nbsp;This allows the US Government to reuse training data and algorithms developed within Scale platforms for other programs. Thus the DoD can create a &ldquo;data flywheel&rdquo; that strengthens as more programs develop production-grade autonomy.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fceb82bb90c4741f2a98cdc32b2de19a4">
Scale currently executes billions of <a href="https://scale.com/guides/data-labeling-annotation-guide" target="_blank" rel="noopener noreferrer">annotations</a> per month, maps 3,800 kilometers of roads per week, and leads the industry as the AI perception data provider. Scale partners with self-driving vehicle companies training and building systems ranging from small delivery vehicles to long-haul trucks. Our partners include Kodiak Robotics, Voyage, Optimus Ride, Toyota Research Institute, Embark, General Motors, Honda innovations, Oshkosh, and Nuro - the first autonomous ground delivery vehicle to receive <a href="https://scale.com/customers/nuro" target="_blank" rel="noopener noreferrer">DoT approval</a> for public roads. Additionally, Scale is helping to develop autonomy in other unique domains, including maritime vessels (<a href="https://scale.com/customers/sea-machines-robotics" target="_blank" rel="noopener noreferrer">Sea Machines</a>), autonomous drones (<a href="https://scale.com/customers/skydio" target="_blank" rel="noopener noreferrer">Skydio</a>), and autonomous warehouse systems (<a href="https://scale.com/customers/otto-motors" target="_blank" rel="noopener noreferrer">OTTO Motors</a>).&nbsp;
Through our in-house capabilities, direct partnerships, and an extensive subcontractor network; Scale collects, curates, and <a href="https://scale.com/guides/data-labeling-annotation-guide" target="_blank" rel="noopener noreferrer">processes</a> more ground truth perception data than any other company in the world.
We are a $7B company with real-world applications at high levels of mission-readiness. Scale has over 700 employees focused on cutting edge ML development, human-machine teaming, content understanding, and novel applications for machine learning in the commercial and federal sectors. Our Modular Open System Architecture reduces cost and time to deployment, increases opportunities for iteration, and allows the government to pick and choose the tools and data it needs, all while maintaining full data rights.
Jonathan Hudgins is a member of the Federal Team at Scale where he focuses on bringing Scale&rsquo;s proven commercial solutions to solve Federal Government problems. Jonathan is a former active duty Air Force officer and F-15E Fighter Pilot.

Autonomy Table Stakes:  DoD’s Data Layer

On November 28th, OpenAI released a new addition to the GPT-3 model family: <code>davinci-003</code>. This latest model builds on <a href="https://openai.com/blog/instruction-following/" target="_blank" rel="noopener noreferrer">InstructGPT</a>, using reinforcement learning with human feedback to better align language models with human instructions. Unlike <code>davinci-002</code>, which uses supervised fine-tuning on human-written demonstrations and highly scored model samples to improve generation quality, <code>davinci-003</code> is a true reinforcement learning with human feedback (RLHF) model. It is trained with <a href="https://openai.com/blog/openai-baselines-ppo/" target="_blank" rel="noopener noreferrer">PPO</a> to optimize the generated text&rsquo;s score against a separate &ldquo;reward model&rdquo;, which is trained on rating comparisons by human graders of different model outputs. More details can be found in OpenAI&rsquo;s <a href="https://beta.openai.com/docs/model-index-for-researchers" target="_blank" rel="noopener noreferrer">model index for researchers</a>. The net result is that <code>davinci-003</code> is tuned to produce outputs that it thinks would be scored highly by humans. Scale is proud to partner with OpenAI to provide this human feedback.
OpenAI&rsquo;s announcement email mentions the following improvements for <code>davinci-003</code>:
<ul>
<li>&ldquo;It produces higher quality writing. This will help your applications deliver clearer, more engaging, and more compelling content.</li>
<li>It can handle more complex instructions, meaning you can get even more creative with how you make use of its capabilities now.</li>
<li>It&rsquo;s better at longer form content generation, allowing you to take on tasks that would have previously been too difficult to achieve.&rdquo;</li>
</ul>
But how much better is <code>davinci-003</code>, really? We decided to put it to the test quantitatively. Using <a href="https://scale.com/spellbook" target="_blank" rel="noopener noreferrer">Scale Spellbook</a>, the platform to build, compare and deploy large language model apps, we evaluated <code>davinci-003</code> versus <code>davinci-002</code> across tasks ranging from few- and zero-shot classification, summarization, and poetry writing. Here&rsquo;s what we found.
<h3>Classification</h3>
It&rsquo;s well known that <code>davinci-002</code> performs very well on classification with a few-shot prompt, and it appears that <code>davinci-003</code> offers comparable (but slightly worse) performance few-shot. Using Spellbook&rsquo;s built-in evaluation feature on 250 Yelp reviews, we compare the two models&rsquo; classification of these reviews as Positive, Negative, or Neutral in a four-shot prompt&mdash;meaning four labeled examples are included in the prompt itself.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F65f8dfd9c9744c15967d03341c5cdc73">
The davinci-003 model has 87.2% accuracy on this 250 Yelp review dataset sample, slightly lower than davinci-002&rsquo;s 91.6%.
However, if we ask the two models to classify these same reviews in a zero-shot prompt, we find that <code>davinci-003</code> scores more than 30% higher in accuracy than <code>davinci-002</code>&mdash;and on par with a <code>davinci-002</code> few-shot prompt! This is a substantial improvement in zero-shot classification performance. In this classification setting, adding examples to the prompt actually lowers <code>davinci-003</code>&rsquo;s accuracy.
<img src="https://lh6.googleusercontent.com/Wr30hJNdDAekBQTxtBRBN40_GR1CIElZlygaJSrAbGd6OK20V-pGS_bI5BoCF2SS4az3z7EFPYde3k5TnoVKHZd-68dwlhtHctxB7XtW5zqOXOjQkY2P8SxCkAjzlmg1PpKQ1hzEgob2W18laWD8FfSjNYShqjurevSldtMENyh8DUALgcMeYi4HQKi-">
Beyond accuracy, we found that asking <code>davinci-003</code> to explain its rationale in a zero-shot prompt works reliably, and provides a better explanation than a one-shot <code>davinci-002</code> about 75% of the time, based on Scale&rsquo;s human evaluation workforce judging a sample of 100 outputs.
<h3>Text Generation</h3>
The most notable performance leap comes in how <code>davinci-003</code> understands word and sentence structure, especially rhyming and syllables. In prompting for a poem that rhymes, <code>davinci-003</code> constructs a desired output, whereas <code>davinci-002</code> largely fails to implement rhyming.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2F31cb06da7e1a488a91f62ef14feee823">
<code>davinci-003</code> is also better equipped to handle requests for complex output structures, such as iambic pentameter (think Shakespeare). <code>davinci-002</code> certainly provides a poem, but ignores the request for lines of 5 pairs of stressed and unstressed syllables (buh-BUM). While not perfect, <code>davinci-003</code>&rsquo;s output is significantly more faithful to the requested prompt.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fc3fc8edf1aea432bb7bb5118a6dc8c82">
<h3>Summarization</h3>
One practical difference between the two variants is verbosity. In our experimentation, <code>davinci-003</code> consistently generates longer outputs than <code>davinci-002</code> on the same prompts. This is most visible in summarization tasks, where the value of the result may depend on its brevity.
In a sample of 60 articles, we tested both variants&rsquo; summarization capabilities by asking them to summarize the introductory paragraphs of lengthy Wikipedia pages. The additional text generated by <code>davinci-003</code> is typically beneficial, including information central to an article&rsquo;s importance. However, the resulting summaries are also wordy. In our analysis, <code>davinci-003</code> generates outputs about 65% longer than <code>davinci-002</code> under identical prompts.
<img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fbbd0ed222dff4216ab9173d07d176890">
It will be important for GPT-3 users to bear in mind <code>davinci-003</code>&rsquo;s talkativeness when choosing between the two variants. In our testing, given the task of generating a 1-sentence summary, <code>davinci-003</code> produced 42 words on average, nearly doubling <code>davinci-002</code>&lsquo;s average of 23. The shortest summary <code>davinci-003</code> gave back was a beefy 32 words, compared to only 12 words for <code>davinci-002</code>. This tendency extends to multi-sentence summaries as well.
For use cases that value brevity, <code>davinci-002</code> is the safer choice. Nevertheless, for many use cases, the longer and more detailed completions provided by <code>davinci-003</code> can be beneficial.&nbsp;
<h3>Conclusion</h3>
Overall, <code>davinci-003</code> is a meaningful improvement from <code>davinci-002</code>, especially for zero-shot classification and complex text generation.
While many of the benefits of the latest iteration of GPT-3 are apparent at the surface, concrete evaluation metrics ensure models and prompt strategies are performing as well as possible.
Luckily, Spellbook makes that easy, allowing users to run instant programmatic evaluation in classification use cases, or sending model outputs to Scale&rsquo;s high-quality workforce to get quantitative evaluation metrics for generative use cases.
<a href="https://scale.com/spellbook" target="_blank" rel="noopener noreferrer">Sign up for Spellbook today!</a>
P.S. If you&rsquo;re looking for a little dramatic flair in your emails to coworkers, <code>davinci-003</code> has you covered. It performs significantly better at creatively translating text or even explaining code as different celebrities in a zero-shot setting.
&nbsp;
BranchCommitmaintestmaintest <img src="https://cdn.builder.io/api/v1/image/assets%2Fe0438815ba51486bbb6a202747122d4b%2Fa4f4e03e1cef4d28a44fcf465150aeca">
<pre class="ql-syntax" spellcheck="false"> Write a short rhyming poem about winter in New York davinci-002
davinci-003
</pre>
In New York City When winter descends
The air is so crisp and chilly
And the wind never ends
The snowflakes are so big and fluffy
You just want to grab a handful
And then make a snowball
People hurry to and fro
Bundled up in coats
Hats, and scarves, and gloves
No one wants to be caught
Lights are twinkling in the night, It's wintertime in New York City so bright.
The snow is glistening, the wind is cold,
The atmosphere is really bold.
The snowflakes fall with a gentle grace,
Adorning the city with a lovely lace.
The air is crisp, and the people merry,
It's a magical winter in New York City.
Text Generation
The most notable performance leap comes in how davinci-003 understands word and sentence structure, especially rhyming and syllables. In prompting for a poem that rhymes, davinci-003 constructs a desired output, whereas davinci-002 largely fails to implement rhyming.

GPT3.png

We evaluate davinci-003 across a range of classification, summarization, and generation tasks.

We evaluate davinci-003 across a range of classification, summarization, and generation tasks. Using Scale Spellbook, the platform for large language model apps, we show where davinci-003 significantly outperforms the prior version and where it still has room to improve.

How Much Better is OpenAI’s Newest GPT-3 Model?

<h3>Background</h3> The rise of social media platforms has driven the popularity of user-generated content (UGC) and it is changing from the bottom up at a pace faster than ever. Platforms are competing for user attention and engagement - some are winning, but some are falling behind. How can a social media platform stay ahead of the game? One of the most effective ways to incentivize users to create and engage with content on the platform is by curating and promoting trends. A trend is a type of content repeated and shared amongst creators -&nbsp;a specific subject matter, action, or format that rises in popularity and eventually becomes viral. Common types of trends include dance challenges, transitions, and comedy skits. While videos in a trend follow a similar format, creators usually express their creativity and individuality by adding a personal touch to the videos. For that reason, the variety of the videos within the same trend can be wide and unpredictable. The key elements of these video trends include the video’s image, motion, audio, and text overlay. For example, a transition trend is a trend where the participants show a sudden change in their appearance, often signified by a dramatic sound effect in the background music and sometimes accompanied by dance moves, lip syncing, and text overlay. The lifecycle of a trend can vary from days to months depending on various factors. To remain competitive, social media platforms need to detect the trends while they are still emerging and put them to use immediately by matching them to the right creators and promoting them to the suitable audience.&nbsp; <a href="https://scale.com/content-understanding" rel="noopener noreferrer" target="_blank">Scale Content Understanding</a> improves platform experiences by enriching content metadata, discovering trend insights, and flagging sensitive content. One feature of Content Understanding is content data enrichment for better trend detection. To facilitate this process, we developed proprietary algorithms to detect trends early, quickly, and thoroughly.&nbsp; <h3>ML Clustering Overview</h3> <img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/449d8647-5509-470c-a6cf-ea38c14a62f3"> We took a human-in-the-loop (HitL) machine learning approach to detect trends. We represent trends as clusters of videos estimated from video-extracted embeddings. Once formed, these clusters are sent to a human pipeline (which we call “Cluster Labeling”) where taskers answer a series of questions about the videos in a given cluster - namely, if a sample of the videos corresponds to a trend and the potential name of the trend. The following are the phases of the ML Clustering process: <h3>Phase 1: Embedding Extraction</h3> To extract embeddings, we use a variety of multi-modal information extraction machine learning models, each of which is trained on large-scale, common datasets with high distribution overlap to the social media domain. <h3>Phase 2: Clustering</h3> Once the embeddings are extracted, we perform a clustering pipeline to take in these embeddings and output clusters. Overview Given the embeddings, we first perform <a href="https://arxiv.org/pdf/1802.03426.pdf" rel="noopener noreferrer" target="_blank">Uniform Manifold Approximation and Projection (UMAP)</a> dimensionality reduction (inspired by <a href="https://arxiv.org/abs/2203.05794" rel="noopener noreferrer" target="_blank">BERTopic</a>). UMAP takes in high-dimensional data and reduces it down to a lower dimensional space scalably (e.g. UMAP performs computation on tens of thousands of points on the order of tens of seconds). It reduces dimensionality in a way that preserves global Riemannian geometric structure and pairwise distances. On these low-dimensional embeddings, we perform an iterative hierarchical clustering process. Each individual point in embedding space is initially represented as a cluster.&nbsp;We then calculate the proximity between all pairs of clusters. We calculate the proximity of two clusters in a given pair as the variance in embedding space (i.e. the average sum of squares from the joined cluster mean embedding vector) of the points corresponding to if these clusters were to be joined. The two clusters that are closest to each other are joined given the above definition of proximity. This cluster-joining process is performed iteratively until all clusters are above a specified variance that is provided as a hyperparameter. <img src="https://drive.tiny.cloud/1/0lbhu31lrngl0y5eo8e6p0jckqgeikc94mnkaig43yuahupv/24053e18-8fbc-46a8-91ba-4ea274599636"> Stop Condition Variance Hyperparameter We carefully tune the variance hyperparameter in the stop condition to balance between making the clusters too coarse or too granular. For large variance hyperparameters, the final clusters are allowed to span a wide region in embedding space, meaning they are not semantically similar enough to represent trends (which tend to be extremely specific). Diametrically opposed, for small variance hyperparameters, the clusters end up with too few videos for users in the cluster labeling pipeline to recognize the trend. We therefore carefully tune the stop condition variance hyperparameter. This provides a balance between the clusters being fine enough that they represent groups of videos that are semantically similar enough to correspond to trends, and coarse enough that users from the cluster labeling pipeline can recognize the trend. This hyperparameter is normalized in order to reduce the amount of per-dataset manual tuning. <h3>Phase 3: Cluster Post-Processing</h3> To improve the quality and throughput of the taskers, we prune out clusters that are composed of a number of videos that is smaller than our definition of a trend. We also add a post-processing density score to help taskers focus on the clusters which are tightest in embedding space - i.e. the ones that are most likely to be semantically similar. This density score is calculated as the intra-cluster variance of each cluster in the dimensionality-reduced embedding space. <h3>Conclusion</h3> Our trend detection product consistently performs at a higher than 90% recall and 90% precision in production. Our HitL ML approach enables us to stay flexible with customization towards specific content niches while maintaining a high throughput. We are able to process tens of millions of videos with a few hours of turnaround. Our priority is to not only detect trends promptly, but also equip our customers with the data and insights on these trends to take actions immediately.&nbsp; At Scale, we believe investing in data is the key to building better platform experiences and discovering trend insights. If you’re interested in learning more about <a href="https://scale.com/content-understanding" rel="noopener noreferrer" target="_blank">Scale Content Understanding</a>, speak to our team today.&nbsp;

Discovering short-form video trends is challenging. In this blog post, we explain how we use an ML + HiTL approach for faster and higher-quality trend detection.

Discovering Emerging Short Form Video Trends with Clustering and HitL

URL redirections for Website. Disclaimer: takes 5~ min for each added or edited.

url-redirects

Use relative path. use `/blog/foo` not `https://scale.com/blog/foo`

blog-filter-type

homepage-quotes

logos

homepage-content

quotes

blog-category-model

announcement-banner

homepage-resources

homepage-event

custom-fonts

author

Event model for new marketing events page

event

Optional second image intended to show on the grid view, should be a more squared image closer to 1:1 aspect ratio than the one used in the Cover field.

guides

announcement-news

homepage-research

legal

slugs must be all lowercase. ie: ‘my-slug’

eval

datastream

customers

Display order on the page /customers. Higher values show first

summit-2023-speakers

event-model

Toggle if the event is featured. Featured events will be moved to the top of the list

basecamp-agenda

basecamp-speakers

Example: https://www.twitter.com/username

Example: https://www.linekin.com/in/username

summit-2023-agenda

summit-2023-partners

enterprise-use-cases-categories

enterprise-use-cases-categories-items

partners

data-samples

enterprise-use-cases

Use only one image or video option. Add another item to add another source.

research

blog-model

Option to list or not list this post in the Blog index. Enabled = not list in the index, Disabled = list in the index

Products

Enterprise

Government

Resources

Customers

Leaderboards →

Blog