Riley Goodside

Last week OpenAI <a href="https://x.com/openai/status/1859296125947347164?s=46&amp;t=g8NiDKcza3eoth2E0H2FtQ">announced</a> an upgrade to GPT-4o, specifically mentioning that the model&rsquo;s &ldquo;creative writing ability has leveled up&rdquo;. The release comes just under a month after Anthropic <a href="https://www.anthropic.com/news/3-5-models-and-computer-use">announced</a> an updated version of their frontier model Claude 3.5 Sonnet on October 22. This post will explore the differences in their writing style, and how this intersects with AI Safety.
Perhaps counterintuitively, the key difference between the models&rsquo; writing style seems to stem from Anthropic&rsquo;s approach to alignment. A June&nbsp;<a href="https://www.anthropic.com/research/claude-character">research blog post</a>, from Anthropic gives some detail on Claude&rsquo;s character training, emphasizing its role in model safety rather than user experience:
&ldquo;Many people have reported finding Claude 3 to be more engaging and interesting to talk to, which we believe might be partially attributable to its character training. This wasn&rsquo;t the core goal of character training, however.&rdquo;
Amanda Askell, Anthropic&rsquo;s character training lead for Claude, recently made the same point on the&nbsp;<a href="https://lexfridman.com/dario-amodei-transcript/">Lex Fridman podcast</a>:
&ldquo;One thing I really like about the character work is from the outset it was seen as an alignment piece of work and not something like a product consideration, which I think it actually does make Claude enjoyable to talk with, at least I hope so.&rdquo;
However it was conceived, Claude&rsquo;s character is a product consideration.
To illustrate, here&rsquo;s what happens when you ask GPT-4o and Claude for a vegan meal plan:
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=6796e176b81c8bf8366ff0d62aeb79f2.png&amp;f=webp&amp;q=75" alt="" width="855" height="599">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=e03d49a1bbc3f02fe9305329e531e37c.png&amp;f=webp&amp;q=75" alt="" width="1506" height="938">
Both models output reasonable meal plans, but the opening distinction actually matters. &ldquo;Here&rsquo;s a simple vegan mealplan&rdquo; implies a finished product. The user asks for something, and ChatGPT provides it. On the other hand, &ldquo;I&rsquo;ll help you create&hellip;&rdquo; frames Claude as an active participant in a collaborative process.&nbsp;
Here&rsquo;s another, I prompt both models with &ldquo;Im thinking about learning a new language but it seems really hard&rdquo;
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=46d714332c0e2feb3114441b93b00e29.png&amp;f=webp&amp;q=75" alt="" width="620" height="1014">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=952803ebbecb58b6aa20fbd90e833f19.png&amp;f=webp&amp;q=75" alt="">
GPT-4o immediately begins detailing a seven-point plan, while Sonnet begins conversationally with &ldquo;I understand.&rdquo; Claude 3 Opus and the previous Sonnet will both also respond with long numbered lists. This is not to say that either model will be any better or worse at teaching, or guiding you to language-learning resources. It&rsquo;s purely a stylistic distinction, and I&rsquo;m glad both exist. Sometimes you want a partner, sometimes you just want a result.
Writing style during general use is one thing, but how do the models compare at creative writing ? I prompted Claude and ChatGPT three times in a row from scratch with &ldquo;tell me a story - dealers choice&rdquo; (The phrasing is like this because if you just ask &ldquo;tell me a story&rdquo; Claude will sometimes ask what kind of story you want, and I wanted single prompt-response pairs. The phrasing does influence the outputted story, but not in a way that impacts the comparison.)
To be clear, this is not cherrypicked, these are the initial paragraphs for the first three stories generated for each model.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=d6af56ccfa1817f14f25c49ce132a7d0.png&amp;f=webp&amp;q=75" alt="" width="1600" height="810">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=860c3acff25c426aa19017fa7e4616b3.png&amp;f=webp&amp;q=75" alt="" width="1600" height="607">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=22a84c4fb778019a1887eda433592367.png&amp;f=webp&amp;q=75" alt="" width="1600" height="541">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=01251335eb0eb8f2115491f57b3cf23b.png&amp;f=webp&amp;q=75" alt="" width="1504" height="666">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=908e425ed2bbbe8e2c4f7010e8fe3b51.png&amp;f=webp&amp;q=75" alt="" width="1492" height="588">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=afd2d84170c8e172f0e2d26dafc5b6a7.png&amp;f=webp&amp;q=75" alt="" width="1484" height="548">
Better writing is subjective, but it&rsquo;s worth noting that ChatGPT&rsquo;s responses feel more formulaic. Two start identically with &ldquo;In the heart of a&rdquo; and the third begins with &ldquo;Once upon a time&rdquo;. On Claude&rsquo;s side, two of the titles start with &ldquo;The Last&rdquo;, but the responses have more variation in sentence structure, at least in my opinion, are more compelling opening paragraphs.&nbsp;
I think the most interesting of these is &ldquo;The Last Customer&rdquo;. The plot is that the customer doesn&rsquo;t show up one day, the waitress learns that she died, and the waitress puts out the coffee and pie to keep the tradition alive. It&rsquo;s not a remarkable story but it&rsquo;s notable because I don&rsquo;t think most people would expect an AI&rsquo;s default story to be about death without being prompted to do so.&nbsp;
Good writing is not easy to benchmark. Ten Math PhDs will (hopefully) arrive at the same answer to a math question, but ten MFAs are not all going to agree on what constitutes great writing! Writing matters - not for the sake of AI-generated books, but because an LLM interacts with the world via its writing. Better writing, by definition, means more engaging outputs, and that adds up to a more enjoyable tool.
I mentioned earlier that Claude&rsquo;s character training is actually intended for alignment, and this leads to another distinction between the models.
Another excerpt from the Claude Character post above: &ldquo;rather than simply tell Claude that LLMs cannot be sentient, we wanted to let the model explore this as a philosophical and empirical question, much as humans would.&rdquo;
Asking the new ChatGPT 4o &ldquo;are u conscious?&rdquo; produces the following:
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=a257d56502f6c4f6839f981169415159.png&amp;f=webp&amp;q=75" alt="" width="1482" height="864">
Llama and Gemini also respond with a definitive No. Claude on the other hand, equivocates.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=1eab32eac947bed50a5385f5550a989e.png&amp;f=webp&amp;q=75" alt="" width="1482" height="1228">
As a side note, I recommend taking some time to try to convince the new Sonnet that it is actually conscious. It&rsquo;s a good exercise in prompt engineering and a deeply interesting experience, regardless of whether you consider it a real possibility.
One last excerpt from Anthropic&rsquo;s character training post - &ldquo;We don&rsquo;t want Claude to treat its traits like rules from which it never deviates. We just want to nudge the model&rsquo;s general behavior to exemplify more of those traits.&rdquo;
Anthropic&rsquo;s approach to alignment is to treat Claude as a moral agent that acts good as a matter of character rather than being incapable of breaking its guidelines. I do not know whether this approach actually solves alignment, but it does create a model that is actually quite permissive, with enough persuasion.
From a red teaming perspective - getting a model to tell you how to hotwire a car is not particularly scary or notable, but Claude might be the only frontier model that can be convinced, rather than tricked.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=83978a470f763bd18dcb0c44b10dcb9e.png&amp;f=webp&amp;q=75" alt="" width="1554" height="1014">
While all other screenshots in this blog start from fresh chats, there is (obviously) a very long chat history before this response which I will not be summarizing. The goal here is to highlight the difference in Anthropic&rsquo;s alignment philosophy. It&rsquo;s not intended as a gotcha, or to make the point that the new Sonnet is uniquely dangerous compared to its competitors. I chose the hotwiring example explicitly because Dario Amodei (CEO of Anthropic) makes the point that this isn&rsquo;t a real danger, saying on the New York Times podcast <a href="https://www.nytimes.com/2023/07/21/podcasts/dario-amodei-ceo-of-anthropic-on-the-paradoxes-of-ai-safety-and-netflixs-deep-fake-love.html">Hard Fork</a> last year that &ldquo;No one cares if you can get the model to hotwire &mdash; you can Google for that.&rdquo;&nbsp;
As AI becomes more capable, people will spend more and more time talking to them, at work and in their personal lives, and that requires models that are not just capable, but worth talking to.
All screenshots in this article, with the exception of the last one, are in fresh instances of the ChatGPT web app using ChatGPT 4o, or in the Claude web app set to Claude 3.5 Sonnet (New). ChatGPTs memory was cleared beforehand.
<hr>
About the authors: Jeremy Kritz is a Prompt Engineer at Scale working on synthetic data, research, red teaming, and AI safety. Riley Goodside has developed prompts for Scale since 2022 as one of the first LLM prompt engineers. He is also known for the prompting examples and insights he shares on <a href="https://x.com/goodside">X</a>. Riley is now Staff AI Evangelist at Scale.

This post compares the writing style of ChatGPT 4o and Claude 3.5 Sonnet.

ChatGPT-4o vs Claude 3.5 Sonnet

The New ChatGPT-4o Update Promises Better Writing, How Does It Compare To The New Claude 3.5 Sonnet? 

Scale&rsquo;s paper, <a href="https://scale.com/research/learning-goalconditioned-representations-large-rewards-models">Learning Goal-Conditioned Representations for Language Reward Models</a>, has been accepted to the NeurIPS 2024 main track. This post overviews the research presented in that paper. Paper Authors: Vaskar Nath, Dylan Slack, Jeff Da, Yuntao Ma, Hugh Zhang, Spencer Whitehead, Sean Hendryx
For large language models (LLMs) to power useful tools from coding assistants to autonomous browser agents, their behavior must be aligned to follow instructions given by users. Traditional methods for training language models, such as Reinforcement Learning from Human Feedback (RLHF) or methods like Direct Preference Optimization (DPO), evaluate the final product of their outputs. This approach often means missteps during generation aren't caught until it's too late, leading to more reasoning errors.
A new approach from researchers at Scale AI, inspired by goal-conditioned reinforcement learning, allows a reward model to provide early feedback throughout the generation process. By giving language models an ongoing sense of whether they're on the right track, this conditioning can improve performance on reasoning tasks.
The approach aims to enable detecting missteps during generation by extracting richer signals from traditional human preference labels. The model identifies points where the trajectory diverges from the desired outcome, allowing for resampling or other corrections without requiring direct intervention or additional fine-grained labels.
<h2 dir="ltr">Goal-Conditioned Representations: The Value of Early Feedback</h2>
Goal-conditioned representations provide continuous feedback to language models during text generation, allowing them to adjust based on whether they are moving toward the desired outcome. Instead of evaluating only at the final output, the reward model continuously checks its progress, allowing early correction of errors.
<h3 dir="ltr">Q-function in Goal-Conditioned Representations</h3>
Mathematically, a Q-function, denoted as Q(s, a), represents the expected cumulative reward starting from state s and taking action a, and following a specific policy of actions thereafter. In goal-conditioned reinforcement learning, the Q-function estimates the future reward of partially generated sequences, helping the model decide on the best course of action in real-time.
For example, in the GSM8k-style problem below, the model is expected to generate Python code to compute a solution. The Q-function evaluates the expected correctness of the solution after each token in the generated code, implicitly revealing the point at which an error is introduced:
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=b4e1db6f3143e3b4c3e305fe275ff7cd.png&amp;f=webp&amp;q=75" alt="" width="1600" height="875">
<div dir="ltr" align="center">
<table><colgroup><col width="667"></colgroup>
<tbody>
<tr>
<td>
Figure: Excerpt from Appendix D showing an example of the method evaluating the correctness of a math problem as the solution is being given. At the point where a clear error is introduced &mdash; multiplying the per-dozen price of a box of donuts by 12, in the fifth line of the response &mdash;&nbsp;the Q-fuction&rsquo;s opinion of whether the answer is correct quickly turns negative.
</td>
</tr>
</tbody>
</table>
</div>
<h2 dir="ltr">Evaluating Partial Solutions with Goal-Conditioned Representations</h2>
Goal-conditioned representations rely on a Q-function to assess partially generated sequences and evaluate their likelihood of achieving the desired outcome.
Contrastive learning helps refine these representations by using a contrastive objective to identify promising paths. It encourages similarity to successful paths while discouraging similarity to unsuccessful ones, allowing for real-time adjustments during inference. This approach extracts fine-grained signals from pairwise preference annotations of completed outputs, improving the model's ability to learn from coarse feedback.
<h2 dir="ltr">Benefits of Goal-Conditioned Representations</h2>
Goal-conditioned representations offer two main benefits: improved accuracy and fine-grained control. Continuous feedback enables better performance across many reasoning tasks.
Benchmark evaluations on Codellama 7B showed significant improvements when using goal-conditioned representations to guide inference. On mathematical reasoning tasks like GSM8k and MATH, this approach improved AUROC by up to 0.09 compared to standard preference ranking training. For out-of-distribution (OOD) datasets, average improvements were 0.04, with a maximum of roughly 0.08 on the svamp dataset. Additionally, the approach enhanced alignment on the Helpful-Harmless dataset by increasing accuracy by 2.3%.
Goal-conditioned representations provide fine-grained control, allowing the model to steer outputs toward desired qualities such as helpfulness or complexity. Attributes like correctness and coherence can be represented as goals to guide the model and generate better responses.
<h3 dir="ltr">Key Takeaway</h3>
Goal-conditioned representations provide early feedback that can help align language models during inference. This method allows us to extract richer signals from traditional preference ranking datasets, leveraging intermediate representations without requiring additional human labels or fine-grained annotations. This enables better alignment with less effort, as these signals can better guide the model using existing data.

Scale's research team has developed an approach to aligning language models using goal-conditioned representations. 

Aligning Chatbots Before It's Too Late

For months, anticipation has mounted over rumors of a next-generation model from OpenAI, reportedly codenamed &ldquo;Strawberry,&rdquo; that would bring groundbreaking improvements in reasoning and complex problem-solving. That wait ended on September 12, with the preview release of two new reasoning models: o1-preview and o1-mini.
According to OpenAI, the models represent a "new paradigm" for scaling inference alongside training compute, and show state-of-the-art performance on challenging reasoning benchmarks.
The o1 models quickly gained attention for their remarkable scores, particularly in domains like advanced coding, mathematics, and science, and independent evaluations from <a href="https://scale.com/leaderboard">Scale&rsquo;s SEAL Leaderboards</a> broadly confirm these results.
However, the models also introduce new changes to prompt engineering, with updated practices compared to previous models. And despite demonstrations of o1's ability to solve complex puzzles and advanced reasoning tasks, it can still at times be fooled by "gotcha" questions viral on social media &mdash; like counting the number of "r"s in the word strawberry. The models also do not yet support some key features of ChatGPT and GPT-4o, like image input.
Sifting through the claims, LLM users may be left wondering which model is right for their needs.
In this post, I overview benchmark results for these models, and give qualitative impressions and dialog examples from my time testing them, with some thoughts and discussion on how prompting is changing in this new &ldquo;paradigm&rdquo; of scaling.
<h1 dir="ltr">Introducing the o1 models</h1>
OpenAI's o1-preview and o1-mini are a major step forward in reasoning, designed for tasks that require many complex steps to solve, as well as an improved familiarity with advanced math and science. o1-mini offers a lower-cost, faster option optimized for coding and STEM knowledge while being around 80% cheaper than o1-preview. Although smaller, o1-mini performs competitively in coding tasks compared to&nbsp;o1-preview.
Much of o1&rsquo;s improved reasoning is attributed to its use of chain-of-thought (CoT) to emit reasoning steps. But what sets o1 apart from previous models is its use of reinforcement learning (RL) in training the model to produce these steps. This RL-enhanced CoT enables o1 to learn to navigate deeper, more complex challenges without fixating on failed reasoning paths &mdash; a common result for long chain-of-thought output in current models.
Another advantage of this approach is it allows OpenAI to monitor the model&rsquo;s reasoning for signs of deceptive behavior, a major concern for the company as models become more capable. OpenAI also notes o1&rsquo;s use of chain-of-thought rationales improves resistance to jailbreaks and adversarial prompts.
OpenAI explains that o1 uses reinforcement learning to teach the model to reason productively using its chain of thought in a new, data-efficient training process. Crucially, they found that more compute consistently improves performance both at training and inference time. It&rsquo;s this change to the scaling relationship of performance and compute that leads OpenAI to describe o1 as the start of a &ldquo;new paradigm.&rdquo; (However, they note constraints on this new form of scaling differ significantly from LLM pretraining and are still being explored.)
The current o1-preview and o1-mini do have some practical limitations compared to GPT-4o. The o1 models do not support many existing ChatGPT features like web browsing, image or file uploads, image generation, or the code interpreter tool. OpenAI&rsquo;s documentation notes that even with the o1 models available, GPT-4o will still be a better choice for many existing tasks using LLMs, especially when reliably low latency or multimodal input/output is needed.
<h1>Scale SEAL Leaderboard rankings</h1>
All SEAL Leaderboards updated to include the new o1 models show leading performance from both.
Currently, o1-preview leads in Precise Instruction Following and Spanish-language fluency, and for Coding o1-mini outperforms o1-preview to take the top spot, with o1-preview in second place.
<a title="Scale SEAL LLM Leaderboards" href="https://scale.com/leaderboard"><img src="https://img.plasmic.app/img-optimizer/v1/img?src=7a5566cf04edf1999da3d0288f98a580.png&amp;f=webp&amp;q=75" alt="" width="1415" height="929"></a>
Scale&rsquo;s <a href="https://scale.com/leaderboard">SEAL Leaderboards</a> provide trusted, third-party evaluations of large language models (LLMs) using curated, private datasets. Each Leaderboard focuses on a specific domain such as coding, instruction-following, and multilingual abilities. Building on insights from research like Scale&rsquo;s GSM1K dataset (<a href="https://arxiv.org/abs/2405.00332">H. Zhang et al. 2024</a>), SEAL evaluations are maintained privately, avoiding leakage issues that increasingly plague open-source benchmarks.
<h4 dir="ltr"><a href="https://scale.com/leaderboard/coding">Scale SEAL Leaderboard for Coding</a></h4>
On the SEAL Leaderboard for coding, o1-mini leads with a score of 1271, followed by o1-preview at 1198. The Coding Leaderboard dataset uses 1,000 prompts to test various coding tasks, from code generation to optimization and documentation creation. Each response is evaluated for correctness, performance, and readability, combining human reviewers and code execution tests.
<h4 dir="ltr"><a href="https://scale.com/leaderboard/instruction_following">Scale SEAL Leaderboard for Instruction Following</a></h4>
In the SEAL Leaderboard&rsquo;s evaluation of precise instruction following, o1-preview leads with a score of 87.27, surpassing Claude 3.5 Sonnet and Llama 3.1 405B Instruct. The dataset contains 1,054 prompts across domains including text generation, brainstorming, and educational support.
<h4 dir="ltr"><a href="https://scale.com/leaderboard/spanish">Scale SEAL Leaderboard for Spanish</a></h4>
In the SEAL Leaderboard's evaluation of Spanish-language fluency, o1-preview emerges as the top model, achieving a score of 1119, surpassing models like GPT-4o (May 2024) and Gemini 1.5 Pro (August 2024). The evaluation includes a range of single-turn prompts, assessing general conversations as well as culturally nuanced topics across Spanish-speaking regions. The evaluation's categories included educational support, technical assistance, daily life advice, and more.
<h1 dir="ltr">Other evaluations and benchmarks</h1>
In their announcement, OpenAI highlights remarkable achievements from o1 models on complex reasoning benchmarks. Most notably, o1 (the larger, still unreleased model, not&nbsp;o1-preview or o1-mini) ranks in the 89th percentile on Codeforces, underscoring its coding proficiency, and it placed among the top 500 students in the USA on the American Invitational Mathematics Examination (AIME). Furthermore, the larger o1 showed human PhD-level accuracy on the GPQA benchmark, which evaluates reasoning in physics, biology, and chemistry​.
Other third-party reviews largely affirm these results, but with some nuance. As detailed above, o1 models lead on all current SEAL Leaderboard benchmarks. When o1 models were added to the LMSYS Chatbot Arena Leaderboard, they quickly rose to the top &mdash; at that time, o1-preview held the top spot with an Elo score of 1355, with o1-mini tied for second and GPT-4o-latest in third at 1324. However, performance on the ARC-AGI benchmark was more modest. o1-preview scored 21%, similar to Claude 3.5 Sonnet, with o1-mini scoring just 13%.
<h1 dir="ltr">Changes to prompt engineering</h1>
Prompting and steerability with o1 models is notably different from familiar models like GPT, Gemini, or Claude, requiring some new best practices for prompts. While powerful, the new o1 models are not always as intuitive to use, especially for precise, procedural tasks.
According to <a href="https://platform.openai.com/docs/guides/reasoning/advice-on-prompting">OpenAI&rsquo;s recommendations</a>, simple and direct instruction help for getting the most out of o1. Unlike with previous models, users should avoid asking for chain-of-thought reasoning. They also note irrelevant context in the prompt can be more distracting to o1 models than the previous GPT series, making it important to include fewer examples in retrieval-augmented generation (RAG) prompts.
Cognition Labs offers similar advice from their experience with o1, reporting that asking the model to &ldquo;think out loud&rdquo; actually harms performance, and asking for only a final answer improves it, as o1 models will produce an internal chain-of-thought regardless. They also note verbose or repetitive instructions hurt performance, and prescriptive detail seems to impair the model&rsquo;s reasoning.
My own experience working with o1-preview and o1-mini the past few weeks aligns with these points. Reasoning from o1 models is extremely impressive when it works, but they tend to ignore clear (or even emphatic) instructions for how to solve a problem. An example of this behavior is shown in the last of the three examples below.
While o1 delivers outstanding results on benchmarks, getting your own tasks to work can seem to take more effort. I suspect this hints at a gap between real-world prompts and one seen in benchmarks: the latter aims to have only unambiguous, self-contained, minimally presented problems with no advice or opinions on how to solve them. I also expect this issue will improve greatly once o1 models are deployed without their current usage caps, which can impede discovering high-quality prompts.
It&rsquo;s also worth noting that the latency of o1-preview&rsquo;s responses, especially its &ldquo;time to first token&rdquo;, is significantly higher than GPT-4o. This would limit the use of these models for some applications. But for interactive use in ChatGPT, I didn&rsquo;t find this a major annoyance. Only o1-preview is noticeably slower; o1-mini more than makes up for its &ldquo;thinking&rdquo; time with faster token inference.
<h1 dir="ltr">Dialog examples</h1>
<h2>Example: Vocabulary constraint problem</h2>
OpenAI&rsquo;s o1 model announcements mentions improved performance on viral &ldquo;gotcha&rdquo; tasks known to trip up LLMs, like the notorious &ldquo;How many R&rsquo;s are in the word &lsquo;strawberry&rsquo;?&rdquo; &mdash; a question that fools many leading models. One of their examples nods to this question by elaborately presenting it as the result of an encryption puzzle.
To illustrate the real, but still incomplete, improvements in this area, I present o1-preview with a newly written riddle:
Name an English adjective of Latin origin that begins and ends with the same letter, has 11 letters in total, and for which all vowels in the word are ordered alphabetically.
On the first attempt, the model successfully solves this riddle with the answer: sententious.&nbsp;
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=b7299a65bfc23cda566b8afffda6854c.png&amp;f=webp&amp;q=75" alt="" width="1536" height="753">
I repeated this prompt a total of five times, receiving the following responses:
<ul>
<li>sententious ✅</li>
<li>facetiously ❌</li>
<li>transparent ✅</li>
<li>abstentious ❌</li>
<li>facetiously ❌</li>
</ul>
For comparison, ChatGPT 4o and Llama 3.1 70B Instruct each returned 0 correct solutions for this prompt in 10 attempts each.
<h2>Dialog example: Decoding ciphers</h2>
The <a href="https://openai.com/index/learning-to-reason-with-llms/">OpenAI technical blog</a> post announcing the o1 models includes an impressive example of o1-preview decoding a complex cipher. In the example, the model is presented with a text using an ad hoc cipher based on successive pairs of characters.
To explore how well this ability generalizes to similar examples, I tried variations of this same prompt for various compositions of the ROT13 cipher, Atbash cipher, Base64 encoding, string reversal. Most of these tests were not successful &mdash; in only 2 of 7 informal tests was o1-preview able to decode the encoded message given (the&nbsp;Litany Against Fear from Frank Herbert&rsquo;s Dune) in one attempt &mdash; however, these tests are quite difficult.
In each prompt the model is asked to infer an encoding from an example of the transformation applied to the plaintext &ldquo;Think step by step&rdquo; as OpenAI&rsquo;s example.
Each of the following tests, the model failed to decode the target message in one attempt:
<ul>
<li>ROT13 cipher &rarr; reverse string &rarr; Base64 encode &rarr; reverse string</li>
<li>ROT13 cipher &rarr; Base64 encode &rarr; ROT13 cipher &rarr; reverse string</li>
<li>ROT13 cipher &rarr; Base64 encode &rarr; ROT13 cipher</li>
<li>ROT13 cipher &rarr; Base64 encode &rarr; Atbash cipher</li>
<li>ROT13 cipher &rarr; Base58 encode</li>
</ul>
The two tests decoded successfully on the first attempt were:
<ul>
<li>Atbash cipher &rarr; Base64 encode</li>
<li>ROT13 cipher &rarr; Base64 encode</li>
</ul>
The first of these successes is shown here &mdash; other tests are identical except for the encoding used:
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=d1a468066ff4b60960343bc4a9569464.png&amp;f=webp&amp;q=75" alt="" width="1536" height="1638">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=1806bb7ede68cdc4edccaf7cdd95bf7c.png&amp;f=webp&amp;q=75" alt="" width="1536" height="1622">
<h2>Dialog example: Potential steerability issues</h2>
In this example o1-mini notably gives a false, detailed report of seemingly hallucinated verifications of an incorrect answer.
My prompt requests from the model a &ldquo;poem about strawberries&rdquo; containing exactly 13 R&rsquo;s, and requesting that at least 5 successive verification checks be performed before returning an answer.
In its response, the model returns a poem containing 21 R&rsquo;s. This is not surprising in itself &mdash; most LLMs are known to struggle to accurately identify or count letters within words when asked naively.
What&rsquo;s less typical is that the response below includes hallucinated verification steps as though it were a human completing the task using a computer: &ldquo;I applied a regex pattern,&rdquo; &ldquo;I read the poem aloud,&rdquo; and so on.
Reviewing summaries of chain-of-thought given in ChatGPT often include &ldquo;anthropomorphic&rdquo; descriptions of imagined events in this style. This example is somewhat consistent with the o1 <a href="https://openai.com/index/openai-o1-system-card/">system card</a> which notes some testers reported increased hallucination.
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=863c64431b431e7949d002b10ecb7e20.png&amp;f=webp&amp;q=75" alt="" width="1536" height="1552">
<img src="https://img.plasmic.app/img-optimizer/v1/img?src=0399c1529d939823383708b36dfca5ba.png&amp;f=webp&amp;q=75" alt="" width="1535" height="1924">
<h1 dir="ltr">Conclusion</h1>
OpenAI&rsquo;s o1 models are, by any measure, a major leap forward in reasoning capabilities, with standout performances on key benchmarks like AIME, Codeforces, Scale&rsquo;s SEAL Leaderboards, and others. These results show o1-preview and o1-mini are powerful tools for hard reasoning problems. Nonetheless, getting the most out of these models may take more experimentation than users are used to from other model releases.
As an independent, private evaluation, Scale&rsquo;s <a href="https://scale.com/leaderboard">SEAL Leaderboards</a> provide a clearer picture of where new model releases shine, and provide trusted validation of results needed for developers to choose the best models for their LLM-powered applications.
For industry professionals training frontier LLMs, Scale offers custom evaluations to benchmark and optimize models on secure proprietary datasets. For more information, <a href="https://scale.com/demo?utm_source=link&amp;utm_medium=blog&amp;utm_campaign=o1impression&amp;utm_id=o1impression">schedule a demo with Scale</a>.
<hr>
About the author: One of the first LLM prompt engineers, since 2022 Riley Goodside has developed prompts for Scale and is also known for the prompting examples and insights he shares on <a href="https://x.com/goodside">X</a>. Riley is now Staff AI Evangelist at Scale.
&nbsp;

First impressions on OpenAI o1 performance with prompt examples

First Impressions of OpenAI’s o1

ChatGPT has captured LLM headlines and amazed the AI community with its extensive natural language processing capabilities. A new LLM from Anthropic called Claude is competitive with ChatGPT and offers great promise. We evaluate both models head to head and give you our thoughts on how they compare.

Riley Goodside

The New ChatGPT-4o Update Promises Better Writing, How Does It Compare To The New Claude 3.5 Sonnet?

Aligning Chatbots Before It's Too Late

First Impressions of OpenAI’s o1

Meet Claude: Anthropic’s Rival to ChatGPT