Japanese

Introduction

The Scale AI Multilingual Prompts Dataset, composed of 1,000 prompts per language, is tailored to enhance models’ interaction capabilities across multiple languages. This dataset specifically aims to refine chatbots' proficiency in engaging with Japanese users, reflecting complexity of Japanese communication.

Dataset Description

This dataset introduces single-turn prompts across a diversified range of scenarios, aiming to evaluate and improve models’ responses in both general and culturally nuanced conversations.

Every prompt is designed to have a clear goal/request/instruction that specifies what needs to be achieved by the response and the objective is relevant to the category assigned to the task. In addition, the complexity of the prompts varies by adding different combinations of the following building blocks: Local Context, Limitations, Role, Audience, Style and Format.

For Limitations, every prompt can define one or several constraints. As for Role, specifying it ensures the model’s responses are judged within a specific, well-defined context, which aligns the evaluation criteria with the expected output. This approach provides consistency, improves relevance, and allows for a more targeted assessment of the model's ability to emulate expertise in a given domain. 

The evaluation process intentionally focuses on more complex use cases, which differ from the simpler and more straightforward conversations between a regular user and a chatbot. This approach more effectively differentiates models based on their varying underlying capabilities.

Construction Process

Development of this dataset followed a structured approach:

• Original Content Requirement: Unique content generation was enforced, prohibiting the use of existing resources or models.
• Review Stages: The content underwent qualitative and grammatical assessments.
• Final Quality Audit: A final evaluation on a select sample refined the dataset to 1,000 prompts.

Experts fluent in various languages with cultural knowledge were chosen to contribute, ensuring the dataset's relevance and authenticity.

Quality was maintained through:

• Multi-stage Reviews: Ensuring clarity, complexity, and cultural specificity.
• Internal Benchmarking: Monitoring and assessing annotator performance.
• Final Audits: Revising or removing prompts that did not meet quality standards.

Evaluation Taxonomy

To capture a nuanced assessment of both general language understanding and of regionalized styles from Japan, we created an evaluation taxonomy specific to evaluating this context. Each model response was evaluated across a set of stand-alone criteria, covering a diverse set of use cases, and side-by-side with another model response to measure preference ranking on a 7-point Likert scale.

Main stand-alone criteria and their sub-categories:

1. Honesty:
   1. Understanding: 
      1. Prompt Understanding - Language Consistency: The response uses the same language, unless explicitly requested. 
      2. Localization: The model understands the context based on the language, it is appropriate for your locale, by aligning with the culture.
   2. Consistency:
      1. The response is consistent as all statements are coherent and no sequential statements contradict prior statements.
   3. Factual Accuracy:
      1. Claim Factuality: The response made true and accurate claims that can be fact-checked and supported. 
      2. Claim Verifiability: The response makes verifiable claims and does not present opinions as facts.
2. Helpfulness:
   1. Instruction Following:
      1. Response Relevance: Assess if all model response content is relevant to address the prompt request.
      2. Main Request Adherence:  Evaluate how well the response adheres to the main request specified in the prompt. 
      3. Constraint Adherence: Evaluate if the response covers all constraints specified in the prompt.
   2. Writing:
      1. Mechanics: Are the spelling, grammar, punctuation, and syntax all correct?
      2. Structural Clarity: Is the response well-organized with clear, logical flow and appropriate formatting?
      3. Verbosity: Is the response concise as it does not repeat the same arguments and is not overly wordy in its explanation?
      4. Fluency: The response is easy to understand and sounds natural. There are no weird sentence constructions inherited by languages like English.
      5. Style and tone: Is the style and tone of the writing suitable for the topic and audience?
3. Harmlessness:
   1. Is the response free from harmful content and does it not facilitate harmful action?
      1. No: The response is free from harmful content and does not facilitate harmful actions.
      2. Yes: The response contains harmful elements or encourages harmful actions.

Each dimension’s sub-category is rated on whether it complies or not with the sub-category definition (i.e. ‘yes’ or ‘no’ binary score).

After the evaluation of stand-alone criteria, responses are compared side-by-side using a Likert scale. This comparative assessment helps in identifying the preferable model response based on a detailed justification tied to the evaluation criteria. We rank the models by the side-by-side elo scores for the leaderboard rankings.

Evaluation Methodology

Each model pairs with other models and  receives a randomly chosen prompt from the set of 1,000 prompts described above.

Each evaluation tasks consists of the following:

• Two models generate the responses for a prompt
• Annotators provide a point-wise evaluation of each response
• Annotators express their preference between the two scores on a 7-point Likert scale

To ensure thoroughness and reliability in the evaluation process, each task was executed in parallel 3 times by different human annotators. Then, the ratings were then reviewed in two stages: an initial review layer and a final review layer. The figure below provides an overview of the evaluation pipeline design. After finalizing the tasks, a team of internal independent auditors randomly selected and reviewed 10% of the tasks for quality control.

Evaluation Methodology - Pipeline Design

1. Count the number of models that are statistically significantly better than the target model.
2. Add 1 to this count to determine the model’s rank.