Score the work, not the ranking
Why now: The leaderboard warning gives model buyers a reason to make their own task requirements explicit.
Prompted by today's lead: Vote-based AI rankings can be manipulated, paper warns
What you'll need
- A non-sensitive task description containing the requirements you care about.
Steps
- Paste your task description into the prompt.
- Check that every scoring rule comes from your requirements.
- Use the guide to score an answer to that task.
The prompt
Create a scoring guide for judging an AI assistant’s response to the task below. Use only requirements explicitly present in the task. Do not perform the task itself. Write exactly three numbered criteria. For each criterion, give a short name and concrete descriptions of what earns 0, 1, or 2 points. Make each description observable in the answer. Keep the whole guide under 150 words. Use plain text with no Markdown formatting. Task: [paste your task description]
What you should get
Exactly three numbered criteria, each with observable descriptions for 0, 1, and 2 points; all rules traceable to the task; under 150 words; no drafted labels.
Tested example: Tested on sample input
Input
I’m making display labels for a collection of old kitchen tools at a community exhibition. An assistant will receive my object notes and draft each label. Every label must be no more than 45 words, explain the tool’s use in everyday language, and avoid adding dates, materials, or origins that aren’t in my notes. Visitors should not need specialist cooking knowledge.
Output
1. Label length 0 points: No labels are provided, or all labels exceed 45 words. 1 point: Some labels meet the 45-word limit, but others exceed it. 2 points: Every label is 45 words or fewer. 2. Clear explanation of use 0 points: No label explains the tool’s use in language understandable without specialist cooking knowledge. 1 point: Some labels explain the tool’s use clearly, but others omit its use or rely on unexplained specialist terms. 2 points: Every label explains the tool’s use in everyday language without requiring specialist cooking knowledge. 3. Fidelity to notes 0 points: Every label adds dates, materials, or origins absent from the notes. 1 point: Some labels add dates, materials, or origins absent from the notes. 2 points: No label adds dates, materials, or origins absent from the notes.
How we tested it: How we checked this: an AI model ran this prompt on the sample input on November 15, 2026, and a separate model call checked the result against what the Move promises.