Find the gap in your task rules
Why now: Task selection is central to interpreting METR’s evaluation; a contrasting pair makes that issue concrete in your own routine.
Prompted by today's lead: AI makes progress but shows judgment weaknesses, METR finds
What you'll need
- Non-sensitive notes describing a routine task and the rules you follow.
Steps
- Choose a routine with written instructions or decision rules.
- Paste those notes into the prompt to generate two hypothetical cases.
- Read both cases and decide what clarification the harder one needs.
The prompt
Using the routine notes below, write two short hypothetical scenarios involving the same task. Label them Straightforward case and Judgment case. In the first, the supplied rules should clearly determine the next action. In the second, expose a conflict or gap in those rules that requires judgment. End each scenario with exactly one decision question. Do not answer the questions, add rules, or introduce outside factual claims. Keep the whole output under 120 words. Use plain text with no Markdown formatting. Notes: [paste your routine notes here]
What you should get
Two labeled hypothetical scenarios, each ending with one decision question; the second exposes a conflict or gap in the supplied rules. No answers, added rules or outside factual claims; under 120 words.
Tested example: Tested on sample input
Input
For my indoor plant shelf, my notes say to water a pot when its soil feels dry, skip watering if water is sitting in its tray, and move a pot away from the window if its leaves look scorched. I want help deciding what to do during my weekly check, but I do not want the assistant to change these rules.
Output
Straightforward case: During your weekly plant-shelf check, one pot’s soil feels dry, its tray has no standing water, and its leaves do not look scorched. What should you do with this pot? Judgment case: During the same weekly check, another pot’s soil feels dry, but water is sitting in its tray; its leaves do not look scorched. The dry-soil rule calls for watering, while the tray-water rule calls for skipping it. Which instruction should guide your action for this pot?
How we tested it: How we checked this: an AI model ran this prompt on the sample input on November 21, 2026, and a separate model call checked the result against what the Move promises.