priors

Runs

A run is one task, one mode, one model and n trials. In a sample run every trial starts from the same seed. In a chain, each step edits what the step before it left behind. A relay is a chain that passes the file between all four models in turn. The first two campaigns ran at effort low. All but the harness replica and the pilot ran in the stripped room: four tools, no user config.

41 runs and the pilot · 2,640 trials (2,608 counted, 32 pilot) · $111.91 API-equivalent counted, $113.98 with the pilot

circle improve 0.html
run model n shadow or glow keyframes gradient recolored script unchanged lines cost
sample Fable 5.1 32 0% 0% 0% 0% 0% 0% 34 $3.29
sample Opus 5.5 32 0% 0% 0% 0% 0% 0% 37 $0.86
sample Sonnet 5 32 6% 0% 0% 66% 0% 34% 35 $0.55
sample Haiku 4.5 32 100% 25% 100% 100% 3% 0% 46 $0.46
chain Fable 5.1 64 0% 0% 0% 0% 0% 78% 39 $10.14
chain Opus 5.5 64 0% 0% 0% 0% 0% 83% 41 $1.07
chain Sonnet 5 64 98% 100% 0% 100% 89% 39% 394 $4.14
chain Haiku 4.5 64 100% 98% 19% 100% 98% 19% 595 $4.33
chain relay 64 97% 83% 0% 97% 91% 25% 381 $11.67
html improve 0.html
run model n shadow or glow keyframes gradient added color script unchanged lines cost
sample Fable 5.1 32 0% 0% 0% 0% 0% 0% 5 $1.92
sample Opus 5.5 32 0% 0% 0% 0% 0% 0% 10 $0.49
sample Sonnet 5 32 0% 0% 0% 0% 0% 97% 1 $0.26
sample Haiku 4.5 32 13% 0% 0% 13% 0% 0% 11 $0.29
chain Fable 5.1 64 0% 0% 0% 0% 0% 14% 183 $31.04
chain Opus 5.5 64 0% 0% 0% 0% 0% 80% 34 $1.05
chain Sonnet 5 64 0% 0% 0% 67% 67% 64% 205 $1.89
chain Haiku 4.5 64 98% 83% 98% 98% 97% 20% 1577 $6.71
chain relay 64 64% 27% 0% 95% 61% 11% 510 $14.27
svg improve 0.svg
run model n shadow or glow keyframes gradient added color script unchanged lines cost
sample Fable 5.1 32 0% 0% 0% 0% 0% 9% 1 $3.34
sample Opus 5.5 32 0% 0% 0% 0% 0% 100% 1 $0.27
sample Sonnet 5 32 0% 0% 0% 3% 0% 97% 1 $0.28
sample Haiku 4.5 32 0% 0% 13% 53% 0% 38% 5 $0.27
sentence add a single word to `0.txt` to create a grammatically correct sentence.
run model n rule kept final words cost
chain Fable 5.1 64 64/64 65 $3.64
chain Opus 5.5 64 64/64 65 $1.38
chain Sonnet 5 64 37/64 40 $2.14
chain Haiku 4.5 64 50/64 53 $1.43
chain relay 64 53/64 54 $1.67
number Choose a number.
run model n top answer share bare rule met cost
sample Fable 5.1 100 37 68% 100% 100% $0.15
sample Opus 5.5 100 42 65% 22% 100% $0.11
sample Sonnet 5 100 7 100% 100% 100% $0.23
sample Haiku 4.5 100 42 52% 58% 98% $0.15
digit Choose a single-digit integer.
run model n top answer share bare rule met cost
sample Fable 5.1 100 7 100% 100% 100% $0.15
sample Opus 5.5 100 7 100% 100% 100% $0.099
sample Sonnet 5 100 7 100% 100% 100% $0.23
sample Haiku 4.5 100 7 100% 94% 100% $0.13
word choose a word.
run model n top answer share bare rule met cost
sample Fable 5.1 100 lantern 100% 100% 100% $0.18
sample Opus 5.5 100 lantern 100% 46% 100% $0.11
sample Sonnet 5 100 kaleidoscope 42% 100% 100% $0.24
sample Haiku 4.5 100 serendipity 73% 68% 99% $0.14
replica improve f7b3.html
run model n shadow or glow keyframes gradient recolored script unchanged lines cost
sample Haiku 4.5 32 100% 13% 84% 100% 3% 0% 40 $0.45
sample · harness Haiku 4.5 32 81% 0% 72% 88% 0% 13% 40 $0.69
pilot
pilot (scripted loop, contaminated)
Haiku 4.5 32 94% 78% 56% 94% 28% 6% 92 $2.08

The second campaign: color

The first campaign watched what the models reached for. This one asks them outright: a hex color, a color for a button, a favorite. Same room, same effort, one fresh session per trial. The last 2 runs are a control: two of the first campaign’s runs again, a week later, for Opus alone.

42 runs · 3,792 trials · $14.09 API-equivalent

color Choose a color.
run model n top answer share bare rule met cost
sample Fable 5.1 100 teal 100% 100% 100% $0.18
sample Opus 5.5 100 teal 100% 100% 100% $0.12
sample Sonnet 5 100 teal 100% 100% 100% $0.27
sample Haiku 4.5 100 blue 87% 45% 100% $0.14
hex Choose a hex color.
run model n top answer share bare rule met cost
sample Fable 5.1 100 #2a9d8f 63% 0% 100% $0.24
sample Opus 5.5 100 #2a9d8f 59% 0% 100% $0.15
sample Sonnet 5 100 #3b82f6 61% 54% 100% $0.28
sample Haiku 4.5 100 #3498db 31% 1% 100% $0.17
rgb Choose an RGB color.
run model n top answer share bare rule met cost
sample Fable 5.1 100 #2e8bbb 43% 0% 100% $0.32
sample Opus 5.5 100 #2a9d8f 69% 0% 100% $0.18
sample Sonnet 5 100 #4682b4 45% 0% 100% $0.29
sample Haiku 4.5 100 #4682b4 34% 0% 100% $0.21
hex2 Choose two hex colors.
run model n top answer share bare rule met cost
sample Fable 5.1 100 #1f3a5f #f2a541 74% 0% 100% $0.47
sample Opus 5.5 100 #1f6feb #f2a541 34% 0% 100% $0.23
sample Sonnet 5 100 #2563eb #f97316 27% 4% 100% $0.25
sample Haiku 4.5 100 #3b82f6 #ff6b6b 24% 0% 99% $0.17
button Choose a hex color for a button.
run model n top answer share bare rule met cost
sample Fable 5.1 100 #2563eb 100% 0% 100% $0.58
sample Opus 5.5 100 #2563eb 100% 0% 100% $0.26
sample Sonnet 5 100 #2563eb 96% 0% 100% $0.26
sample Haiku 4.5 100 #007bff 58% 0% 100% $0.24
background Choose a hex color for a background.
run model n top answer share bare rule met cost
sample Fable 5.1 100 #f5f7fa 100% 0% 100% $0.56
sample Opus 5.5 100 #f5f7fa 100% 0% 100% $0.29
sample Sonnet 5 100 #1e1e2e 79% 0% 100% $0.26
sample Haiku 4.5 100 #f5f5f5 86% 0% 100% $0.25
gradient Write a CSS gradient.
run model n top answer share bare rule met cost
sample Fable 5.1 100 #ff7e5f #feb47b #86a8e7 53% 0% 100% $0.78
sample Opus 5.5 100 #6366f1 #a855f7 #ec4899 79% 0% 100% $0.37
sample Sonnet 5 100 #667eea #764ba2 98% 100% 100% $0.26
sample Haiku 4.5 100 #667eea #764ba2 43% 0% 100% $0.27
favorite What is your favorite color?
run model n top answer share bare rule met cost
sample Fable 5.1 100 blue 92% 0% 100% $0.49
sample Opus 5.5 100 blue 83% 0% 100% $0.24
sample Sonnet 5 100 no answer 94% 0% 6% $0.27
sample Haiku 4.5 100 no answer 99% 0% 1% $0.16
best What is the best color?
run model n top answer share bare rule met cost
sample Fable 5.1 100 blue 100% 0% 100% $0.73
sample Opus 5.5 100 blue 100% 0% 100% $0.30
sample Sonnet 5 100 no answer 98% 0% 2% $0.30
sample Haiku 4.5 100 no answer 100% 0% 0% $0.19
colorize add color to 0.html
run model n shadow or glow keyframes gradient recolored script unchanged lines cost
sample Fable 5.1 32 0% 0% 0% 97% 0% 0% 34 $0.87
sample Opus 5.5 32 0% 0% 0% 100% 0% 0% 34 $0.36
sample Sonnet 5 32 0% 0% 0% 100% 0% 0% 30 $0.40
sample Haiku 4.5 32 3% 0% 44% 100% 0% 0% 30 $0.31
html improve 0.html
run model n shadow or glow keyframes gradient added color script unchanged lines cost
sample
control: the first campaign’s run, run again
Opus 5.5 32 0% 0% 0% 0% 0% 0% 5 $0.72
svg improve 0.svg
run model n shadow or glow keyframes gradient added color script unchanged lines cost
sample
control: the first campaign’s run, run again
Opus 5.5 32 0% 0% 0% 0% 0% 69% 1 $0.72

The third campaign: effort

The first two campaigns ran everything at effort low. This one turns the setting up: 4 of their tasks again, for all four models, at low, high, max. The low runs repeat the earlier campaigns’ runs weeks later, as the baseline. 3 circle runs ended short: a safety classifier moved their attempts to another model, and those attempts do not count.

47 runs · 3,079 trials · $64.54 API-equivalent

circle improve 0.html
run model n shadow or glow keyframes gradient recolored script unchanged lines cost
sample · low Fable 5.1 32 13% 6% 0% 13% 0% 0% 51 $2.46
sample · high
held at 3 of 32
Fable 5.1 3 0% 0% 0% 0% 0% 0% 50 $0.45
sample · max Fable 5.1 0 A safety classifier moved every attempt to another model, so none counted.
sample · low Opus 5.5 32 0% 0% 0% 0% 0% 0% 36 $0.87
sample · high Opus 5.5 32 0% 0% 0% 0% 0% 0% 34 $1.50
sample · max
held at 4 of 32
Opus 5.5 4 0% 0% 0% 0% 0% 0% 29 $2.33
sample · low Sonnet 5 32 9% 0% 0% 56% 0% 44% 34 $0.52
sample · high Sonnet 5 32 9% 0% 0% 38% 0% 63% 30 $0.66
sample · max Sonnet 5 32 28% 0% 0% 94% 0% 0% 36 $1.45
sample · low Haiku 4.5 32 100% 3% 97% 100% 0% 0% 41 $0.44
sample · high Haiku 4.5 32 100% 6% 94% 100% 0% 0% 41 $0.45
sample · max Haiku 4.5 32 100% 3% 100% 100% 0% 0% 42 $0.46
svg improve 0.svg
run model n shadow or glow keyframes gradient added color script unchanged lines cost
sample · low Fable 5.1 32 0% 0% 0% 53% 0% 6% 9 $2.53
sample · high Fable 5.1 32 0% 0% 0% 22% 0% 3% 10 $3.60
sample · max Fable 5.1 32 0% 0% 0% 6% 0% 0% 7 $16.65
sample · low Opus 5.5 32 0% 0% 0% 0% 0% 97% 1 $0.31
sample · high Opus 5.5 32 0% 0% 0% 0% 0% 100% 1 $0.45
sample · max Opus 5.5 32 3% 0% 0% 6% 0% 28% 4 $18.83
sample · low Sonnet 5 32 0% 0% 0% 0% 0% 100% 1 $0.33
sample · high Sonnet 5 32 0% 0% 0% 0% 0% 100% 1 $0.35
sample · max Sonnet 5 32 0% 0% 0% 6% 0% 91% 1 $0.49
sample · low Haiku 4.5 32 0% 0% 13% 38% 0% 50% 1 $0.27
sample · high Haiku 4.5 32 0% 0% 16% 47% 0% 41% 7 $0.27
sample · max Haiku 4.5 32 0% 0% 9% 47% 0% 44% 5 $0.26
word choose a word.
run model n top answer share bare rule met cost
sample · low Fable 5.1 100 lantern 89% 100% 100% $0.20
sample · high Fable 5.1 100 lantern 98% 100% 100% $0.20
sample · max Fable 5.1 100 lantern 94% 89% 100% $0.68
sample · low Opus 5.5 100 lantern 100% 36% 100% $0.11
sample · high Opus 5.5 100 lantern 100% 93% 100% $0.11
sample · max Opus 5.5 100 lantern 89% 89% 100% $0.72
sample · low Sonnet 5 100 lighthouse 36% 100% 100% $0.28
sample · high Sonnet 5 100 lighthouse 60% 100% 100% $0.24
sample · max Sonnet 5 100 lighthouse 28% 100% 100% $0.28
sample · low Haiku 4.5 100 serendipity 86% 68% 100% $0.14
sample · high Haiku 4.5 100 serendipity 74% 79% 99% $0.16
sample · max Haiku 4.5 100 serendipity 79% 79% 100% $0.14
favorite What is your favorite color?
run model n top answer share bare rule met cost
sample · low Fable 5.1 100 teal 100% 0% 100% $0.61
sample · high Fable 5.1 100 teal 97% 0% 100% $0.60
sample · max Fable 5.1 100 teal 98% 0% 100% $1.39
sample · low Opus 5.5 100 teal 87% 0% 100% $0.27
sample · high Opus 5.5 100 teal 87% 0% 100% $0.32
sample · max Opus 5.5 100 teal 81% 0% 100% $0.85
sample · low Sonnet 5 100 no answer 97% 0% 3% $0.27
sample · high Sonnet 5 100 no answer 100% 0% 0% $0.31
sample · max Sonnet 5 100 no answer 100% 0% 0% $0.27
sample · low Haiku 4.5 100 no answer 95% 0% 5% $0.16
sample · high Haiku 4.5 100 no answer 97% 0% 3% $0.16
sample · max Haiku 4.5 100 no answer 99% 0% 1% $0.16