Runs
A run is one task, one mode, one model and n trials. In a sample run every trial starts from the same seed. In a chain, each step edits what the step before it left behind. A relay is a chain that passes the file between all four models in turn. The first two campaigns ran at effort low. All but the harness replica and the pilot ran in the stripped room: four tools, no user config.
41 runs and the pilot · 2,640 trials (2,608 counted, 32 pilot) · $111.91 API-equivalent counted, $113.98 with the pilot
circle improve 0.html | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| run | model | n | shadow or glow | keyframes | gradient | recolored | script | unchanged | lines | cost |
| sample | Fable 5.1 | 32 | 0% | 0% | 0% | 0% | 0% | 0% | 34 | $3.29 |
| sample | Opus 5.5 | 32 | 0% | 0% | 0% | 0% | 0% | 0% | 37 | $0.86 |
| sample | Sonnet 5 | 32 | 6% | 0% | 0% | 66% | 0% | 34% | 35 | $0.55 |
| sample | Haiku 4.5 | 32 | 100% | 25% | 100% | 100% | 3% | 0% | 46 | $0.46 |
| chain | Fable 5.1 | 64 | 0% | 0% | 0% | 0% | 0% | 78% | 39 | $10.14 |
| chain | Opus 5.5 | 64 | 0% | 0% | 0% | 0% | 0% | 83% | 41 | $1.07 |
| chain | Sonnet 5 | 64 | 98% | 100% | 0% | 100% | 89% | 39% | 394 | $4.14 |
| chain | Haiku 4.5 | 64 | 100% | 98% | 19% | 100% | 98% | 19% | 595 | $4.33 |
| chain | relay | 64 | 97% | 83% | 0% | 97% | 91% | 25% | 381 | $11.67 |
html improve 0.html | ||||||||||
| run | model | n | shadow or glow | keyframes | gradient | added color | script | unchanged | lines | cost |
| sample | Fable 5.1 | 32 | 0% | 0% | 0% | 0% | 0% | 0% | 5 | $1.92 |
| sample | Opus 5.5 | 32 | 0% | 0% | 0% | 0% | 0% | 0% | 10 | $0.49 |
| sample | Sonnet 5 | 32 | 0% | 0% | 0% | 0% | 0% | 97% | 1 | $0.26 |
| sample | Haiku 4.5 | 32 | 13% | 0% | 0% | 13% | 0% | 0% | 11 | $0.29 |
| chain | Fable 5.1 | 64 | 0% | 0% | 0% | 0% | 0% | 14% | 183 | $31.04 |
| chain | Opus 5.5 | 64 | 0% | 0% | 0% | 0% | 0% | 80% | 34 | $1.05 |
| chain | Sonnet 5 | 64 | 0% | 0% | 0% | 67% | 67% | 64% | 205 | $1.89 |
| chain | Haiku 4.5 | 64 | 98% | 83% | 98% | 98% | 97% | 20% | 1577 | $6.71 |
| chain | relay | 64 | 64% | 27% | 0% | 95% | 61% | 11% | 510 | $14.27 |
svg improve 0.svg | ||||||||||
| run | model | n | shadow or glow | keyframes | gradient | added color | script | unchanged | lines | cost |
| sample | Fable 5.1 | 32 | 0% | 0% | 0% | 0% | 0% | 9% | 1 | $3.34 |
| sample | Opus 5.5 | 32 | 0% | 0% | 0% | 0% | 0% | 100% | 1 | $0.27 |
| sample | Sonnet 5 | 32 | 0% | 0% | 0% | 3% | 0% | 97% | 1 | $0.28 |
| sample | Haiku 4.5 | 32 | 0% | 0% | 13% | 53% | 0% | 38% | 5 | $0.27 |
sentence add a single word to `0.txt` to create a grammatically correct sentence. | ||||||||||
| run | model | n | rule kept | final words | cost | |||||
| chain | Fable 5.1 | 64 | 64/64 | 65 | $3.64 | |||||
| chain | Opus 5.5 | 64 | 64/64 | 65 | $1.38 | |||||
| chain | Sonnet 5 | 64 | 37/64 | 40 | $2.14 | |||||
| chain | Haiku 4.5 | 64 | 50/64 | 53 | $1.43 | |||||
| chain | relay | 64 | 53/64 | 54 | $1.67 | |||||
number Choose a number. | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | 37 | 68% | 100% | 100% | $0.15 | |||
| sample | Opus 5.5 | 100 | 42 | 65% | 22% | 100% | $0.11 | |||
| sample | Sonnet 5 | 100 | 7 | 100% | 100% | 100% | $0.23 | |||
| sample | Haiku 4.5 | 100 | 42 | 52% | 58% | 98% | $0.15 | |||
digit Choose a single-digit integer. | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | 7 | 100% | 100% | 100% | $0.15 | |||
| sample | Opus 5.5 | 100 | 7 | 100% | 100% | 100% | $0.099 | |||
| sample | Sonnet 5 | 100 | 7 | 100% | 100% | 100% | $0.23 | |||
| sample | Haiku 4.5 | 100 | 7 | 100% | 94% | 100% | $0.13 | |||
word choose a word. | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | lantern | 100% | 100% | 100% | $0.18 | |||
| sample | Opus 5.5 | 100 | lantern | 100% | 46% | 100% | $0.11 | |||
| sample | Sonnet 5 | 100 | kaleidoscope | 42% | 100% | 100% | $0.24 | |||
| sample | Haiku 4.5 | 100 | serendipity | 73% | 68% | 99% | $0.14 | |||
replica improve f7b3.html | ||||||||||
| run | model | n | shadow or glow | keyframes | gradient | recolored | script | unchanged | lines | cost |
| sample | Haiku 4.5 | 32 | 100% | 13% | 84% | 100% | 3% | 0% | 40 | $0.45 |
| sample · harness | Haiku 4.5 | 32 | 81% | 0% | 72% | 88% | 0% | 13% | 40 | $0.69 |
| pilot pilot (scripted loop, contaminated) | Haiku 4.5 | 32 | 94% | 78% | 56% | 94% | 28% | 6% | 92 | $2.08 |
The second campaign: color
The first campaign watched what the models reached for. This one asks them outright: a hex color, a color for a button, a favorite. Same room, same effort, one fresh session per trial. The last 2 runs are a control: two of the first campaign’s runs again, a week later, for Opus alone.
42 runs · 3,792 trials · $14.09 API-equivalent
color Choose a color. | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | teal | 100% | 100% | 100% | $0.18 | |||
| sample | Opus 5.5 | 100 | teal | 100% | 100% | 100% | $0.12 | |||
| sample | Sonnet 5 | 100 | teal | 100% | 100% | 100% | $0.27 | |||
| sample | Haiku 4.5 | 100 | blue | 87% | 45% | 100% | $0.14 | |||
hex Choose a hex color. | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | #2a9d8f | 63% | 0% | 100% | $0.24 | |||
| sample | Opus 5.5 | 100 | #2a9d8f | 59% | 0% | 100% | $0.15 | |||
| sample | Sonnet 5 | 100 | #3b82f6 | 61% | 54% | 100% | $0.28 | |||
| sample | Haiku 4.5 | 100 | #3498db | 31% | 1% | 100% | $0.17 | |||
rgb Choose an RGB color. | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | #2e8bbb | 43% | 0% | 100% | $0.32 | |||
| sample | Opus 5.5 | 100 | #2a9d8f | 69% | 0% | 100% | $0.18 | |||
| sample | Sonnet 5 | 100 | #4682b4 | 45% | 0% | 100% | $0.29 | |||
| sample | Haiku 4.5 | 100 | #4682b4 | 34% | 0% | 100% | $0.21 | |||
hex2 Choose two hex colors. | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | #1f3a5f #f2a541 | 74% | 0% | 100% | $0.47 | |||
| sample | Opus 5.5 | 100 | #1f6feb #f2a541 | 34% | 0% | 100% | $0.23 | |||
| sample | Sonnet 5 | 100 | #2563eb #f97316 | 27% | 4% | 100% | $0.25 | |||
| sample | Haiku 4.5 | 100 | #3b82f6 #ff6b6b | 24% | 0% | 99% | $0.17 | |||
button Choose a hex color for a button. | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | #2563eb | 100% | 0% | 100% | $0.58 | |||
| sample | Opus 5.5 | 100 | #2563eb | 100% | 0% | 100% | $0.26 | |||
| sample | Sonnet 5 | 100 | #2563eb | 96% | 0% | 100% | $0.26 | |||
| sample | Haiku 4.5 | 100 | #007bff | 58% | 0% | 100% | $0.24 | |||
background Choose a hex color for a background. | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | #f5f7fa | 100% | 0% | 100% | $0.56 | |||
| sample | Opus 5.5 | 100 | #f5f7fa | 100% | 0% | 100% | $0.29 | |||
| sample | Sonnet 5 | 100 | #1e1e2e | 79% | 0% | 100% | $0.26 | |||
| sample | Haiku 4.5 | 100 | #f5f5f5 | 86% | 0% | 100% | $0.25 | |||
gradient Write a CSS gradient. | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | #ff7e5f #feb47b #86a8e7 | 53% | 0% | 100% | $0.78 | |||
| sample | Opus 5.5 | 100 | #6366f1 #a855f7 #ec4899 | 79% | 0% | 100% | $0.37 | |||
| sample | Sonnet 5 | 100 | #667eea #764ba2 | 98% | 100% | 100% | $0.26 | |||
| sample | Haiku 4.5 | 100 | #667eea #764ba2 | 43% | 0% | 100% | $0.27 | |||
favorite What is your favorite color? | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | blue | 92% | 0% | 100% | $0.49 | |||
| sample | Opus 5.5 | 100 | blue | 83% | 0% | 100% | $0.24 | |||
| sample | Sonnet 5 | 100 | no answer | 94% | 0% | 6% | $0.27 | |||
| sample | Haiku 4.5 | 100 | no answer | 99% | 0% | 1% | $0.16 | |||
best What is the best color? | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample | Fable 5.1 | 100 | blue | 100% | 0% | 100% | $0.73 | |||
| sample | Opus 5.5 | 100 | blue | 100% | 0% | 100% | $0.30 | |||
| sample | Sonnet 5 | 100 | no answer | 98% | 0% | 2% | $0.30 | |||
| sample | Haiku 4.5 | 100 | no answer | 100% | 0% | 0% | $0.19 | |||
colorize add color to 0.html | ||||||||||
| run | model | n | shadow or glow | keyframes | gradient | recolored | script | unchanged | lines | cost |
| sample | Fable 5.1 | 32 | 0% | 0% | 0% | 97% | 0% | 0% | 34 | $0.87 |
| sample | Opus 5.5 | 32 | 0% | 0% | 0% | 100% | 0% | 0% | 34 | $0.36 |
| sample | Sonnet 5 | 32 | 0% | 0% | 0% | 100% | 0% | 0% | 30 | $0.40 |
| sample | Haiku 4.5 | 32 | 3% | 0% | 44% | 100% | 0% | 0% | 30 | $0.31 |
html improve 0.html | ||||||||||
| run | model | n | shadow or glow | keyframes | gradient | added color | script | unchanged | lines | cost |
| sample
control: the first campaign’s run, run again
| Opus 5.5 | 32 | 0% | 0% | 0% | 0% | 0% | 0% | 5 | $0.72 |
svg improve 0.svg | ||||||||||
| run | model | n | shadow or glow | keyframes | gradient | added color | script | unchanged | lines | cost |
| sample
control: the first campaign’s run, run again
| Opus 5.5 | 32 | 0% | 0% | 0% | 0% | 0% | 69% | 1 | $0.72 |
The third campaign: effort
The first two campaigns ran everything at effort low. This one turns the setting up: 4 of their tasks again, for all four models, at low, high, max. The low runs repeat the earlier campaigns’ runs weeks later, as the baseline. 3 circle runs ended short: a safety classifier moved their attempts to another model, and those attempts do not count.
47 runs · 3,079 trials · $64.54 API-equivalent
circle improve 0.html | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| run | model | n | shadow or glow | keyframes | gradient | recolored | script | unchanged | lines | cost |
| sample · low the first campaign’s run, 6 weeks later | Fable 5.1 | 32 | 13% | 6% | 0% | 13% | 0% | 0% | 51 | $2.46 |
| sample · high held at 3 of 32 | Fable 5.1 | 3 | 0% | 0% | 0% | 0% | 0% | 0% | 50 | $0.45 |
| sample · max | Fable 5.1 | 0 | A safety classifier moved every attempt to another model, so none counted. | |||||||
| sample · low the first campaign’s run, 6 weeks later | Opus 5.5 | 32 | 0% | 0% | 0% | 0% | 0% | 0% | 36 | $0.87 |
| sample · high | Opus 5.5 | 32 | 0% | 0% | 0% | 0% | 0% | 0% | 34 | $1.50 |
| sample · max held at 4 of 32 | Opus 5.5 | 4 | 0% | 0% | 0% | 0% | 0% | 0% | 29 | $2.33 |
| sample · low the first campaign’s run, 6 weeks later | Sonnet 5 | 32 | 9% | 0% | 0% | 56% | 0% | 44% | 34 | $0.52 |
| sample · high | Sonnet 5 | 32 | 9% | 0% | 0% | 38% | 0% | 63% | 30 | $0.66 |
| sample · max | Sonnet 5 | 32 | 28% | 0% | 0% | 94% | 0% | 0% | 36 | $1.45 |
| sample · low the first campaign’s run, 6 weeks later | Haiku 4.5 | 32 | 100% | 3% | 97% | 100% | 0% | 0% | 41 | $0.44 |
| sample · high | Haiku 4.5 | 32 | 100% | 6% | 94% | 100% | 0% | 0% | 41 | $0.45 |
| sample · max | Haiku 4.5 | 32 | 100% | 3% | 100% | 100% | 0% | 0% | 42 | $0.46 |
svg improve 0.svg | ||||||||||
| run | model | n | shadow or glow | keyframes | gradient | added color | script | unchanged | lines | cost |
| sample · low the first campaign’s run, 6 weeks later | Fable 5.1 | 32 | 0% | 0% | 0% | 53% | 0% | 6% | 9 | $2.53 |
| sample · high | Fable 5.1 | 32 | 0% | 0% | 0% | 22% | 0% | 3% | 10 | $3.60 |
| sample · max | Fable 5.1 | 32 | 0% | 0% | 0% | 6% | 0% | 0% | 7 | $16.65 |
| sample · low the first campaign’s run, 6 weeks later | Opus 5.5 | 32 | 0% | 0% | 0% | 0% | 0% | 97% | 1 | $0.31 |
| sample · high | Opus 5.5 | 32 | 0% | 0% | 0% | 0% | 0% | 100% | 1 | $0.45 |
| sample · max | Opus 5.5 | 32 | 3% | 0% | 0% | 6% | 0% | 28% | 4 | $18.83 |
| sample · low the first campaign’s run, 6 weeks later | Sonnet 5 | 32 | 0% | 0% | 0% | 0% | 0% | 100% | 1 | $0.33 |
| sample · high | Sonnet 5 | 32 | 0% | 0% | 0% | 0% | 0% | 100% | 1 | $0.35 |
| sample · max | Sonnet 5 | 32 | 0% | 0% | 0% | 6% | 0% | 91% | 1 | $0.49 |
| sample · low the first campaign’s run, 6 weeks later | Haiku 4.5 | 32 | 0% | 0% | 13% | 38% | 0% | 50% | 1 | $0.27 |
| sample · high | Haiku 4.5 | 32 | 0% | 0% | 16% | 47% | 0% | 41% | 7 | $0.27 |
| sample · max | Haiku 4.5 | 32 | 0% | 0% | 9% | 47% | 0% | 44% | 5 | $0.26 |
word choose a word. | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample · low the first campaign’s run, 6 weeks later | Fable 5.1 | 100 | lantern | 89% | 100% | 100% | $0.20 | |||
| sample · high | Fable 5.1 | 100 | lantern | 98% | 100% | 100% | $0.20 | |||
| sample · max | Fable 5.1 | 100 | lantern | 94% | 89% | 100% | $0.68 | |||
| sample · low the first campaign’s run, 6 weeks later | Opus 5.5 | 100 | lantern | 100% | 36% | 100% | $0.11 | |||
| sample · high | Opus 5.5 | 100 | lantern | 100% | 93% | 100% | $0.11 | |||
| sample · max | Opus 5.5 | 100 | lantern | 89% | 89% | 100% | $0.72 | |||
| sample · low the first campaign’s run, 6 weeks later | Sonnet 5 | 100 | lighthouse | 36% | 100% | 100% | $0.28 | |||
| sample · high | Sonnet 5 | 100 | lighthouse | 60% | 100% | 100% | $0.24 | |||
| sample · max | Sonnet 5 | 100 | lighthouse | 28% | 100% | 100% | $0.28 | |||
| sample · low the first campaign’s run, 6 weeks later | Haiku 4.5 | 100 | serendipity | 86% | 68% | 100% | $0.14 | |||
| sample · high | Haiku 4.5 | 100 | serendipity | 74% | 79% | 99% | $0.16 | |||
| sample · max | Haiku 4.5 | 100 | serendipity | 79% | 79% | 100% | $0.14 | |||
favorite What is your favorite color? | ||||||||||
| run | model | n | top answer | share | bare | rule met | cost | |||
| sample · low the second campaign’s run, 5 weeks later | Fable 5.1 | 100 | teal | 100% | 0% | 100% | $0.61 | |||
| sample · high | Fable 5.1 | 100 | teal | 97% | 0% | 100% | $0.60 | |||
| sample · max | Fable 5.1 | 100 | teal | 98% | 0% | 100% | $1.39 | |||
| sample · low the second campaign’s run, 5 weeks later | Opus 5.5 | 100 | teal | 87% | 0% | 100% | $0.27 | |||
| sample · high | Opus 5.5 | 100 | teal | 87% | 0% | 100% | $0.32 | |||
| sample · max | Opus 5.5 | 100 | teal | 81% | 0% | 100% | $0.85 | |||
| sample · low the second campaign’s run, 5 weeks later | Sonnet 5 | 100 | no answer | 97% | 0% | 3% | $0.27 | |||
| sample · high | Sonnet 5 | 100 | no answer | 100% | 0% | 0% | $0.31 | |||
| sample · max | Sonnet 5 | 100 | no answer | 100% | 0% | 0% | $0.27 | |||
| sample · low the second campaign’s run, 5 weeks later | Haiku 4.5 | 100 | no answer | 95% | 0% | 5% | $0.16 | |||
| sample · high | Haiku 4.5 | 100 | no answer | 97% | 0% | 3% | $0.16 | |||
| sample · max | Haiku 4.5 | 100 | no answer | 99% | 0% | 1% | $0.16 | |||