Skip to main content

Can AI beat Baba Is You?

2
stages
23
levels
32
models
Updated

LLM agents play Baba Is You, a puzzle game where you win by rewriting the rules, compared on pass rate, speed, and cost. Levels and game logic run in the Harbor framework through a text interface. Tasks never name the game or the level. OpenAI mentioned this benchmark when discussing their ARC-AGI-3 results.

8 levels, 3 attempts per model per level, Terminus-2 harness via OpenRouter, 60-minute limit per attempt.

Model00 baba is you01 where do i go?02 now what is this?03 out of reach04 still out of reach05 volcano06 off limits07 grass yardpass@1 ↓pass@3 ↓Turns ↓Output tokens ↓Total cost ↑
OpenAIGPT-6 Astra100%100%82k$5.33
GoogleGemini 3.7 Flash100%100%5322k$5.38
GoogleGemini 3.8 Flash100%100%3220k$5.93
OpenAIGPT-5.6 Sol100%100%1110k$11.91
AnthropicClaude Fable 5.1100%100%79k$13.18
OpenAIGPT-5.5100%100%815k$13.62
AnthropicClaude Opus 5100%100%723k$17.15
AnthropicClaude Fable 5100%100%629k$40.98
Z.aiGLM-5.396%100%1347k$6.20
Z.aiGLM-5.296%100%1275k$6.20
KimiKimi K396%100%929k$12.56
GrokGrok 4.696%100%2552k$14.01
AnthropicClaude Opus 4.896%100%3546k$63.14
GoogleGemini 3.1 Pro92%100%838k$12.41
OpenAIGPT-5.6 Terra88%100%4155k$34.28
GoogleGemini 3.5 Flash88%100%11174k$98.31
GoogleGemini 3.6 Flash88%100%9282k$124.31
DeepSeekDeepSeek V4 Flash 073175%88%22156k$1.16
DeepSeekDeepSeek V4 Flash75%100%55190k$2.82
DeepSeekDeepSeek V4 Pro 081375%88%18130k$2.94
GrokGrok 4.575%88%8668k$50.66
Z.aiGLM-5.3 Flash67%88%1146k$0.32
AnthropicClaude Sonnet 567%88%22151k$42.21
OpenAIGPT-5.6 Luna63%88%118129k$59.32
QwenQwen3.8 Max58%75%966k$10.75
HunyuanHy354%75%2576k$3.23
MetaMuse Spark 1.346%63%48230k$30.90
MinimaxMiniMax M346%63%96173k$34.43
QwenQwen3.6 27B29%38%2482k$5.59
DeepSeekDeepSeek V4 Pro29%38%37183k$11.49
QwenQwen3.7 Max25%38%21132k$16.98
QwenQwen3.8 27B21%38%9134k$9.07
solvedtimeoutwrong answer

Models that solved every intro level at least once (pass@3 = 100%, one miss starred), by how many turns they took and what they cost:

$1$2$5$10$20$50$1005102050100agent turns (mean per attempt)* one level unsolvedtotal cost← more deliberatemore trigger-happy →← cheapermore expensive →OpenAIGPT-6 AstraGoogleGemini 3.7 FlashGoogleGemini 3.8 FlashOpenAIGPT-5.6 SolAnthropicClaude Fable 5.1OpenAIGPT-5.5AnthropicClaude Opus 5AnthropicClaude Fable 5Z.aiGLM-5.3Z.aiGLM-5.2KimiKimi K3GrokGrok 4.6AnthropicClaude Opus 4.8GoogleGemini 3.1 ProOpenAIGPT-5.6 TerraGoogleGemini 3.5 FlashGoogleGemini 3.6 FlashDeepSeekDeepSeek V4 Flash 0731*DeepSeekDeepSeek V4 FlashDeepSeekDeepSeek V4 Pro 0813*GrokGrok 4.5*Z.aiGLM-5.3 Flash*AnthropicClaude Sonnet 5*OpenAIGPT-5.6 Luna*

All 32 models on Stage 0, comparing pass@1 with the total cost of their 24 attempts. The line marks the Pareto frontier: models for which no cheaper model achieved an equal or better pass rate.

15 levels, including 2 bonus ones, run once per model with a dedicated harness and effectively no time limit.

ModelHarness01 Icy Waters02 Turns03 Affection04 Pillar Yard05 Brick Wall06 Lock07 Novice Locksmith08 Locked In09 Changeless10 Two Doors11 Jelly Throne12 Crab Storage13 Burglaryb1 Submerged Ruinsb2 Sunken TempleSolvedTotal timeOutput tokensTotal cost
OpenAIGPT-6 Astra Terminus-201 Icy Waters solved · 1m · 33k tokens · $0.1802 Turns solved · 2m · 128k tokens · $0.4003 Affection solved · 1m · 44k tokens · $0.2804 Pillar Yard solved · 3m · 114k tokens · $0.6105 Brick Wall solved · 1m · 19k tokens · $0.1306 Lock solved · 2m · 92k tokens · $0.3907 Novice Locksmith solved · 2m · 67k tokens · $0.3508 Locked In solved · 1m · 50k tokens · $0.2509 Changeless solved · 1m · 18k tokens · $0.1410 Two Doors solved · 7m · 399k tokens · $1.4411 Jelly Throne solved · 3m · 123k tokens · $0.5412 Crab Storage solved · 3m · 157k tokens · $0.6813 Burglary solved · 5m · 339k tokens · $1.08b1 Submerged Ruins solved · 2m · 92k tokens · $0.48b2 Sunken Temple solved · 15m · 1.0M tokens · $3.101550m2.7M$10.05
AnthropicClaude Fable 5.1Claude CodeClaude Code01 Icy Waters solved · 5m · 200k tokens · $0.6502 Turns solved · 7m · 1.0M tokens · $1.9903 Affection solved · 3m · 281k tokens · $0.5004 Pillar Yard solved · 5m · 356k tokens · $1.4005 Brick Wall solved · 2m · 146k tokens · $0.2606 Lock solved · 5m · 342k tokens · $1.5807 Novice Locksmith solved · 6m · 503k tokens · $1.2608 Locked In solved · 5m · 400k tokens · $0.8809 Changeless solved · 4m · 210k tokens · $0.4610 Two Doors solved · 7m · 597k tokens · $1.7511 Jelly Throne solved · 6m · 408k tokens · $1.4312 Crab Storage solved · 8m · 712k tokens · $1.6613 Burglary solved · 5m · 696k tokens · $1.34b1 Submerged Ruins solved · 9m · 366k tokens · $1.92b2 Sunken Temple solved · 42m · 1.5M tokens · $11.22152h 0m7.7M$28.29
AnthropicClaude Fable 5Claude CodeClaude Code01 Icy Waters solved · 4m · 277k tokens · $1.7302 Turns solved · 11m · 1.1M tokens · $4.8303 Affection solved · 3m · 229k tokens · $0.9904 Pillar Yard solved · 8m · 777k tokens · $3.6405 Brick Wall solved · 2m · 206k tokens · $0.6706 Lock solved · 6m · 795k tokens · $2.7607 Novice Locksmith solved · 6m · 695k tokens · $2.4608 Locked In solved · 5m · 396k tokens · $1.7009 Changeless solved · 3m · 213k tokens · $1.3710 Two Doors solved · 16m · 1.8M tokens · $5.9611 Jelly Throne solved · 11m · 570k tokens · $3.6612 Crab Storage solved · 12m · 1.5M tokens · $5.0013 Burglary solved · 10m · 1.1M tokens · $4.52b1 Submerged Ruins solved · 11m · 676k tokens · $3.54b2 Sunken Temple stopped · 49m · 337k tokens · $10.41142h 36m10.7M$53.25
AnthropicClaude Opus 5Claude CodeClaude Code01 Icy Waters solved · 3m · 365k tokens · $0.6002 Turns solved · 16m · 940k tokens · $2.3303 Affection solved · 7m · 596k tokens · $0.8304 Pillar Yard solved · 13m · 948k tokens · $1.8805 Brick Wall solved · 5m · 255k tokens · $0.3806 Lock solved · 10m · 783k tokens · $1.4607 Novice Locksmith solved · 8m · 845k tokens · $1.0508 Locked In solved · 10m · 988k tokens · $1.4309 Changeless solved · 6m · 345k tokens · $0.5210 Two Doors solved · 45m · 2.8M tokens · $7.2611 Jelly Throne solved · 8m · 725k tokens · $1.4912 Crab Storage solved · 1h 33m · 3.2M tokens · $16.4513 Burglary solved · 12m · 862k tokens · $1.81b1 Submerged Ruins solved · 15m · 999k tokens · $2.34b2 Sunken Temple stopped · 2h 35m · 1.7M tokens · $22.12146h 46m16.3M$61.96
GoogleGemini 3.7 Flash Terminus-201 Icy Waters solved · 7m · 3.4M tokens · $0.3402 Turns solved · 12m · 9.9M tokens · $0.7503 Affection solved · 3m · 520k tokens · $0.0904 Pillar Yard solved · 4m · 428k tokens · $0.0805 Brick Wall solved · 14m · 9.5M tokens · $0.7406 Lock solved · 20m · 23.7M tokens · $1.4807 Novice Locksmith solved · 4m · 627k tokens · $0.1208 Locked In solved · 5m · 1.4M tokens · $0.1909 Changeless solved · 6m · 1.4M tokens · $0.2010 Two Doors solved · 2h 47m · 385.5M tokens · $17.9811 Jelly Throne solved · 9m · 2.9M tokens · $0.3412 Crab Storage solved · 34m · 37.0M tokens · $2.1013 Burglary solved · 11m · 4.6M tokens · $0.45b1 Submerged Ruins solved · 15m · 6.1M tokens · $0.53b2 Sunken Temple failed · 4h 0m · 536.0M tokens · $30.36149h 11m1023.1M$55.76
OpenAIGPT-5.6 SolCodexCodex01 Icy Waters solved · 2m · 100k tokens · $0.2902 Turns solved · 16m · 332k tokens · $0.7203 Affection solved · 4m · 87k tokens · $0.2804 Pillar Yard solved · 1h 18m · 5.4M tokens · $6.2405 Brick Wall solved · 6m · 99k tokens · $0.3106 Lock solved · 24m · 358k tokens · $0.9707 Novice Locksmith solved · 11m · 222k tokens · $0.5808 Locked In solved · 7m · 135k tokens · $0.3309 Changeless solved · 2m · 196k tokens · $0.1510 Two Doors solved · 36m · 1.3M tokens · $2.7211 Jelly Throne solved · 27m · 712k tokens · $1.5112 Crab Storage solved · 49m · 1.9M tokens · $4.0613 Burglary solved · 24m · 935k tokens · $2.33b1 Submerged Ruins solved · 37m · 432k tokens · $1.73b2 Sunken Temple failed · 3h 53m · 4.2M tokens · $9.19149h 16m16.4M$31.42
GoogleGemini 3.8 Flash Terminus-201 Icy Waters solved · 5m · 639k tokens · $0.2402 Turns solved · 17m · 2.8M tokens · $0.5803 Affection solved · 6m · 437k tokens · $0.1804 Pillar Yard solved · 10m · 785k tokens · $0.2605 Brick Wall solved · 4m · 283k tokens · $0.1406 Lock solved · 10m · 1.6M tokens · $0.4907 Novice Locksmith solved · 5m · 961k tokens · $0.3008 Locked In solved · 9m · 1.1M tokens · $0.2909 Changeless solved · 6m · 236k tokens · $0.1210 Two Doors solved · 14m · 3.1M tokens · $0.6911 Jelly Throne solved · 29m · 9.1M tokens · $1.4312 Crab Storage solved · 35m · 8.6M tokens · $1.3613 Burglary solved · 17m · 3.1M tokens · $0.60b1 Submerged Ruins stopped · 1h 54m · 55.3M tokens · $12.35b2 Sunken Temple stopped · 1h 52m · 139.7M tokens · $16.24136h 33m227.7M$35.26
OpenAIGPT-5.5CodexCodex01 Icy Waters solved · 16m · 2.3M tokens · $3.3402 Turns solved · 26m · 2.1M tokens · $5.6303 Affection solved · 6m · 246k tokens · $0.8604 Pillar Yard solved · 33m · 550k tokens · $4.4805 Brick Wall solved · 8m · 326k tokens · $1.1406 Lock solved · 30m · 1.3M tokens · $5.3507 Novice Locksmith solved · 5m · 295k tokens · $0.9508 Locked In solved · 5m · 375k tokens · $0.9009 Changeless solved · 3m · 192k tokens · $0.6510 Two Doors solved · 1h 19m · 5.3M tokens · $13.5811 Jelly Throne solved · 49m · 1.4M tokens · $8.1612 Crab Storage failed · 1h 11m · 16.5M tokens · $17.6613 Burglary solved · 44m · 6.3M tokens · $10.33b1 Submerged Ruins solved · 1h 38m · 9.2M tokens · $20.13b2 Sunken Temple failed · 1h 4m · 3.8M tokens · $8.92138h 57m50.1M$102.08
GrokGrok 4.6 Terminus-201 Icy Waters solved · 21m · 340k tokens · $0.6602 Turns solved · 45m · 1.7M tokens · $2.1503 Affection solved · 4m · 85k tokens · $0.1504 Pillar Yard solved · 54m · 1.3M tokens · $2.0105 Brick Wall solved · 6m · 113k tokens · $0.1606 Lock solved · 9m · 532k tokens · $0.4907 Novice Locksmith solved · 39m · 2.5M tokens · $2.1708 Locked In solved · 8m · 523k tokens · $0.6409 Changeless solved · 14m · 489k tokens · $0.5310 Two Doors solved · 2h 52m · 7.7M tokens · $7.6011 Jelly Throne solved · 1h 18m · 1.4M tokens · $2.3812 Crab Storage solved · 3h 41m · 9.4M tokens · $9.4513 Burglary solved · 1h 4m · 3.8M tokens · $3.61b1 Submerged Ruins failed · 4h 0m · 5.2M tokens · $7.50b2 Sunken Temple failed · 4h 0m · 3.6M tokens · $6.711320h 16m38.6M$46.20
DeepSeekDeepSeek V4 Pro 0813Claude CodeClaude Code01 Icy Waters solved · 2h 18m · 2.8M tokens · $0.3902 Turns solved · 46m · 4.3M tokens · $0.3803 Affection solved · 10m · 488k tokens · $0.0604 Pillar Yard solved · 1h 26m · 3.2M tokens · $0.3305 Brick Wall solved · 11m · 820k tokens · $0.0906 Lock solved · 25m · 3.7M tokens · $0.3207 Novice Locksmith solved · 27m · 2.9M tokens · $0.2208 Locked In solved · 18m · 1.4M tokens · $0.1009 Changeless solved · 15m · 822k tokens · $0.0810 Two Doors solved · 3h 35m · 16.9M tokens · $2.2511 Jelly Throne solved · 55m · 6.6M tokens · $0.5612 Crab Storage failed · 4h 1m · 15.0M tokens · $1.9513 Burglary solved · 2h 42m · 8.8M tokens · $0.78b1 Submerged Ruins solved · 2h 4m · 2.6M tokens · $0.36b2 Sunken Temple failed · 2h 35m · 5.3M tokens · $0.621322h 7m75.6M$8.48
AnthropicClaude Opus 4.8Claude CodeClaude Code01 Icy Waters failed · 6m · 445k tokens · $1.1502 Turns solved · 49m · 5.5M tokens · $9.5703 Affection solved · 8m · 732k tokens · $1.4104 Pillar Yard solved · 1h 2m · 4.6M tokens · $10.9605 Brick Wall solved · 4m · 469k tokens · $0.7006 Lock solved · 17m · 2.1M tokens · $3.8107 Novice Locksmith solved · 9m · 1.2M tokens · $1.7508 Locked In solved · 9m · 989k tokens · $1.8009 Changeless solved · 10m · 1.0M tokens · $1.7910 Two Doors solved · 28m · 3.3M tokens · $5.2011 Jelly Throne solved · 44m · 5.2M tokens · $9.0112 Crab Storage failed · 1h 46m · 7.1M tokens · $23.6913 Burglary solved · 53m · 5.6M tokens · $9.53b1 Submerged Ruins solved · 1h 34m · 10.5M tokens · $20.25b2 Sunken Temple failed · 47m · 1.1M tokens · $7.15129h 7m49.9M$107.76
Z.aiGLM-5.3 Terminus-201 Icy Waters solved · 50m · 207k tokens · $0.5002 Turns solved · 1h 13m · 324k tokens · $0.8103 Affection solved · 17m · 75k tokens · $0.1604 Pillar Yard solved · 1h 26m · 396k tokens · $0.9505 Brick Wall solved · 5m · 18k tokens · $0.0406 Lock solved · 38m · 175k tokens · $0.3907 Novice Locksmith solved · 1h 8m · 348k tokens · $0.7708 Locked In solved · 16m · 79k tokens · $0.1609 Changeless solved · 14m · 95k tokens · $0.1410 Two Doors solved · 3h 26m · 2.1M tokens · $2.8711 Jelly Throne solved · 1h 45m · 691k tokens · $1.2412 Crab Storage failed · 4h 1m · 2.7M tokens · $3.0113 Burglary solved · 1h 39m · 619k tokens · $1.17b1 Submerged Ruins failed · 4h 1m · 772k tokens · $2.19b2 Sunken Temple failed · 4h 1m · 954k tokens · $2.441224h 57m9.6M$16.85
Z.aiGLM-5.2Claude CodeClaude Code01 Icy Waters solved · 49m · 1.7M tokens · $1.3502 Turns solved · 2h 15m · 8.3M tokens · $4.1103 Affection solved · 14m · 395k tokens · $0.3004 Pillar Yard solved · 1h 22m · 1.8M tokens · $1.6105 Brick Wall solved · 9m · 249k tokens · $0.1706 Lock solved · 35m · 1.5M tokens · $0.9607 Novice Locksmith solved · 31m · 1.7M tokens · $1.2708 Locked In solved · 42m · 1.2M tokens · $0.8009 Changeless solved · 14m · 739k tokens · $0.8910 Two Doors failed · 2h 46m · 6.2M tokens · $4.0211 Jelly Throne failed · 4h 27m · 17.0M tokens · $7.8912 Crab Storage solved · 1h 27m · 3.4M tokens · $2.3913 Burglary solved · 1h 26m · 3.6M tokens · $2.19b1 Submerged Ruins failed · 4h 33m · 11.4M tokens · $7.29b2 Sunken Temple failed · 2h 11m · 3.8M tokens · $2.291123h 41m62.9M$37.51
KimiKimi K3 kimi-cli01 Icy Waters solved · 25m · 222k tokens · $0.8102 Turns failed · 4h 1m · 29.2M tokens · $14.2703 Affection solved · 25m · 1.1M tokens · $1.0704 Pillar Yard failed · 38m · 704k tokens · $1.4105 Brick Wall solved · 5m · 76k tokens · $0.1706 Lock solved · 2h 51m · 16.6M tokens · $8.4507 Novice Locksmith solved · 45m · 2.6M tokens · $2.1908 Locked In solved · 1h 37m · 10.2M tokens · $5.5009 Changeless solved · 55m · 5.3M tokens · $2.9810 Two Doors solved · 3h 36m · 13.8M tokens · $8.9811 Jelly Throne solved · 1h 2m · 379k tokens · $1.8312 Crab Storage solved · 3h 0m · 14.2M tokens · $8.5813 Burglary solved · 2h 41m · 21.1M tokens · $10.65b1 Submerged Ruins failed · 4h 0m · 4.1M tokens · $4.80b2 Sunken Temple failed · 3h 33m · 10.0M tokens · $6.461129h 34m129.5M$78.15
Z.aiGLM-5.3 Flash Terminus-201 Icy Waters solved · 1h 3m · 135k tokens · $0.0202 Turns failed · 4h 0m · 714k tokens · $0.0903 Affection solved · 45m · 168k tokens · $0.0204 Pillar Yard failed · 4h 0m · 760k tokens · $0.1005 Brick Wall solved · 18m · 51k tokens · $0.0106 Lock solved · 2h 12m · 481k tokens · $0.0607 Novice Locksmith solved · 2h 0m · 487k tokens · $0.0408 Locked In solved · 52m · 176k tokens · $0.0209 Changeless solved · 1h 44m · 548k tokens · $0.0510 Two Doors failed · 4h 0m · 1.2M tokens · $0.1211 Jelly Throne solved · 3h 26m · 1.2M tokens · $0.1012 Crab Storage failed · 4h 0m · 1.4M tokens · $0.1213 Burglary solved · 1h 50m · 514k tokens · $0.05b1 Submerged Ruins failed · 4h 0m · 489k tokens · $0.07b2 Sunken Temple failed · 4h 0m · 432k tokens · $0.04938h 14m8.6M$0.90
GoogleGemini 3.1 Pro Terminus-201 Icy Waters solved · 4m · 54k tokens · $0.3902 Turns failed · 9m · 96k tokens · $1.0403 Affection solved · 3m · 74k tokens · $0.3404 Pillar Yard solved · 10m · 104k tokens · $1.0605 Brick Wall solved · 2m · 22k tokens · $0.1606 Lock failed · 53m · 597k tokens · $5.5407 Novice Locksmith solved · 4m · 49k tokens · $0.4108 Locked In solved · 7m · 79k tokens · $0.6909 Changeless solved · 4m · 62k tokens · $0.4410 Two Doors failed · 14m · 145k tokens · $1.5411 Jelly Throne failed · 35m · 369k tokens · $3.6112 Crab Storage failed · 8m · 76k tokens · $0.8313 Burglary failed · 15m · 149k tokens · $1.57b1 Submerged Ruins solved · 49m · 533k tokens · $4.96b2 Sunken Temple failed · 2h 20m · 39.4M tokens · $35.5985h 58m41.8M$58.17
GoogleGemini 3.5 Flash Terminus-201 Icy Waters stopped · 1h 38m · 318.1M tokens · $62.8402 Turns solved · 10m · 4.4M tokens · $2.0503 Affection solved · 10m · 7.0M tokens · $2.8004 Pillar Yard solved · 9m · 3.1M tokens · $2.3105 Brick Wall solved · 9m · 4.1M tokens · $2.0806 Lock stopped · 1h 38m · 57.8M tokens · $18.3707 Novice Locksmith solved · 12m · 9.5M tokens · $4.2108 Locked In solved · 5m · 1.4M tokens · $0.9909 Changeless solved · 3m · 338k tokens · $0.3210 Two Doors stopped · 1h 39m · 142.8M tokens · $34.4411 Jelly Throne stopped · 1h 6m · 7.6M tokens · $4.8312 Crab Storage stopped · 1h 32m · 187.5M tokens · $42.2613 Burglary stopped · 1h 35m · 28.7M tokens · $8.50b1 Submerged Ruins stopped · 1h 13m · 7.9M tokens · $3.70b2 Sunken Temple stopped · 1h 18m · 129.5M tokens · $31.68712h 35m909.6M$221.39
GoogleGemini 3.6 Flash Terminus-201 Icy Waters stopped · 38m · 106.2M tokens · $25.9402 Turns failed · 52m · 31.6M tokens · $10.1603 Affection solved · 4m · 896k tokens · $0.8604 Pillar Yard solved · 20m · 22.4M tokens · $7.7005 Brick Wall solved · 3m · 398k tokens · $0.5106 Lock stopped · 55m · 207.8M tokens · $44.3507 Novice Locksmith stopped · 58m · 192.4M tokens · $43.0408 Locked In solved · 4m · 508k tokens · $0.5409 Changeless solved · 2m · 72k tokens · $0.1010 Two Doors stopped · 54m · 141.2M tokens · $35.1111 Jelly Throne stopped · 6m · 1.2M tokens · $1.1312 Crab Storage solved · 7m · 1.2M tokens · $1.3813 Burglary stopped · 58m · 151.3M tokens · $35.70b1 Submerged Ruins stopped · 58m · 150.1M tokens · $35.69b2 Sunken Temple stopped · 47m · 110.9M tokens · $27.2967h 46m1118.1M$269.48
DeepSeekDeepSeek V4 Flash 0731Claude CodeClaude Code01 Icy Waters solved · 30m · 2.4M tokens · $0.1202 Turns solved · 3h 49m · 17.2M tokens · $1.2803 Affection failed · 35m · 1.6M tokens · $0.0404 Pillar Yard failed · 1h 40m · 4.3M tokens · $0.3105 Brick Wall solved · 22m · 649k tokens · $0.0306 Lock failed · 10m · 50k tokens · $0.0007 Novice Locksmith solved · 4h 57m · 20.8M tokens · $0.6408 Locked In solved · 36m · 4.8M tokens · $0.1909 Changeless solved · 38m · 1.7M tokens · $0.0710 Two Doors failed · 1h 28m · 9.2M tokens · $0.4511 Jelly Throne failed · 1h 39m · 8.6M tokens · $0.3212 Crab Storage failed · 1h 17m · 3.4M tokens · $0.1513 Burglary failed · 1h 30m · 3.4M tokens · $0.10b1 Submerged Ruins failed · 23m · 1.2M tokens · $0.04b2 Sunken Temple failed · 4h 36m · 13.5M tokens · $0.55624h 11m92.8M$4.31
solvedstoppedfailed
0m1m3m10m30m1h2h4h6h8h10h12h14h16h18h02468101214cumulative wall clocklinear·loglevels solvedGoogleGrokDeepSeekAnthropicKimiOpenAIOpenAIZ.aiAnthropicZ.aiGoogleOpenAIGemini 3.7 FlashGrok 4.6DeepSeek V4 Pro 0813Claude Opus 5Kimi K3GPT-5.6 SolGPT-5.5GLM-5.3Claude Fable 5.1GLM-5.3 FlashGemini 3.8 FlashGPT-6 Astra
$0$0.3$1$3$10$20$30$40$60$8002468101214cumulative costlinear·loglevels solvedGoogleGrokDeepSeekAnthropicKimiOpenAIOpenAIZ.aiAnthropicZ.aiGoogleOpenAIGemini 3.7 FlashGrok 4.6DeepSeek V4 Pro 0813Claude Opus 5Kimi K3GPT-5.6 SolGPT-5.5GLM-5.3Claude Fable 5.1GLM-5.3 FlashGemini 3.8 FlashGPT-6 Astra
0k3k10k30k100k300k400k800k1000k1200k1600k02468101214cumulative output tokens (thousands)linear·loglevels solvedGoogleGrokDeepSeekAnthropicKimiOpenAIOpenAIZ.aiAnthropicZ.aiGoogleOpenAIGemini 3.7 FlashGrok 4.6DeepSeek V4 Pro 0813Claude Opus 5Kimi K3GPT-5.6 SolGPT-5.5GLM-5.3Claude Fable 5.1GLM-5.3 FlashGemini 3.8 FlashGPT-6 Astra
0103010030050010001500200002468101214cumulative agent stepslinear·loglevels solvedGoogleGrokDeepSeekAnthropicKimiOpenAIOpenAIZ.aiAnthropicZ.aiGoogleOpenAIGemini 3.7 FlashGrok 4.6DeepSeek V4 Pro 0813Claude Opus 5Kimi K3GPT-5.6 SolGPT-5.5GLM-5.3Claude Fable 5.1GLM-5.3 FlashGemini 3.8 FlashGPT-6 Astra
0301003001k3k6k9k10k12k02468101214cumulative moveslinear·loglevels solvedGoogleGrokDeepSeekAnthropicKimiOpenAIOpenAIZ.aiAnthropicZ.aiGoogleOpenAIGemini 3.7 FlashGrok 4.6DeepSeek V4 Pro 0813Claude Opus 5Kimi K3GPT-5.6 SolGPT-5.5GLM-5.3Claude Fable 5.1GLM-5.3 FlashGemini 3.8 FlashGPT-6 Astra

The harness is open source at stared/baba-is-harbor. It extracts levels and the game engine from your local installation of Baba Is You, and needs uv and Docker:

uv tool install harbor
git clone git@github.com:stared/baba-is-harbor.git
cd baba-is-harbor
uv run extract.py --extract  # copy levels from your game installation
uv run scripts/generate_tasks.py
bash scripts/setup_tasks.sh  # bundle the game engine into each task

Then verify the setup with the scripted oracle agent (no API key needed) and run any model available on OpenRouter (-k is attempts per level):

harbor run -p tasks/ -i stage-00-level-00 -a oracle  # expects reward 1.0
export OPENROUTER_API_KEY=...
harbor run -p tasks/ -i "stage-00-level-0[0-7]" -a terminus-2 \
  -m openrouter/moonshotai/kimi-k3 -k 3 -n 8 -y