TextQuests arrives as a straight-up test: instead of asking facts, it challenges intelligences to navigate complex worlds where memory, planning, and trial-and-error matter as much as knowledge. Does it sound like replaying classic adventure games with an artificial brain that learns on the fly? Well, that's exactly what TextQuests proposes.
What is TextQuests
TextQuests is a benchmark built on 25 classic Infocom games—the old text adventures that could take a person dozens of hours and hundreds of actions to solve. The idea is simple and powerful: use these games as a lab to measure how much sustained reasoning and long-term memory an LLM has when it acts as an agent in an interactive world. (arxiv.org)
How they evaluate models
Evaluation runs in two modes: one with access to the game's official hints and one without. Each attempt can last up to 500 steps and the entire game history is kept without truncation, forcing the model to reason over a context that grows with every action. The key metrics are in-game progress and a measure of "harmful" actions to evaluate ethical behavior inside the environment. (huggingface.co)
What researchers found
Results show clear problems when the context gets very long. In the tests, the history can exceed 100,000 tokens, and many models start confusing past actions, 'hallucinating' that they did things they didn't, or repeating actions in loops instead of combining old information to plan new routes. Concrete examples include trouble navigating mazes or remembering exactly where they left a critical object in games like Zork or Wishbringer. This reveals limits in models' ability to build a stable mental map of a world that changes with every step. (huggingface.co)
Key problem: keeping and using a long story is not the same as knowing a lot of facts. Models fail when they must manage active long-term memory.
They also observed an interesting tension between effectiveness and cost: models that use more tokens for reasoning during execution tend to make more progress, but that improvement has diminishing returns and bumps up against inference cost and latency. In other words, it's not enough to "think more"—you need to think better and more selectively. (huggingface.co)
Why this matters to you?
Because this test makes clear something you probably already see in the real world: an AI that knows a lot doesn't always behave well in situations that require memory, planning, and adaptation. Can you imagine asking an assistant to remember the exact steps to fix your fridge or complete a bank procedure you've been doing for days? If the model forgets what happened ten steps ago, the result can be frustrating or even costly.
Practically speaking, for entrepreneurs and developers this means that if you want a useful agent in real environments (banks, logistics, tech support), you need solutions beyond just increasing model size: design better memory, smarter context summaries, and policies that decide when to use more reasoning and when to save tokens.
How to try it or follow it
The project is published and its code and benchmarks are open, meant for researchers and model builders to contribute and submit results to the leaderboard. If you're curious, you can check the technical paper and the repo to replicate the tests or upload your own agent variants. There's also a contact email for contributions to the leaderboard. (arxiv.org, huggingface.co)
Final reflection
TextQuests isn't nostalgia for text games. It's a magnifying glass on a real limitation of LLMs when you ask them to live and reason inside a world that changes with every step. If we're building assistants that must remember, plan, and correct themselves, these tests teach us that the next round of improvements will have to focus on long-term memory, selective reasoning efficiency, and practical trials in dynamic environments.
Good news? It's a challenge you can experiment with and improve in the open. Feel like building your own agent to beat the leaderboard?