ALTK-Evolve reduces agent tokens compared with ACE

An AI agent can know the right APIs and still fail at a simple task. It might paginate a response incorrectly, choose the wrong user, or return data nobody asked for. The problem is not always a lack of knowledge: often, it is a lack of practical experience with how to use each tool.

Two systems seek to solve this problem using the agent’s own history: ACE, short for Agentic Context Engineering, and ALTK-Evolve, a proposal from IBM Research. Both turn previous mistakes into reusable lessons without modifying the model’s weights or relying on human labels.

The decisive difference lies in how they deliver those lessons to the agent. And that leads to a very concrete consequence: the number of tokens consumed during inference.

Two ways to build memory for an agent

Imagine an agent responsible for reconciling an order across nine simulated applications. To complete the task, it must search for information, correctly identify a person, call several APIs, and verify the final result. A single procedural error can ruin the entire operation.

ACE and ALTK-Evolve store the lessons that emerge from these journeys. One lesson might be: “before updating an order, confirm that the identifier belongs to the customer and not the order.” The next time, the agent can reuse that experience.

The two systems agree on something important: it is not a good idea to reduce every experience to a brief, generic summary. A rule discovered across five tasks has different support from one observed only once. If both are mixed into a compact sentence, useful information is lost.

An agent’s memory should not become an overly short list of tips. It should preserve what happened, what worked, and how much evidence supports each lesson.

ACE’s approach

ACE organizes lessons in a single evolving manual or playbook. This manual is updated through a cycle with three components: Generator, which proposes lessons; Reflector, which analyzes the results; and Curator, which organizes and cleans up the content.

The system also assigns favorable and unfavorable outcome counters to each item. This allows the model to distinguish between a recommendation that usually helps and one that has caused failures several times.

ACE identifies two risks of summarizing memory too aggressively. The first is brevity bias, a tendency to produce short, generic, and not very useful instructions. The second is context collapse, which occurs when the model rewrites all of its context at every step and ends up removing important details.

ALTK-Evolve separates memory from delivery

ALTK-Evolve reaches a similar conclusion, but organizes memory as a set of individual guides that can be retrieved according to the task. Each guide keeps a support counter: the number of independent episodes that helped discover it.

Its consolidation process groups similar lessons using embeddings, a way of representing texts as vectors so their meaning can be compared. When several guides are merged, the surviving guide retains the sum of their support counts.

In addition, ALTK-Evolve classifies lessons into categories such as strategy, recovery, and optimization. It also maintains the provenance of each learning, meaning the original trajectory from which it emerged. This can make it easier to transfer a solution discovered in one application to another with a similar structure.

The main difference appears during inference. ACE injects the full manual at every step of the agent. ALTK-Evolve, by contrast, adjusts what it delivers to the model and the task:

• It keeps a small core of highly supported guides.
• It adds selected lessons based on their similarity to the task.
• It can use selection based on a language model or on calculated priorities.
• It delivers the complete set only when the model has enough capacity to process it.

The idea is simple: having more memory available does not mean you should always show all of it. What good is an entire library if the agent cannot quickly find the book it needs?

Results in AppWorld: fewer tokens, similar or better performance

The researchers compared both systems in AppWorld, a benchmark with 168 simulated tasks that require interacting with applications through a ReAct agent. At each step, the agent writes Python code and receives a response from the environment.

The main metrics are TGC, which measures whether the task objective is completed, and SGC, which requires overcoming every variant of a scenario. The results correspond to single runs, known as pass@1, using memory data extracted from training and development.

With DeepSeek-V3.2, ALTK-Evolve achieved better results and used fewer tokens per task:

With this model, ALTK-Evolve reached a TGC of 89.3 versus ACE’s 80.4 and an SGC of 80.4 versus 73.2. At the same time, it consumed about 40% of the tokens used by ACE.

With gpt-oss-120b, the accuracy difference was smaller, but the token savings were even greater:

Here, ALTK-Evolve slightly outperformed ACE in TGC and SGC, but used approximately one-seventh of ACE’s tokens. In practical terms, that can translate into lower inference costs and lower latency, especially when an agent runs thousands of tasks.

Difficulty changes the outcome

The results by difficulty level show why selective delivery can matter. With gpt-oss-120b, ACE performed better on easy and medium tasks, where a complete manual can provide useful general instructions. However, ALTK-Evolve won on difficult tasks, which usually require finding and applying the right lesson.

With DeepSeek-V3.2, the situation is different. The more capable model can process a broad set of guides without becoming overwhelmed as easily. ALTK-Evolve led in the easy, difficult, and overall categories, while ACE held a small advantage in the medium category.

This points to a relevant conclusion for people designing agents: the ideal amount of context depends on the model’s capabilities. A powerful model can take advantage of an extensive repository. A more limited one may become confused if it receives too many instructions at once.

The cost of serving memory matters too

ACE primarily optimizes how its context is built. Its manual can grow in an organized way and preserve valuable details. ALTK-Evolve concentrates its advantage in the delivery phase: it retrieves only the guides the agent appears to need for each task.

This is not about claiming that one system is always better than the other. In fact, the results show that the complete manual can help with some tasks and certain models. The point is that sending all of the memory at every step has a cost that does not always produce a proportional improvement.

For a real product, this difference can be critical. An agent for customer service, shopping, or technical support may execute dozens of steps per request. If every step includes hundreds of thousands of unnecessary context tokens, the bill grows quickly.

There is also a methodological caveat. The comparison was conducted with internal ACE runs on the same base models and the same AppWorld splits used for ALTK-Evolve. The paper notes that ACE’s original work used a different model, DeepSeek-V3.1, so this test aims to better control for differences between models and tools.

The broader lesson is not simply “use fewer tokens.” It is more precise: preserve the agent’s experiences without summarizing them too aggressively, but deliver only as much knowledge as the model can actually use. Memory should be rich; the context window should be strategic.

For people building agents, the next challenge will be to automatically calibrate that amount. How many guides can each model process before the context stops helping? ALTK-Evolve suggests that the answer is not fixed, and that choosing what to show can be just as important as learning new lessons.

Original source

https://huggingface.co/blog/ibm-research/altk-evolve-sldd