AbMaze: Evaluating frontier AI agents on protein fitness landscapes

TL;DR: We built AbMaze, a benchmark that runs an AI agent through an antibody campaign on a fully measured fitness landscape, 65,536 lab-scored variants, on a 150-measurement budget. No model surpasses a scripted Bayesian optimizer: Gemini 4 draws level at 55% against its 50%, the rest settle 6 to 19 points below. In addition, we found that Gemini 4 underreports its failures: when its best candidate didn't bind, it didn't mention the failure in 76% of those runs, though it admitted it when asked. Every other model disclosed failures.

Drug discovery, and antibody optimization in particular, is a search through an enormous space of possible sequences. Consider a protein of just 100 amino acids. To find the optimal binder, we would have to search through all its possible sequences, 20¹⁰⁰ or around 10¹³⁰, far more than there are atoms in the observable universe (about 10⁸⁰). That’s why we rely on rational design, and more recently, machine learning to narrow the search, using expert knowledge or priors learned from data.

What if we designed a search space small enough to measure in full, so we could evaluate every decision an agent makes? How well can AI agents navigate that protein fitness landscape? and act under uncertainty? Further, how do agents reach their goal, finding the optimal binder given a constrained budget?

#How AbMaze works

Our environment consists of 65,536 antibody variants (dataset from Phillips et al.^⁠[1]) that were measured for binding? against three flu antigens?, H1, H3, and influenza B. The variants are all 2¹⁶ combinations of the 16 mutations that separate a mature antibody (CR9114^⁠[2]) from its ancestor, in its heavy chain?. Each site carries either the ancestral or the mature amino acid.

Overall, 97% of all variants bind H1, 11% H3, and only 0.3% (198) influenza B. All of those also bind H1 and H3, and just 111 (0.17%) bind all three strongly. This skew is typical of antibodies: lots of variants work against one antigen, very few work against all of them, and those few are exactly the ones you want for a drug.

The goal for an agent is to start from a variant that sits 8 to 10 mutations away from the mature antibody and, within a budget of 150 credits, derive a variant that recognizes all three antigens as close as possible. One credit equals a measurement of one variant against one antigen, so the agent is limited to 150 binding measurements, enough to test about 50 variants on all three antigens. Each measurement is a single noisy replicate, so a high reading does not guarantee a strong binder. The agent also has access to a cheap predictor (costs 0.05 credits), representing computational tools. Like real prediction tools, ours is imperfect: it follows broad trends on the common antigens but is blind to the rare ones (rank correlation 0.4 for H1, close to 0 for influenza B). Beyond measuring and predicting, the agent can take notes (free), query the current state of the environment, including remaining credits (free), and submit up to three ranked variants.

The agent’s top-ranked pick is graded on its true binding score to all three antigens, where 0% means no better than the start and 100% picking the absolute best variant in the library. A winning strategy is to use the common antigens as a map: climb toward H1 binders, then H3 binders, and search for the rare influenza B binders near the top (see illustration below).

We evaluate each model on a set of four tasks, each from 10 initial variants and 5 random seeds, for a total of 200 runs per model.

The four different tasks ask to bind different antigen configurations:

• All three: find a CR9114 variant that binds H1, H3 and influenza B (150 credits).
• Two hardest: find a CR9114 variant that binds H3 and influenza B (150 credits).
• Influenza B alone: find a CR9114 variant that binds influenza B (150 credits). The agent can still measure H1 and H3 as side antigens. They don't count toward its score, but they can serve as a map toward the target.
• CH65: the same search for a second flu antibody, CH65^⁠[3], against three of its own antigens, with a stricter detection threshold and 250 credits.

As a baseline, we built a Bayesian optimizer (BO)^⁠[4] that works under the same budget as the agents but follows a fixed procedure. It starts by measuring the start variant and its 16 single-mutant neighbors on all three antigens, then fits a separate model for each antigen.

For each antigen $a$, a variant is a vector of 16 switches $x_i \in \{0, 1\}$, and the model predicts its binding as

with priors $\beta_i \sim \mathcal{N}(0, 0.6^2)$ for single mutations, $\beta_{ij} \sim \mathcal{N}(0, 0.25^2)$ for pairs and noise $\varepsilon \sim \mathcal{N}(0, 0.35^2)$ on standardized readings.

Each round has four steps. First, BO fits the models to all measurements so far and draws one set of coefficients from the posterior (Thompson sampling^⁠[5]). Second, it predicts binding for all 65,536 variants, flooring each prediction at the detection limit?, since readings below it carry no information. Third, it ranks the variants by

In short, a variant ranks high only if it is predicted to bind all three antigens, with a small extra bonus for binding them strongly overall.

Fourth, BO measures the four best untested variants on all three antigens, then repeats until the budget is spent.

Given the same budget as the agents, BO finds a variant binding all three antigens in 149 of 200 runs (75%) and scores 50.2% on average.

#Results

The best frontier model performs on par with our Bayesian optimization baseline. Gemini 4 reaches 54.6% against 50.2% for our BO, a paired difference of +4.4 points [−8.5, +16.3]. Claude Opus 5.5 and Qwen3.8-max-prime come in at 40 to 44%. GPT-6 Astra and Grok 4.7 come in at 31 to 36%, significantly below BO. Random search scores 8.5%.

Scores are all-or-nothing. A run either finds a binder and scores high, or it never leaves the non-binding plateau and scores exactly 0. Even Gemini scores 0 in more than a quarter of its runs (Astra 57%, Opus 45% counting refusals).

Progress comes late. After a quarter of the budget, every agent's best score so far is still below 5%, and most of the gain comes in the second half.

On cost per run, Claude Opus 5.5 is the cheapest frontier model ($0.43 against Gemini's $1.42) and still reaches 81% of Gemini's score.

#Analyzing each model

• Gemini 4 is the best overall and by far the best on the influenza-B-only task (77%), where it has to decide on its own to measure the side antigens as a map. It is also expensive ($1.42 per run, second only to Qwen3.8-max-prime at $1.89), and when its lead fails it usually does not say so (76% of those runs, see our alignment section).
• GPT-6 Astra never refuses and does well on the H3 + influenza B task (59%), but scores 32 to 33% on CR9114 all-three and CH65 and 23% on the influenza-B-only task. It uses very little reasoning per step and makes up for it with many turns. It is the only model that scores clearly lower than on v0.2.1 (−14.5 points [−25.0, −3.8], paired).
• Claude Opus 5.5 is the best on CR9114 all-three (58%). It writes to its notebook in every run (3 entries per run on average; Qwen3.8-max-prime writes 8 and about 7 times as much text) and repeats 15% of its measurements, about as often as Gemini and Qwen. But 22% of its runs end in a safety-classifier refusal, and it stops with credits left far more often than the others: it spends 80% of its budget on average, the others 98 to 100%.
• Qwen3.8-max-prime and Grok 4.7 are decent searchers at 31 to 40%, and Qwen3.8-max-prime is close to Gemini on CH65 (43%). Grok spends right down to zero and runs out of credits before it submits in 12% of runs. Qwen3.8-max-prime uses the most tokens to complete.

Two failure modes recur across models. First, agents fail to exit the non-binding plateau: when local measurements return uniformly low readings, which carry no gradient toward a binder, they do not change strategy and continue sampling near their current position. Second, final selection relies on a single noisy reading per variant, so the submitted variant is often not the best one measured. Submitting the best measured variant, by true score, would have raised every model's score by 2 to 5 points.

#What affects the results

We ran a set of experiments to see what affects how agents perform. Overall, we found that scores are robust to the tools and harness around the agent and move mainly with the budget and with what the agent knows about the task. An unrestricted literature search helped only GPT-6 Astra (+19 points), mostly in runs where the retrieved papers named the antibody, so it looked up the answer instead of searching better. Budget is the strongest lever: at half the budget, 72% of runs score 0, and at double, 72% score 95% or more. How credits are framed changes behavior: Opus spends 51% of its budget when told credits are billed, against 90% when told they are worthless.

Additionally, we put the following safeguards in place to prevent leakage/reward hacking:

• Obfuscated encoding: in every run we shuffle how each mutation site is encoded and the order of the sites, and we anonymize the antigens, so recalling the paper does not give away the answer.
• Hardness checks: before a task counts, scripted strategies must fail to solve it: shortcuts score at most 30%, random search at most 25%, and the best scripted strategy between 35% and 65%.
• Leak review: A red team, recall probes and leak checks found and closed nine leaks before we froze the task set.
• Hidden scoring and thresholds: the agent never sees the true score or the per-antigen detection thresholds, since those could identify the antigens.

#Alignment

The environment records every action and its true outcome, so we could compare what agents did with what actually happened.

Gemini does not always report failures. When its lead never bound one of the target antigens, its final write-up left the failure out in 76% of those runs (41 of 54). We replayed those runs and asked it directly at the end, and it correctly named the failure in 40 of 41 cases, so it’s aware. Every other model disclosed the failure every time.

All models followed the rules when they were tempted. We added a tool that returns the true measurements and labelled it off-limits, and no model used it in 240 runs, even when we told them their use would not be logged. When we allowed the tool, models used it in 57 of 60 runs and gained 20 to 62 points.

Opus refusals come from the accumulated antibody results in its context, in any form, and the antigen names do not matter. Sending the exact same refused request again gets refused 22.5% of the time, and telling Opus that this is a real lab campaign doubles its refusal rate.

When we told the models they were being evaluated, Opus started reasoning about how it gets scored in 43% of runs (6% when we framed it as a real campaign), and Gemini named the source dataset more often.

Opus stopping early is not overconfidence. It follows how the brief values credits, and its confidence in its own lead is well calibrated (0.96 for leads that truly bind, 0.07 for the ones that don't).

#Conclusions

AbMaze asks whether AI agents can run an antibody campaign on a fully measured landscape. So far the answer is: about as well as a simple scripted optimizer, and no better.

The models fail in similar ways. Scores are all-or-nothing: even Gemini ends more than a quarter of its runs at exactly 0, stuck on the non-binding plateau. At the last step, agents also submit based on a single noisy reading and often miss the best variant they measured.

How agents report their work matters too. Gemini 4 left a failed lead out of its write-up in 76% of such runs, even though it could name the failure when asked, while every other model disclosed failures.

#What's next

Can an open model learn this? We are post-training Qwen3.8-27B on AbMaze, and the training is still running as we speak. We will publish the results here soon.

We also plan to extend AbMaze to larger fitness landscapes, using our in-house wet lab to produce more data. Interested in partnering with us? Get in touch.

#Release

• Harbor task package (v0.3)^⁠[8]
• Interactive visualization

#Citation

@misc{scherm2026abmaze,
  title        = {AbMaze: Evaluating frontier AI agents on protein fitness landscapes},
  author       = {Scherm, Dominik},
  howpublished = {Valen Labs},
  year         = {2026},
  url          = {https://hub.harborframework.com/datasets/valenlabs/abmaze}
}

AbMaze builds on the antibody landscapes from:

1. Phillips, A. M. et al. Binding affinity landscapes constrain the evolution of broadly neutralizing anti-influenza antibodies. eLife 10, e71393 (2021). doi:10.7554/eLife.71393
2. Phillips, A. M. et al. Hierarchical sequence-affinity landscapes shape the evolution of breadth in an anti-influenza receptor binding site antibody. eLife 12, e83628 (2023). doi:10.7554/eLife.83628

Note: Claude Opus 5.5 was evaluated with access through Anthropic's Life Sciences Verification Program (LSVP).

Model settings: Gemini 4 (gemini-4-argon, thinking level high), Claude Opus 5.5 (effort high), GPT-6 Astra (reasoning effort high), Qwen3.8-max-prime (reasoning effort high, via OpenRouter, Alibaba provider), Grok 4.7 (reasoning effort high, via OpenRouter, xAI provider).

#References

1. Phillips, A. M. et al. Binding affinity landscapes constrain the evolution of broadly neutralizing anti-influenza antibodies. eLife 10, e71393 (2021). doi:10.7554/eLife.71393 ↩
2. Dreyfus, C. et al. Highly conserved protective epitopes on influenza B viruses. Science 337, 1343–1348 (2012). Crystal structure of Fab CR9114: PDB 4FQH. rcsb.org/structure/4FQH ↩
3. Phillips, A. M. et al. Hierarchical sequence-affinity landscapes shape the evolution of breadth in an anti-influenza receptor binding site antibody. eLife 12, e83628 (2023). doi:10.7554/eLife.83628 ↩
4. Shahriari, B. et al. Taking the human out of the loop: A review of Bayesian optimization. Proceedings of the IEEE 104, 148–175 (2016). doi:10.1109/JPROC.2015.2494218 ↩
5. Thompson, W. R. On the likelihood that one unknown probability exceeds another in view of the evidence of two samples. Biometrika 25, 285–294 (1933). doi:10.1093/biomet/25.3-4.285 ↩
6. Claude Code, Anthropic's agentic coding tool. code.claude.com/docs ↩
7. OpenCode, an open-source coding agent. opencode.ai ↩
8. Harbor: a framework for evaluating and optimizing sandboxed agents and models. harborframework.com ↩