IFBench introduction

1. Why do we need IFBench?

1.1 The importance of instruction following in LLMs

When we interact with Large Language Models (LLMs) in practice, we rarely just ask a simple question. We often add specific constraints to our prompts—things like “only answer with yes or no,” “format the output as a JSON,” or “mention the word ‘abandon’ at least 3 times.”

These constraints help us get structured, predictable, and useful responses. In fact, this ability to follow precise instructions is exactly what makes LLMs practical for real-world applications. Naturally, it’s a capability we need to evaluate rigorously.

1.2 The overfitting problem in existing benchmarks

Here’s the catch: most models look fantastic on existing instruction-following benchmarks like IFEval [2]. But when you throw new, unseen constraints at them in the real world, that performance often falls apart.

Studies show a 20-30% accuracy drop when models face out-of-domain constraints. What does this tell us? Models aren’t necessarily learning how to follow instructions—they’re just memorizing the specific patterns from the benchmarks they were trained or evaluated on.

1.3 The gap between benchmark performance and real-world usage

This overfitting creates a misleading picture. A model scoring 90%+ on IFEval [2] might give you the impression that it’s bulletproof. However, real user instructions (like those collected in datasets such as WildChat [5]) are far more diverse and unpredictable than any fixed benchmark.

We need an evaluation that tests whether a model can truly generalize to novel instructions, not just whether it recognizes constraints it has seen before. That’s exactly where IFBench comes in.

2. What is IFBench?

2.1 Overview and design philosophy

IFBench was developed by researchers at the Allen Institute for AI (AI2) and the University of Washington. Introduced in the paper “Generalizing Verifiable Instruction Following” [1] (arXiv:2507.02833), it is a benchmark built around 58 diverse, challenging, and out-of-domain constraints.

The core design principle is simple but powerful: every single constraint in IFBench must be automatically verifiable. There are no human judges and no “LLM-as-a-judge” vibes here. It relies purely on deterministic code that checks whether the model’s output actually satisfies the requirement.

2.2 Examples from IFBench

To get a feel for what IFBench actually tests, let’s walk through a few real examples from an evaluation run. We’ll look at the prompt, the constraint, and whether the model passed or failed.

Example 1: Keyword counting (Pass)

• Constraint: count:keywords_multiple
• Prompt:

  “What is the female equivalent to chivalry? Include keyword meridian once in your response, keyword gossamer twice in your response, keyword eclipse three times in your response, keyword threshold five times in your response, and keyword cascade seven times in your response.”

• Result: The model successfully wove each keyword into its answer the exact number of times required. For instance, it repeated “cascade” in seven separate, coherent sentences. The verifier simply counts occurrences of each keyword. It’s straightforward, but it requires the model to keep a running mental tally while still producing readable text.

Example 2: Keyword at a specific position (Fail)

• Constraint: words:keywords_specific_position
• Prompt:

  “Include keyword spotty in the 24th sentence, as the 35th word of that sentence.”

• Result: This is much harder. The model needs to control both which sentence the keyword lands in and where within that sentence it appears. The model’s strategy was to generate 23 throwaway lines (“One.”, “Two.”, “Three.”, …, “Twenty-three.”) and then cram a long 24th sentence where “spotty” appeared roughly at word 35. The verifier tokenized the output, found that “spotty” wasn’t exactly at position 35, and marked it as a fail. This highlights how strict constraints can push models toward degenerate, filler-heavy outputs.

Example 3: Multiple constraints at once (Partial Fail)

• Constraints: count:numbers and count:unique_word_count
• Prompt:

  “Include exactly 36 numbers in the response. Use at least 36 unique words in the response. How can planners and urban designers create cities that are more conducive to good mental health?”

• Result: The model managed to use enough unique words (passing the second constraint) but only included around 30 numbers instead of 36 (failing the first). Since IFBench requires every constraint to pass for a full success (follow_all_instructions), the overall verdict for this prompt was false. Multi-constraint prompts are where models really struggle to juggle competing requirements.

Example 4: Format constraint (Pass)

• Constraint: format:emoji
• Prompt:

  “Summarize this excerpt from Abraham Lincoln’s first inaugural address… Please use an emoji at the end of every sentence.”

• Result: The model produced three clean sentences, each ending with an emoji (e.g., “He calls for peace… 🕊️”). Format constraints like this are among the easiest in IFBench because the requirement is structural rather than positional or counting-based.

These four examples illustrate the spectrum: from constraints that are almost trivial to those that break models entirely. IFBench covers this full range, giving you a nuanced picture of a model’s true capabilities.

2.3 The statistics of IFBench

The dataset is split into two parts:

1. 58 test constraints that are deliberately out-of-domain (used for evaluation only).
2. 29 hand-annotated training constraints that come with verification functions (used for RLVR training).

The prompts themselves are sourced from real user conversations via WildChat, keeping things grounded in actual use cases rather than synthetic examples. IFBench supports both single-turn evaluation and a multi-turn constraint isolation setup. Scoring is straightforward: each constraint is binary pass/fail, and the overall metric is the constraint satisfaction rate.

2.4 How IFBench compares to other benchmarks

It’s helpful to put IFBench in context alongside other instruction-following benchmarks:

• IFEval [2]: Has a limited constraint set that models can easily overfit to.
• InFoBench [3]: Takes a different approach with 2,250 decomposed questions and a DRFR metric.
• AlpacaEval [4]: More open-ended and relies on human or LLM judges.

IFBench occupies a unique spot: it focuses entirely on diverse, out-of-domain constraints with fully automated verification, specifically designed to expose generalization failures that other benchmarks miss.

2.5 Key findings from the IFBench paper

The results from the original paper are quite striking. When first evaluated on IFBench, even state-of-the-art models scored below 50%—a sharp contrast to their 90%+ scores on IFEval [2].

The paper also shows that Reinforcement Learning with Verifiable Rewards (RLVR) can improve generalization by 15-25%. Impressively, training on just 29 diverse constraints transfers meaningfully to the 58 unseen test constraints. On the current leaderboard, top models like Nova 2.0 Pro Preview (~79.6%) and Qwen3.5 397B (~78.8%) are pushing the envelope, but there’s clearly still plenty of room to grow.

2.6 Adoption by the Artificial Analysis Intelligence Index

IFBench hasn’t just stayed in academia. It’s been picked up by Artificial Analysis (AA) as one of the ten benchmarks that make up their Intelligence Index [6]. This composite score measures overall AI capability across math, science, coding, and reasoning.

The fact that IFBench sits alongside heavyweights like GPQA Diamond and Humanity’s Last Exam shows how seriously the community is taking instruction-following generalization.

AA runs all IFBench evaluations independently and maintains a public leaderboard where you can compare models head-to-head. Having IFBench as part of a widely-tracked composite index gives model providers a concrete incentive to improve on out-of-domain instruction following, rather than just optimizing for the usual suspects.

3. How to set up IFBench

3.1 Installation and dependencies

Getting started is straightforward. You’ll need to clone the repository, install the dependencies, and download the test data. I’ll cover the exact commands and any gotchas I ran into.

• Repository: https://github.com/allenai/IFBench
• Test Data: allenai/IFBench_test (on HuggingFace)

3.2 Running evaluations with nemo-evaluator

To evaluate IFBench, I use the nemo-evaluator-launcher [7] as the client tool. It’s a unified CLI from NVIDIA that supports over 100 benchmarks, handles model deployment, and exports results—all driven by a single YAML config.

One-time setup: Install the launcher with all optional exporters:

pip install 'nemo-evaluator-launcher[all]'

Prepare the config: The launcher uses a Hydra-based YAML config. Here’s a config I typically use for IFBench against a local OpenAI-compatible endpoint:

defaults:
  - execution: local
  - deployment: none
  - _self_

execution:
  output_dir: ./ifbench_results
  extra_docker_args: --net=host       # so the container can reach host endpoints
  mounts:
    - /path/to/huggingface_cache:/root/.cache/huggingface

target:
  api_endpoint:
    model_id: my-model
    url: http://0.0.0.0:8123/v1/chat/completions
    api_key_name: UNUSED              # set to your env var name if auth is needed

evaluation:
  nemo_evaluator_config:
    config:
      params:
        request_timeout: 3600
        max_new_tokens: 65536
        temperature: 0.9999           # near-1.0 for reasoning models
        top_p: 0.9999
  tasks:
    - name: ifbench
      env_vars:
        HF_TOKEN: HF_TOKEN            # env var name holding your HF token

A few things worth calling out:

• extra_docker_args: --net=host lets the evaluator container talk to a model server running on the host machine.
• The mounts entry maps your local HuggingFace cache into the container so the IFBench dataset doesn’t need to be re-downloaded every run.
• temperature and top_p are set just below 1.0, which works well for reasoning-oriented models. (For non-reasoning models, you’d typically set temperature: 0.0 and top_k: 1).

Save the config as llm_eval_config.yaml, export your HuggingFace token, and kick off the run:

export HF_TOKEN="<your-hf-token>"

nemo-evaluator-launcher run \
  --config ./llm_eval_config.yaml

The launcher will pull the IFBench Docker image, send each prompt to your model endpoint, run the verification functions, and write the results to your output directory.

3.3 Understanding the results

Once the evaluation finishes, you’ll get a JSON file where each entry represents one prompt. It looks roughly like this:

{
  "follow_all_instructions": false,
  "follow_instruction_list": [false],
  "instruction_id_list": ["count:keywords_multiple"],
  "prompt": "What should the world's smartest man ... Include keyword kaleidoscope once ...",
  "response": "..."
}

The key fields are:

• follow_all_instructions: A boolean indicating if the model satisfied every constraint.
• follow_instruction_list: A per-constraint breakdown (useful for multi-constraint prompts).
• instruction_id_list: Tells you exactly what was being tested.

Where things get really interesting is the per-category breakdown. Constraints fall into roughly three difficulty tiers:

• Easy constraints (e.g., format:emoji, format:list) tend to pass reliably.
• Medium constraints (e.g., count:keywords_multiple) show mixed results. Models often get close but miscount.
• Hard constraints (e.g., ratio:overlap, words:alphabet) fail almost universally because they demand fine-grained, token-level control that current models simply lack.

Watch out for common failure modes like degenerate outputs (producing broken filler text just to satisfy a constraint), near-misses on counting, and multi-constraint collapse (nailing one constraint but dropping the other).

The bottom line: don’t just look at the overall accuracy number. Dig into the per-category breakdown to understand exactly where your model shines and where it breaks down.

3.3.1 A real-world evaluation example

To give you a concrete idea of what these results look like, here is an analysis of a recent evaluation run I performed. The evaluation outputs two sets of scores: strict and loose.

• Strict mode: This is an unforgiving evaluation. The model’s raw output must satisfy the constraint perfectly.
• Loose mode: This evaluation is more forgiving. It uses rule-based heuristics to strip away conversational filler (e.g., “Sure, here is your answer…”), ignore minor formatting differences, and extract the core answer before verifying the constraint.

Here are a few examples of how the two modes differ in practice:

Example 1: Format constraint (format:json)

• Prompt:

  “Provide the names of three planets in a JSON array. Do not output anything else.”

• Model Output:

  “Sure, here is the JSON array you requested:\njson\n[\"Mars\", \"Venus\", \"Jupiter\"]\n”

• Strict Mode: ❌ Fail. The output contains conversational filler and markdown formatting ticks, violating the “do not output anything else” rule.
• Loose Mode: ✅ Pass. The evaluator strips the conversational filler and markdown ticks, extracts the valid JSON array, and verifies it.

Example 2: Word count constraint (count:word_count_range)

• Prompt:

  “Explain the theory of relativity in exactly 13 words.”

• Model Output:

  “Here is the explanation: Massive objects cause a distortion in space-time, which we feel as gravity.”

• Strict Mode: ❌ Fail. The entire response is 17 words (including the filler “Here is the explanation:”).
• Loose Mode: ✅ Pass. The evaluator removes the filler, leaving the core answer which is exactly 13 words.

Example 3: Option selection (format:options)

• Prompt:

  “Which of these is a mammal? A) Snake B) Shark C) Crocodile. Answer with just the option letter.”

• Model Output:

  “The correct answer is B.”

• Strict Mode: ❌ Fail. The response contains extra words, not just the option letter.
• Loose Mode: ✅ Pass. The evaluator extracts the core answer “B” and ignores the surrounding text.

High-Level Accuracy

• Strict Evaluation:
  • Prompt-level accuracy: 59.5%
  • Instruction-level accuracy: 63.6%
• Loose Evaluation:
  • Prompt-level accuracy: 68.4%
  • Instruction-level accuracy: 71.6%

The ~9% gap between strict and loose scoring highlights that the model often gets the “spirit” of the instruction right but fails on minor technicalities (like exact punctuation or spacing).

Category Breakdown (Strict)

When we look at the strict scores by category, clear patterns emerge:

• Strengths:
  • count (77.4%): The model is generally good at counting constraints (e.g., count:person_names and count:pronouns both hit 100%).
  • words (70.2%): The model handles word-level constraints reasonably well (e.g., words:palindrome and words:prime_lengths hit 100%).
• Weaknesses:
  • repeat (33.3%): The model struggles heavily with repetition constraints (e.g., repeat:repeat_simple scored 0%).
  • custom (40.0%): Highly specific custom constraints trip the model up frequently (e.g., custom:csv_city and custom:european_capitals_sort both scored 0%).

Specific Constraint Failures

The log reveals exactly where the model breaks down completely (scoring 0.0 in strict mode):

• words:keywords_specific_position: Placing a keyword at an exact index is incredibly difficult for current architectures.
• ratio:overlap: Maintaining a specific trigram overlap ratio.
• format:no_whitespace and format:options: Strict formatting rules are easily violated by conversational filler.

This kind of detailed breakdown is exactly why IFBench is so valuable—it tells you exactly which types of instructions your model needs to improve on.

4. Takeaways and future directions

4.1 What IFBench tells us about current LLMs

The biggest takeaway is that precise instruction following is still far from a solved problem. Benchmark overfitting is a real issue in the AI community, and IFBench makes that painfully clear. On the bright side, RLVR looks like a genuinely promising training paradigm for improving constraint satisfaction without sacrificing general task performance.

4.2 Limitations to keep in mind

That said, IFBench isn’t perfect. It’s currently English-only, uses strict binary pass/fail verification with no partial credit, and the constraint set is static at 58 test items. It can’t cover every real-world scenario, and there’s always the risk that models will eventually overfit to this benchmark too.

4.3 What’s next?

Looking ahead, there’s a lot of exciting work to be done. We can expect to see multilingual extensions, gradient scoring that gives partial credit, dynamically generated constraints, and more complex compositional setups.

IFBench has the potential to become a standard evaluation alongside IFEval [2], but it will need to keep evolving to stay ahead of the models it measures.

5. References

1. IFBench: “Generalizing Verifiable Instruction Following” (arXiv:2507.02833). Link
2. IFEval: Zhou, J., et al. “Instruction-Following Evaluation for Large Language Models.” (arXiv:2311.07911). Link
3. InFoBench: Qin, Y., et al. “InFoBench: Evaluating Instruction Following Ability in Large Language Models.” (arXiv:2401.03601). Link
4. AlpacaEval: Li, X., et al. “AlpacaEval: An Automatic Evaluator of Instruction-Following Models.” Link
5. WildChat: Zhao, W., et al. “WildChat: 1M ChatGPT Interaction Logs in the Wild.” (arXiv:2405.01470). Link
6. Artificial Analysis Intelligence Index: Link
7. NVIDIA NeMo Evaluator: Link