τ-emotion-bench

τ-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions

Rishitosh Kumar Singh*, Amir Saeidi*†, Shamanthak Hegde*, Shrinidhi Kumbhar, Chitta Baral

*Indicates Equal Contribution   †Corresponding Author
Paper Leaderboard Model Dataset Code
Bar chart of Avg@3 per model on τ-Emotion-Bench with two user simulators: under Qwen3-235B-A22B-Instruct-2507 the best is DeepSeek-V4.1-Flash at 55.9, under GLM-5.2 the best is GLM-5.3 at 78.6. Tracer models are hatched
Avg@3 of LLM agents on τ-Emotion-Bench over three runs. Dark bars: Qwen3-235B-A22B-Instruct-2507 as the user; light bars: GLM-5.2 as the user. Hatched bars are our Tracer models.

With emotional users, the best frontier agent reaches only 55.9% Avg@3 (78.6% with GLM-5.2 as the user), and no model passes more than 32.2% of tasks in all three trials (66.5% with GLM-5.2).

TL;DR

  • τ-Emotion-Bench: 580 executable tool-calling tasks in 10 domains, where the simulated user carries one of 594 emotion conditions and can walk away if the agent handles the conversation badly.
  • Tracer: an automated pipeline that builds policy-valid, verified tasks and reuses the same harness to collect interaction-efficient training trajectories.
  • Findings: frontier agents are fragile under emotional users, and fine-tuning Qwen3 on 1,958 Tracer trajectories adds 13.8–19.7 points of Avg@3 (17.1–25.2 with a GLM-5.2 user) and transfers to unseen domains, τ-Bench and τ-Trait.
580Tasks
10Domains
173Tools
22Emotions
594Emotion Conditions
1,958Training Trajectories
Abstract

In long-horizon tool-calling tasks, LLM agents can fail not only because of tool-use errors, but also because of ineffective interaction with users. Yet existing benchmarks largely assume cooperative and behaviorally stable users, leaving the effect of emotional variation on agent reliability underexplored. We introduce τ-Emotion-Bench, a ten-domain benchmark for evaluating tool-calling agents under controlled emotion-conditioned user interactions while preserving the underlying task objective. To scale both benchmark construction and agent training, we introduce Tracer, a scalable agentic data-generation pipeline that constructs grounded, policy-valid tasks and collects trajectories through an interactive agent–user–environment harness. By allowing dissatisfied users to terminate inefficient interactions, the harness provides an interaction-aware signal that filters trajectories with excessive user dependence and favors effective tool use. Experiments with two user simulators show that frontier agents remain unreliable, with substantial gaps between average task success and consistent task completion. Tracer-generated supervision consistently improves Qwen3 models across scales, generalizes to unseen domains, and transfers to τ-Bench and τ-Trait. Behavioral analyses further show that higher task success is associated with a better balance between tool execution and necessary user follow-up. Overall, τ-Emotion-Bench and Tracer provide a scalable framework for evaluating agent reliability under diverse user behaviors and improving tool-calling agents through interaction-aware supervision.

The Benchmark

Overview of τ-Emotion-Bench and the Tracer pipeline: dependency graph construction, trace sampling, reverse refinement, emotion generation and task-conditioned mixing, verification and filtering
  1. Dependency graphs of valid tool transitions per domain, manually verified.
  2. Trace sampling of executable multi-step workflows, including multi-intent tasks.
  3. Reverse refinement into a natural user instruction with goals and preferences.
  4. Emotion mixing: a task-specific feeling mapped to one of 594 emotion conditions.
  5. Verification for executability and policy consistency, plus manual review of all 580 tasks.

Emotions follow Ekman's Atlas of Emotions, crossed with politeness, urgency and trust: Anger Disgust Enjoyment Fear Sadness The task stays fixed; if the user ends the episode with ###UNSATISFIED###, the agent fails.

DomainRead ToolsWrite ToolsEmotions (Family / Sub)Tasks
Airline865 / 1670
Retail975 / 1770
Telecom1265 / 1370
Telehealth1853 / 870
School944 / 1050
Basketball1731 / 250
Restaurant962 / 650
Travel972 / 650
Movie1333 / 750
Bank1393 / 850

Results

Consistency is the hard part: the best average Pass^3 is 32.2%, and every model swings widely across domains. Bold marks the best score in each column. Further results with GLM-5.2 as the user agent are on the leaderboard.

Model AirlineRetailTelecomTelehealthSchool BasketballRestaurantTravelMovieBank Avg.
Non-thinking Models
Qwen3-32B10.021.412.910.020.04.06.00.00.010.09.4
Qwen3-235B-A22B17.115.710.021.422.024.04.00.00.024.013.8
DeepSeek-V4.1-Flash32.944.317.131.442.038.026.030.022.038.032.2
Thinking Models
Qwen3-32B8.621.48.615.714.012.02.02.02.00.08.6
Qwen3-1.7B0.00.08.62.90.00.00.00.00.00.01.2
Qwen3-4B2.91.48.614.30.08.00.00.00.04.03.9
Qwen3-8B4.314.312.922.90.012.04.02.00.012.08.4
Qwen3-235B-A22B-T22.931.47.120.020.026.08.04.00.026.016.5
Qwen3.8-27B32.938.611.430.048.028.030.016.010.028.027.3
MiniMax-M37.142.911.414.316.034.012.012.04.022.017.6
Muse-Glimmer-30B20.00.08.614.364.028.010.018.012.030.020.5
MiniMax-M2.718.627.17.112.946.022.04.00.02.028.016.8
Gemma-4-31B-it30.037.112.922.956.026.018.06.012.026.024.7
GLM-5.338.645.72.920.048.062.028.014.030.028.031.7
Kimi-K2.7-Code30.044.310.014.344.034.012.012.08.052.026.1
Tracer-High-1.7B (Ours)17.117.111.414.38.012.00.00.00.024.010.4
Tracer-High-4B (Ours)22.930.08.622.934.020.06.00.02.026.017.2
Tracer-High-8B (Ours)27.131.415.731.438.016.012.02.02.022.019.8

Pass^3 (%) on the ten τ-Emotion-Bench domains, with Qwen3-235B-A22B-Instruct as the user simulator. Shaded rows are our Tracer models.

Training With Tracer

Tracer keeps only successful trajectories in which the user did not walk away, which favors agents that use tools instead of leaning on the user. Tracer-High is trained on all ten domains (1,958 trajectories); Tracer-Low on four domains only (1,498).

  • Narrows the gap: Tracer-High-8B reaches 38.1% Avg@3, above MiniMax-M3 (37.0%) and Qwen3-235B-A22B-Thinking (36.0%). With a GLM-5.2 user it reaches 54.7%, above Qwen3-235B-A22B-Instruct (51.7%).
  • Generalizes: Tracer-Low has the best out-of-domain Pass^3 at every scale, against methods trained on up to 91K trajectories.
  • Transfers: gains carry over to τ-Bench and τ-Trait, which use different tasks and user simulators.
Method In-Domain Out-of-Domain
Pass^1Pass^3Pass^1Pass^3
Qwen3-1.7B
Base8.42.93.30.0
AReaL11.41.91.90.3
EnvScaler18.25.26.50.7
Tracer-Low (Ours)34.312.98.42.7
Tracer-High (Ours)36.215.217.7†7.3†
Qwen3-4B
Base24.34.311.72.0
AReaL23.66.27.61.7
EnvScaler37.617.616.93.3
Arctic-AWM26.37.110.91.7
SPADE7.61.91.00.0
Tracer-Low (Ours)45.720.518.67.7
Tracer-High (Ours)45.720.531.3†14.7†
Qwen3-8B
Base32.810.516.55.0
AReaL24.38.17.51.0
EnvScaler37.816.720.87.0
Arctic-AWM30.59.514.22.3
Simia-Tau-SFT33.511.414.54.3
Simia-Tau-SFT-RL34.812.914.02.7
SPADE7.31.90.80.0
Tracer-Low (Ours)47.824.320.78.3
Tracer-High (Ours)46.424.731.4†15.3†

Average Pass^1 and Pass^3 (%). In-domain: Airline, Retail, Telecom. Out-of-domain: the six domains Tracer-Low never saw. † Tracer-High is trained on all ten domains, so these six domains are not unseen for it; its scores are averaged from the paper's per-domain appendix table and are excluded from the bold best-score marking.

Grouped bars of pass^1, pass^2 and pass^3 for Qwen3 base, Tracer-Low and Tracer-High at 1.7B, 4B and 8B on τ-Bench Airline, τ-Bench Retail, τ-Trait Telecom and τ-Trait Telehealth
Qwen3 base vs. Tracer-Low and Tracer-High on τ-Bench and τ-Trait (Pass^1 to Pass^3).

Key Findings

Policy Violations Persist

Violations drop with capability and Tracer tuning, but reach about 67% in Telehealth and 91% in Telecom even for stronger models.

Dot plots of policy violation rate per emotion family in each domain, for Qwen3, Tracer-Low, Tracer-High, stronger models and weaker models

Balance Tool Use and Follow-Up

Higher Avg@3 goes with a moderate tool-call to follow-up ratio; more aggressive tool use does not help.

Five scatter panels, one per emotion family, of Avg@3 against the tool-call to follow-up ratio

Empathy Adds Follow-Up

Empathetic replies come with more follow-up questions and often lower success, especially under Enjoyment.

Heatmap of change in success with empathy by emotion and model group, and share of follow-up questions with and without empathy

The User Simulator Is Reliable

Of 6,961 failed trajectories, only 124 user turns are confirmed user errors (1.1% of failed trials), at most 1.4% per emotion family.

Heatmap of user hallucination rates by sub-emotion and domain, mostly empty

Why Tracer Generalizes

Tool calls outnumber follow-up actions in 76–79% of Tracer trajectories, versus 56% for AReaL and 22% for Simia.

Scatter plots of tool calls against follow-up actions in AReaL, Simia, Tracer-Low and Tracer-High training trajectories

Acknowledgments

We thank the Research Computing (RC) at Arizona State University (ASU) for their generous support in providing computing resources. The views and opinions of the authors expressed herein do not necessarily state or reflect those of the funding agencies and employers. This research was also partially supported by funding from DARPA.

BibTeX

@article{singh2026tauemotionbench,
  title={$\tau$-Emotion-Bench: Exposing the Interaction Fragility of LLM Agents through Real-World User Emotions},
  author={Singh, Rishitosh Kumar and Saeidi, Amir and Hegde, Shamanthak and Kumbhar, Shrinidhi and Baral, Chitta},
  journal={arXiv preprint arXiv:2610.19560},
  year={2026},
  url={https://arxiv.org/abs/2610.19560}
}