XYBench is a benchmark of 8K+ questions that tests how large language models respond to requests based on misconceptions, measuring their ability to identify and address the underlying goal. The benchmark evaluates models' ability to recognize and redirect users toward effective solutions, focusing on pragmatic emphasis and misconception identification. The results show that current models often prioritize the stated request over the underlying goal, highlighting a need for improvement in model evaluation and training methods. AI summary
Firehose
Filtered to Companies, tagged “question-answering” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives