This paper recovers vehicle trajectories from CAN signals in naturalistic driving and presents a new dataset for evaluating such methods. Practitioners might care because it shows how to accurately track vehicle movements even when GPS data is removed, which could be useful for usage-based insurance and location-based services.
Firehose
Filtered to Papers, tagged “benchmarks” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives
Browse by tag
This paper creates a benchmark for large language models to evaluate their performance in finance, specifically in Japan, and shows that some models can excel in this domain even with limited external knowledge.
This paper introduces a new benchmark for web agents that can perform tasks across multiple web applications and modalities, including text, vision, audio, and video. Practitioners can care about this benchmark because it provides a comprehensive evaluation framework for web agents that can handle complex, real-world tasks.
This paper explores how video models can be used to render 3D worlds created by code, and whether they can accurately preserve the world's structure and appearance over time. Practitioners might care because reliable generative rendering is crucial for applications like video games and animation.
This paper explores the capabilities of decision models in long-horizon agents, examining whether a poorly performing model can still be effective in teaching a policy through reinforcement learning. Practitioners might care about the implications for designing more robust and effective reinforcement learning systems.
This paper introduces SO-Arena, a benchmark that evaluates the capabilities and reliability of System One models, which are fast decision-making models, and shows that even the best-performing models can make unreliable decisions, highlighting the need for a more comprehensive understanding of these models' strengths and limitations.