Sean M. Hendryx

ResearchRubrics: A Benchmark of Prompts and Rubrics For Evaluating Deep Research Agents

M. Sharma, Chaithanya Bandi, Clinton Wang, Ankit Aich et autres

Deep Research (DR) is an emerging agent application that leverages large language models (LLMs) to address open-ended queries. It requires the integration of several capabilities, including multi-step reasoning, cross-document synthesis, and the generation of evidence-backed, long-form answers. Evaluating DR remains challenging because …

SWE-Bench Pro: Can AI Agents Solve Long-Horizon Software Engineering Tasks?

Xiang Bo Deng, Jeff Da, Edwin Pan, Yun He et autres

We introduce SWE-Bench Pro, a substantially more challenging benchmark that builds upon the best practices of SWE-BENCH [25], but is explicitly designed to capture realistic, complex, enterprise-level problems beyond the scope of SWE-BENCH. SWE-BENCH PRO contains 1,865 problems sourced from a diverse …

Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domains

Anisha Gunjal, Anthony Wang, Elaine Lau, Vaskar Nath et autres

Reinforcement Learning with Verifiable Rewards (RLVR) has proven effective for complex reasoning tasks with clear correctness signals such as math and coding. However, extending it to real-world reasoning tasks is challenging, as evaluation depends on nuanced, multi-criteria judgments rather than binary correctness. …

Adaptive Guidance Accelerates Reinforcement Learning of Reasoning Models

Vaskar Nath, Elaine Lau, Anisha Gunjal, Manasi Sharma et autres

We study the process through which reasoning models trained with reinforcement learning on verifiable rewards (RLVR) can learn to solve new problems. We find that RLVR drives performance in two main ways: (1) by compressing pass@$k$ into pass@1 and (2) via "capability …

Agent-RLVR: Training Software Engineering Agents via Guidance and Environment Rewards

Jeff Da, C. Wang, Xiang Bo Deng, Yuntao Ma et autres

Reinforcement Learning from Verifiable Rewards (RLVR) has been widely adopted as the de facto method for enhancing the reasoning capabilities of large language models and has demonstrated notable success in verifiable domains like math and competitive programming tasks. However, the efficacy of …

EnigmaEval: A Benchmark of Long Multimodal Reasoning Challenges

Clinton J. Wang, Cristina Menghini, Johannes Mols, Jack Doughty et autres

As language models master existing reasoning benchmarks, we need new challenges to evaluate their cognitive frontiers. Puzzle-solving events are rich repositories of challenging multimodal problems that test a wide range of advanced reasoning and knowledge capabilities, making them a unique testbed for …

ToolComp: A Multi-Tool Reasoning & Process Supervision Benchmark

Vaskar Nath, P. Raja, Sean M. Hendryx

Despite recent advances in AI, the development of systems capable of executing complex, multi-step reasoning tasks involving multiple tools remains a significant challenge. Current benchmarks fall short in capturing the real-world complexity of tool-use reasoning, where verifying the correctness of not only …

Synthesizing field plot and airborne remote sensing data to enhance national forest inventory mapping in the boreal forest of Interior Alaska

Pratima Khatri‐Chhetri, Hans‐Erik Andersen, Bruce D. Cook, Sean M. Hendryx et autres

The boreal biome, the largest terrestrial biome on Earth, is increasingly vulnerable to climate change due to warming twice as rapidly as the global average. Climate change has increased the temperature, frequency, severity, and amount of area burned, which is leading to …

us (code pays fourni par la source)

Pre-Training Multimodal Hallucination Detectors with Corrupted Grounding Data

Spencer Whitehead, Jacob R. Phillips, Sean M. Hendryx

Multimodal language models can exhibit hallucinations in their outputs, which limits their reliability. The ability to automatically detect these errors is important for mitigating them, but has been less explored and existing efforts do not localize hallucinations, instead framing this as a …

Out-of-Distribution Detection & Applications With Ablated Learned Temperature Energy

Will LeVine, Benjamin Pikus, Jacob R. Phillips, Berk Norman et autres

As deep neural networks become adopted in high-stakes domains, it is crucial to identify when inference inputs are Out-of-Distribution (OOD) so that users can be alerted of likely drops in performance and calibration despite high confidence -- ultimately to know when networks' …

A Baseline Analysis of Reward Models' Ability To Accurately Analyze Foundation Models Under Distribution Shift

Sean M. Hendryx

Foundation models, specifically Large Language Models (LLMs), have lately gained wide-spread attention and adoption. Reinforcement Learning with Human Feedback (RLHF) involves training a reward model to capture desired behaviors, which is then used to align LLM's. These reward models are additionally used …

Enhancing individual tree mortality mapping: The impact of models, data modalities, and classification taxonomy

Pratima Khatri‐Chhetri, Liz van Wagtendonk, Sean M. Hendryx, Van R. Kane

Tree mortality is rapidly increasing as a result of more frequent and extensive droughts and forest fires across the globe. The increasing pace and scale of disturbance and resulting mortality necessitates the study of tree mortality at the scale at which it …

us (code pays fourni par la source)