Marco Túlio Ribeiro

Sequential Diagnosis with Language Models

Harsha Nori, Mayank Daswani, Christopher Kelly, Scott Lundberg et autres

Artificial intelligence holds great promise for expanding access to expert medical knowledge and reasoning. However, most evaluations of language models rely on static vignettes and multiple-choice questions that fail to reflect the complexity and nuance of evidence-based medicine in real-world settings. In …

Adaptive Testing of Computer Vision Models

Irena Gao, Gabriel Ilharco, Scott Lundberg, Marco Túlio Ribeiro

Vision models often fail systematically on groups of data that share common semantic characteristics (e.g., rare objects or unusual scenes), but identifying these failure modes is a challenge. We introduce AdaVision, an interactive process for testing vision models which helps users identify …

us, gb (code pays fourni par la source)

Supporting Human-AI Collaboration in Auditing LLMs with LLMs

Charvi Rastogi, Marco Túlio Ribeiro, Nicholas S. P. King, Harsha Nori et autres

Large language models (LLMs) are increasingly becoming all-powerful and pervasive via deployment in sociotechnical systems. Yet these language models, be it for classification or generation, have been shown to be biased, behave irresponsibly, causing harm to people at scale. It is crucial …

us (code pays fourni par la source)

AHA!: Facilitating AI Impact Assessment by Generating Examples of Harms

Zana Buçinca, Chau Pham, Maurice Jakesch, Marco Túlio Ribeiro et autres

While demands for change and accountability for harmful AI consequences mount, foreseeing the downstream effects of deploying AI systems remains a challenging task. We developed AHA! (Anticipating Harms of AI), a generative framework to assist AI practitioners and decision-makers in anticipating potential …

Targeted Data Generation: Finding and Fixing Model Weaknesses

Zexue He, Marco Túlio Ribeiro, Fereshte Khani

Even when aggregate accuracy is high, state-of-the-art NLP models often fail systematically on specific subgroups of data, resulting in unfair outcomes and eroding user trust. Additional data collection may not help in addressing these weaknesses, as such challenging subgroups may be unknown …

Collaborative Development of NLP models

Fereshte Khani, Marco Túlio Ribeiro

Despite substantial advancements, Natural Language Processing (NLP) models often require post-training adjustments to enforce business rules, rectify undesired behavior, and align with user values. These adjustments involve operationalizing "concepts"--dictating desired model responses to certain inputs. However, it's difficult for a single entity …

Supporting Human-AI Collaboration in Auditing LLMs with LLMs

Charvi Rastogi, Marco Túlio Ribeiro, Nicholas S. P. King, Harsha Nori et autres

Large language models are becoming increasingly pervasive and ubiquitous in society via deployment in sociotechnical systems. Yet these language models, be it for classification or generation, have been shown to be biased and behave irresponsibly, causing harm to people at scale. It …

us, gb (code pays fourni par la source)

ScatterShot: Interactive In-context Example Curation for Text Transformation

Sherry Wu, Hua Shen, Daniel S. Weld, Jeffrey Heer et autres

The in-context learning capabilities of LLMs like GPT-3 allow annotators to customize an LLM to their specific tasks with a small number of examples. However, users tend to include only the most obvious patterns when crafting examples, resulting in underspecified in-context functions …

us (code pays fourni par la source)

Sparks of Artificial General Intelligence: Early experiments with GPT-4

Sébastien Bubeck, Varun Chandrasekaran, Ronen Eldan, Johannes E. Gehrke et autres

Artificial intelligence (AI) researchers have been developing and refining large language models (LLMs) that exhibit remarkable capabilities across a variety of domains and tasks, challenging our understanding of learning and cognition. The latest model developed by OpenAI, GPT-4, was trained using an …

ART: Automatic multi-step reasoning and tool-use for large language models

Bhargavi Paranjape, Scott Lundberg, Sameer Kumar Singh, Hannaneh Hajishirzi et autres

Large language models (LLMs) can perform complex reasoning in few- and zero-shot settings by generating intermediate chain of thought (CoT) reasoning steps. Further, each reasoning step can rely on external tools to support computation beyond the core LLM capabilities (e.g. search/running code). …

Targeted Data Generation: Finding and Fixing Model Weaknesses

Zexue He, Marco Túlio Ribeiro, Fereshte Khani

Even when aggregate accuracy is high, stateof-the-art NLP models often fail systematically on specific subgroups of data, resulting in unfair outcomes and eroding user trust.Additional data collection may not help in addressing these weaknesses, as such challenging subgroups may be unknown to …

us (code pays fourni par la source)

Editing Models with Task Arithmetic

Gabriel Ilharco, Marco Túlio Ribeiro, Mitchell Wortsman, Suchin Gururangan et autres

Changing how pre-trained models behave -- e.g., improving their performance on a downstream task or mitigating biases learned during pre-training -- is a common practice when developing machine learning systems. In this work, we propose a new paradigm for steering the behavior …