Title: Orchestrating AI Agents among Humans

Abstract: As AI agents are deployed in real-world settings, determining when to expose users to AI assistance becomes increasingly critical.  Effective use of AI agents requires invoking the right agent at the right time. We introduce Modiste, an interactive tool for learning personalized decision support policies that dynamically adjust user access to AI agents. Modiste leverages tools from contextual bandits to optimize when and how AI agents provide support to balance performance, cost, and constraints. We further characterize the theoretical conditions under which orchestration between agents is beneficial. Our empirical studies show how selective access to AI agents, including deliberate disengagement from AI, can improve decision outcomes, reduce unnecessary AI use, and align AI agents with real-world constraints. We conclude with a call for human-centered interactive evaluation of AI agents by assessing the effectiveness of AI agents via multi-turn interactions with experts.

Vitae: Umang Bhatt is an Assistant Professor & Faculty Fellow at the Center for Data Science at New York University and a Senior Research Associate in Safe and Ethical AI at the Alan Turing Institute. He completed his PhD in the Machine Learning Group at the University of Cambridge. His research lies in human-AI collaboration, AI governance, and algorithmic transparency. His work has been supported by a JP Morgan PhD Fellowship and a Mozilla Fellowship. Previously, he was a Research Fellow at the Partnership on AI, a Fellow at Harvard’s Center for Research on Computation and Society, and an Advisor to the Responsible AI Institute. Umang received his MS and BS in Electrical and Computer Engineering from Carnegie Mellon University.