SAP Director Urges Risk-Based Oversight for Enterprise AI Agents

Shyam Ravindranathan, Senior Director of Product Management at SAP, warns that current human-in-the-loop models for AI agents in enterprise software have devolved into ineffective "rubber stamps." He argues that reviewers, who are often underqualified or overwhelmed, frequently batch-approve AI outputs without meaningful oversight, creating a risk of irreversible errors and ambiguous accountability during system failures.

To address these failures, Ravindranathan proposes a risk-based oversight framework where the intensity of human review scales with the potential impact of the action. He suggests a clear distinction in oversight requirements between low-stakes tasks, such as small purchases, and high-stakes operations, such as multi-million dollar procurement orders.

Ravindranathan further advocates for strict accountability measures, including the documentation of every action taken by an AI agent and the assignment of a named human owner responsible for the results. He emphasizes that AI verification must be performed by subject matter experts rather than generic quality assurance teams, particularly during the early stages of deployment.