The Overconfidence Problem
AI agents, by default, are too confident. Given a task, they'll attempt to complete it regardless of whether they have the information, tools, or capability to do so. This overconfidence leads to outputs that look plausible but are wrong, actions that are taken without sufficient context, and resources wasted on approaches that can't succeed.
Teaching agents to recognize uncertainty and escalate to humans when appropriate is one of the most impactful design decisions you can make. An agent that says "I am not sure about this, can you clarify?" is more valuable than one that confidently produces the wrong answer.
Designing Escalation Points
Escalation points are specific conditions under which the agent should stop and ask for human input. They can be defined explicitly in the agent's instructions or implemented as checks in the agent framework.
Common escalation points include: ambiguous instructions (when the task can be interpreted in multiple ways), missing information (when the agent needs data it doesn't have and can't obtain through tools), high-stakes actions (when the action is irreversible or has significant consequences), and repeated failures (when the agent has tried multiple approaches without success).
The instructions you give the agent matter enormously. Including phrases like "If the user's request is ambiguous, ask for clarification before proceeding" and "If you're unable to verify a fact, say so rather than guessing" shapes the agent's behavior toward appropriate humility.
Confidence Estimation
Some agent frameworks implement confidence estimation, where the agent assesses how confident it's in its planned action before executing. If the confidence falls below a threshold, the agent escalates instead of acting.
Confidence estimation can be as simple as asking the model to rate its confidence (surprisingly effective) or as sophisticated as running multiple reasoning paths and checking for consistency (more reliable but more expensive). The right approach depends on the stakes involved. For low-stakes tasks, self-reported confidence is sufficient. For high-stakes tasks, multi-path verification is worth the additional cost.
The UX of Asking for Help
How an agent asks for help matters. A good help request is specific: "I found three files that might contain the configuration you mentioned. Could you tell me which one is correct?" A bad help request is vague: "I need more information." Specific requests help the human provide useful input quickly.
Providing context with help requests is also important. Explaining what the agent has already tried, what information it has found, and why it's stuck gives the human the context they need to provide useful guidance. An agent that says "I tried querying the users table but it returned no results. Should I try a different table name?" is much easier to help than one that just says "I am stuck."
Balancing Autonomy and Caution
An agent that asks for help too often is annoying and defeats the purpose of automation. An agent that never asks for help is unreliable. The balance point depends on the specific application and user tolerance for interruption.
A good design pattern is to define tiers of autonomy. Routine, well-understood tasks proceed without confirmation. Unusual but low-risk tasks proceed with a notification. High-risk or ambiguous tasks pause for confirmation. This tiered approach provides automation where it's safe while maintaining human oversight where it matters.