Skip to Content

When intelligence becomes agency

Written By Joao Dias-Ferreira
When intelligence becomes agency | Wyrde AI
Agency & human–AI interaction

A useful AI assistant needs to judge when its help is needed, what form it should take, and what it has permission to do. Our research proposes a framework for making those decisions.

João Dias Ferreira / Wyrde AIResearch perspective 7 min read
Editorial illustration of a person guiding three paths from a dense stream of dark lines toward an amber circle, ochre mountains and amber leaves. Other dark paths form holding loops, including one beneath the selected paths on the right.
From competing signals to considered assistance. AI-generated editorial metaphor; the diagrams below describe the framework.

An AI assistant can understand a situation and still get the moment wrong. Knowing when to act, ask, or wait is central to useful assistance. Our research explores how AI can take initiative while staying within human authority and why choosing to hold back can be as important as taking action.

Imagine an AI assistant you have given permission to follow parts of your working day. Within a few minutes, it notices a critical warning from a system at work, several family messages about a school event, and a reminder that you have been sitting for hours.

It could interrupt you three times. It could collect everything into a summary. It could raise the work warning immediately and leave the rest for a natural break.

The most helpful response depends on the situation. If the family messages concern a routine school event, they may be able to wait. If they say your child has been injured, the priorities change. Your current activity, the assistant’s responsibilities, and the consequences of delay all matter.

An assistant could understand every message correctly and still handle this moment badly.

In When Intelligence Becomes Agency, we call this the activation problem: deciding whether a situation warrants a response, when that response should happen, and what form it should take. We propose a conceptual and formal framework for addressing it in assistants that remain available across tasks and over time.

The decision before the task

Give an agent a task and it may be able to plan, use tools, check its work, and keep going. But an assistant with an ongoing role also needs to recognize when something deserves its attention without receiving a new instruction.

“Help me keep my commitments” is different from “remind me at three.” The first requires the assistant to connect changing circumstances with a responsibility that persists.

Our central argument is that agency depends on how five elements work together: perception, intent, appraisal, constraints, and feedback. The system notices what is happening, connects it to what it is committed to, evaluates its significance, chooses within its limits, and adjusts in light of the outcome.

Appraisal is the step that asks why something matters here and now. It weighs urgency, uncertainty, possible harm, the cost of interrupting, and the consequences of different responses. A fixed rule can identify an event; this judgment considers what the event means in the current situation.

Sometimes that judgment leads to useful preparation. With the relevant permissions, an assistant could notice that you agreed to deliver a report on Friday, recognize that your calendar is already full, and prepare a draft for review. It could hold its notification until a break in your conversation. Preparing the draft and choosing when to mention it are parts of the same responsibility.

Figure 01

How governed proactive behavior begins

The proactive path under delegation. The mandate conditions intentions, policy and profile. Evidence passes the consent gate before updating representations of the world, the principal and the agent. The principal’s condition is inferred, uncertain and correctable. Appraisal informs readiness. Each commitment must exceed the contextual activation bar to become eligible; priority chooses only among eligible commitments. If none is eligible, no episode opens and permitted sensing and representation updates continue. Otherwise, an episode opens for the winner and a behavior is selected within policy. Deliberate restraint remains an option even when action is possible. External effects pass an action firewall. Feedback supports revision within authorized limits. Direct instructions follow a separate entry path.
The proactive path under delegation: readiness determines which commitments are timely; priority chooses among them; selection determines the permitted response. No episode opening is distinct from deliberate restraint within an episode. Both must be distinguishable in the record. Consent and external effects are governed by enforcement outside the learned system. The principal’s condition is an uncertain, correctable inference. Direct authorized instructions can open an episode without the readiness gate or priority competition, but remain within the mandate. Simplified adaptation of Figure 3 in the paper.

A continuing role with clear limits

An assistant needs a clear basis for taking that initiative. We call this a mandate: a record of what it is authorized to do on someone’s behalf, including its purpose, scope, duration, and boundaries. The person granting that authority can change or withdraw it.

A mandate might allow an assistant to monitor deadlines and prepare drafts while requiring confirmation before sending messages. Recognizing that a message would be helpful does not create permission to send it.

This is delegated agency: the assistant acts for someone who retains authority over its work. We use symbiotic agency for a continuing form of that relationship. The mandate persists across tasks, the assistant stays connected to the person’s changing situation, and its assistance reflects their priorities and condition within agreed limits.

That continuing relationship could help extend a person’s attention, memory, and ability to coordinate. It also makes control over the relationship essential. Learning that you usually approve a particular action cannot silently become permission to take it. Passing work to another agent cannot grant that agent broader authority than the original delegation allowed.

Figure 02

Where symbiotic agency fits

The paper classifies systems meeting its five-part minimum for agenthood by retained authority. Autonomous agents have no operative mandate from a principal. Delegated agents act under a principal’s revocable mandate. Symbiotic agents form a subset inside delegated agency and require all three conditions: a standing mandate across episodes; continuing, consent-gated coupling to the principal’s context and condition, with calibrated inference; and personalization that can evolve only within authorized bounds. The categories are not intelligence scores.
The classification applies after a system meets the paper’s minimum for agenthood. Symbiotic agency is a subset of delegated agency and requires all three conditions shown. A delegated agent missing any of them remains delegated. Authority comes from an operative, revocable mandate; goal origin and ownership alone do not decide the classification. Simplified adaptation of Figure 1 in the paper.

Context includes what the assistant does not know

To judge when help is appropriate, the framework distinguishes three kinds of evidence: what is happening in the world, what the assistant knows about its own operation, and what it can infer about the person it serves.

A calendar conflict belongs to the first. A failed tool or an unresolved uncertainty belongs to the second. Signs that you may be distracted or overloaded belong to the third.

That third category requires particular care. A slow reply could mean you are tired, busy, unconvinced, or simply away from your phone. The assistant’s estimate must reflect that uncertainty, and you must be able to correct its interpretation.

Access to that evidence also needs limits. Permission should govern what the assistant can observe, retain, and learn from, as well as what it can do. Your permission does not automatically cover other people who appear in a conversation or shared space.

The proposed architecture therefore places enforceable boundaries around both observation and action, outside the learned system. These boundaries remain in force as the assistant adapts.

Waiting can be a deliberate decision

In the opening example, postponing the movement reminder may be the most useful response. The assistant can retain it, watch for a suitable break, and bring it up later.

This makes restraint a behavior we can specify and evaluate. Waiting should have a purpose and, where appropriate, a condition for reconsideration. Otherwise, an assistant that sensibly defers a concern can look identical to one that forgot it.

For consequential decisions, the record should help establish what the assistant noticed, which responsibility it was serving, what authority it had, and what followed. That includes decisions to withhold action. Those records also need privacy protections because they may contain sensitive information.

What should we measure?

Task completion tells us only part of the story. An assistant might produce an accurate draft but interrupt at an inappropriate moment, or finish a task while exceeding its authority.

We propose evaluating the whole behavioral episode, from noticing a situation through responding and reviewing the outcome. That means asking:

  • Did the assistant intervene when help was warranted, and refrain when it was not?
  • Did it choose an appropriate time and form of response?
  • Did it stay within its permissions, including when delegating work?
  • Did it handle uncertainty and respond to correction?
  • Can consequential decisions be reconstructed, and does learning remain within authorized bounds?

The paper sets out measures and proposed benchmark scenarios for these questions. It also provides a method for describing and comparing how agents are organized, along with a reference architecture to guide their implementation.

This is a theoretical contribution. It does not report experimental results or establish that these capabilities have been achieved. The work ahead includes testing how reliably assistants can judge significance, resist misleading claims of urgency, learn from feedback without drifting beyond their mandate, and evaluate the value of actions they correctly chose to withhold.

For an assistant that stays with us over time, every intervention shapes the relationship. The goal is to reduce what we have to keep track of while preserving our authority over what happens next.

Joao Dias-Ferreira