The human in the loop, at machine speed

title: "Human in the loop, as a property of the lease and a person at the other end" description: "Why full autonomy and instant human takeover are not in tension in DeskVNC: the lease plus revocation model, and the attended Boundary path for the person who has not saved a machine yet." date: 2026-10-09 tags: ["ai-agents", "human-in-the-loop", "mcp", "remote-desktop", "support", "boundary"]

There is an apparent contradiction in letting an AI agent drive a real desktop, and the contradiction is the reason most programmes stop. Full autonomy sounds like the agent runs the machine on its own. A human being able to take over at any instant sounds like the agent has to wait. The two feel like they point in opposite directions, and the operator's first question is usually which one gets compromised in the design.

The honest answer is that neither gets compromised, because the property that delivers them is older than both. A lease is a permission to act that someone else can revoke. The agent has full autonomy inside the lease. The human has full authority over the lease. The two are not opposites. They are the same primitive, seen from two sides.

DeskVNC builds the property into the same binary that drives the screen, and the property shows up in two places. On the operator side, a per-machine lease the human can revoke by focusing the window. On the recipient side, an attended Boundary path where the person who needs help presses a button, approves the connection, and can revoke or end the session at any time, with no account required. The autonomy is full. The oversight is full. The two are not trading against each other.

The lease, from the agent's side

An agent that opens a saved machine through the dvv MCP server gets a lease handle back. The lease is per machine, with a timeout the server renews while the agent is making progress. Every input the agent sends goes through the lease. Every screen the agent reads is tagged with a generation counter tied to the lease. The agent's loop is the lease's loop, and the two are the same thing.

The loop runs at the speed of the wire. A 1920x1080 Windows desktop over a LAN answers an observe-then-act cycle in 19 ms end to end, with one dvv_screen at scale 0.25, one dvv_click, and one damage-crop screen read. The reciprocal is roughly 52 actions a second per machine, which is the budget the agent plans against. Within the lease, the agent has all of that budget. Nothing the human does to the keyboard on the operator's side is in the way, because the human is not at the keyboard while the agent is running.

{
  "jsonrpc": "2.0",
  "id": 12,
  "method": "tools/call",
  "params": {
    "name": "dvv_open",
    "arguments": {
      "hostId": "design-srv-01",
      "perceive": true
    }
  }
}

{
  "jsonrpc": "2.0",
  "id": 13,
  "method": "tools/call",
  "params": {
    "name": "dvv_control",
    "arguments": {
      "limbId": "L-design",
      "action": "acquire"
    }
  }
}

{
  "jsonrpc": "2.0",
  "id": 14,
  "method": "tools/call",
  "params": {
    "name": "dvv_screen",
    "arguments": {
      "limbId": "L-design",
      "form": "damage-crop",
      "scale": 0.25
    }
  }
}

{
  "jsonrpc": "2.0",
  "id": 15,
  "method": "tools/call",
  "params": {
    "name": "dvv_click",
    "arguments": {
      "limbId": "L-design",
      "x": 412,
      "y": 188,
      "generation": 9131
    }
  }
}

That block of JSON is four calls, two seconds apart from the agent's reasoning time, all on the same machine. The lease is the handle. The generation is the safety property. The agent plans in this budget and acts in this budget, and nothing inside the lease says the human is supposed to be involved.

The revocation, from the human's side

The human's authority over the lease is not a feature flag. It is the same mechanism that delivers the autonomy. The dvv MCP server runs inside the same binary as the DeskVNC client UI. The agent's input queue and the operator's input queue are the same input queue, owned by the same code. When the operator clicks into the window, the same keystroke that paints the focus highlight also invalidates the lease the agent was holding.

The agent's next call comes back with a structured error rather than a silently swallowed click. The error code is LEASE_REVOKED. The agent's recovery is to ask dvv_control yield_status to find out what happened, and the answer is "human_input_on_window" or "lease_lapsed". On the human side, that is a single click. On the agent side, that is a structured error with a reason the agent can act on.

The recovery is the design. A well-built agent treats LEASE_REVOKED the way a polite coworker treats "I'll take that one". The agent stops, asks the operator whether to continue, and either resumes when asked or hands the task back. The pattern shows up in the installable agent skill that ships with the repository: a LEASE_REVOKED is the one case where the agent stops and tells the user, because the only thing that triggers it is a person who wants control back.

Three error codes an agent has to handle shape the autonomy. They are real codes the dvv server returns, not invented ones, and every agent built on this control plane handles them:

LIMB_GONE: list limbs, reopen the machine.
SCREEN_CHANGED: read the screen, retry the click or key.
LEASE_REVOKED: ask dvv_control yield_status, stop if a person took over, reopen if the lease lapsed.

Each one is a structured response with a known recovery. None of them is an exception. None of them is a hang. The loop continues after the recovery the same way it was running before.

The attended path, from the recipient's side

The lease story covers the operator's side: a saved machine the agent opens. The other half of the human-in-the-loop story is the person on the other end of the wire, the one who has not saved a machine and probably does not want a saved machine. That person needs help with a desktop they are sitting in front of, and they want to approve the connection before it happens. That is the attended Boundary path.

The recipient does not need an account. They do not need a saved machine. They do not need to install anything other than the small DeskVNC Support app, which the operator sends them as a download from the same release page as the main client. The app is signed and notarized on macOS, signed on Windows, and ships as an x86_64 tarball on Linux. The recipient runs it, and the app shows a single button: Get a code. Pressing it does one of two things, depending on whether a code service is configured.

When a code service is configured, the operator can share a short number directly. The app displays four digit groups like 1234 5678 9012. The number expires after a single lookup, and the person at the remote computer still approves the session. The operator types the number into Boundary support in the main DeskVNC client and connects. Boundary tries a direct encrypted path first and falls back to its relay when the networks do not allow a direct path, which is the part that means the support call works the same way whether the two machines are on the same LAN or separated by a corporate NAT.

When no code service is configured, the button reads Create invitation instead, and the app generates a full invitation the operator pastes into the main client. Either path lands at the same approval screen on the recipient's side. The recipient sees who is connecting, what the connection is for, and the controls to revoke or end the session at any time. The recipient approves, and the session opens.

# The recipient's view, before approval
DeskVNC Support is waiting for your approval.
Session requested by: ops@example.com
Network: direct encrypted path (preferred)
Controls: revoke control at any time, end session at any time
[ Approve ]   [ Decline ]

The approval screen is the human-in-the-loop half of the lease contract. The operator's authority on the operator side is a lease the agent holds. The recipient's authority on the recipient side is an approval gate the recipient holds. Both sides have full authority over their half of the connection, and both halves meet in the same moment of approval.

What revocation looks like in attended Boundary

The attended Boundary session has the same revocation property as the operator-side lease, with the roles reversed. The recipient can revoke control without ending the session. Revoking control stops the operator's input but keeps the connection open, so the recipient can watch, take screenshots, or hand control back. The recipient can also end the session entirely. Either action takes one button press on the recipient's DeskVNC Support app.

The operator sees the revocation as the same structured error the agent saw in the operator-side lease: LEASE_REVOKED. The MCP loop on the agent side, if the agent is driving this session, asks dvv_control yield_status, sees that a recipient revoked control, and stops. The session is not just paused. The session is over on the agent's authority, because the recipient said so. There is no override on the agent's side. The recipient's authority is final.

This is the part of the design where autonomy and oversight meet. The agent is allowed to drive the machine. The recipient is allowed to stop the agent. The two are not in tension, because the second is the condition that makes the first acceptable. The agent is not a free roaming process on the recipient's desktop. The agent is a tenant on a lease the recipient can revoke. The revocation is one click away, always, on either end.

Why this is the property that lets the agent touch real machines

The QA team that wants an agent to drive a hundred Citrix desktops has to answer the question "what if it goes wrong" before the deposit clears. The right answer is not "the agent is well trained". The right answer is "the human can stop it in one keystroke, and the agent hears about it in band". That answer is the lease and the generation counter from the operator side, and the approval gate and the recipient-side revocation from the support side. Same primitive, both halves of the connection.

The same answer applies to the helpdesk engineer who has to support a user who does not want their machine in any saved library. The recipient's authority is the only authority. The operator gets a session on the recipient's terms, with the recipient's approval, and the recipient can close the door at any moment. The support app asks for Screen Recording and Accessibility permissions on macOS when the session needs them, and the recipient's approval covers the permissions the same way it covers the connection.

The repository at github.com/psmux/DeskVNC has the client, the MCP server, the installable agent skill, the attended Boundary support apps for all three platforms, and the release history of a design that puts the human at the centre of the authority model. The agent is fully autonomous inside the lease. The human is fully authoritative over the lease. Both halves are the same property, and both halves are the reason the agent is allowed to touch real machines at all.