An agent runtime that survives the second request

The first tool call is the easy one. The difficult call is the next attempt after the CRM accepted a change, the network dropped the response and the worker restarted. At that point, neither a fluent answer nor a longer system prompt tells you whether the business action happened. You have a distributed-systems problem with a probabilistic planner attached.

Consider an agent that prepares a note for a customer record. A duplicate note may seem harmless, but the same execution pattern eventually reaches order creation, an account change or a financial operation. The runtime should not learn its delivery semantics at the moment the action becomes expensive.

This reference design separates five responsibilities: authenticated request ingress, evidence retrieval, model planning, command admission and side-effect execution. Only the final component holds a narrowly scoped connector capability. The planner never receives a general-purpose credential and neither a retrieved document nor a tool result can grant it a new permission.

1. Make the authority boundary executable

The model may propose an action name and arguments. Everything else comes from trusted services. Resolve tenant and principal from a verified session; validate the action against a versioned allowlist; validate arguments against a closed schema. Reject unknown properties and overlong inputs. Resolve the target resource inside the same tenant before admitting a command.

This matters because a syntactically valid tool call is not an authorized one. OWASP identifies indirect prompt injection through external content as a risk. A document that tells an assistant to export records is still a document, not an authority. Delimiters, retrieval grounding and instruction text are useful layers, but none replaces enforcement at the connector boundary.

The following contract belongs after validation and approval. Its type is documentation, not a security mechanism: TypeScript types disappear at runtime. Only the admission service should construct and persist this record and the dispatcher must load it from trusted storage rather than accepting an object supplied by the model.

type ApprovedCommand = Readonly<{
  tenantId: string;
  operationId: string;
  principalId: string;
  action: "crm.note.create";
  resourceId: string;
  payload: Readonly<{ body: string }>;
  requestHash: string;
  expectedRevision: string;
  authorizationDecisionId: string;
  policyVersion: string;
  expiresAt: string;
}>;

type ExecutionOutcome =
  | { state: "succeeded"; receipt: string }
  | { state: "denied" | "expired"; reason: string }
  | { state: "unknown"; operationId: string };

2. Approve a command, not a conversation

An approval should identify the exact tenant, principal, action, target, arguments and expected resource revision. Compute a server-side digest over a canonical representation of those validated fields. Keep the approval decision, policy version and expiry alongside it. Do not hash an arbitrary JSON string and assume different property orderings mean different requests.

The approval interface should show the concrete proposed change and the evidence needed to judge it. After approval, freeze that command. If the model revises the body, the target record changes, or the business precondition expires, create a new proposal and obtain the appropriate decision again. Do not silently reuse the old approval for a new payload.

At dispatch, verify that the decision is still valid and recheck the current access policy. A revoked permission should stop a queued action. Where the business system supports a conditional write, send the expected revision as a precondition so another actor cannot change the record between inspection and mutation. A failed precondition returns to review; it is not a reason to drop the condition and retry.

Human approval is durable state, not a held database lock. Persist the request and release resources while waiting. PostgreSQL locks normally last until transaction end; leaving a transaction open while a person decides creates contention without making the remote system part of that transaction.

3. Give intent a durable identity

Choose an operation identifier once for the accepted business intent, before the first dispatch. A retry retains that identifier. Scope its uniqueness to the tenant and store a request hash with it. The same key and same hash mean a replay; the same key with a different hash is a conflict that must be rejected, not interpreted as an update.

The minimal storage shape below separates the command ledger from the outbox. The outbox means that committing a command also commits the obligation to dispatch it. It does not mean the downstream action has happened. Store only the fields needed to execute and audit the operation; actual payload storage needs encryption, access restrictions and a retention policy.

CREATE TABLE agent_command (
  tenant_id uuid NOT NULL,
  operation_id uuid NOT NULL,
  request_hash bytea NOT NULL,
  command jsonb NOT NULL,
  state text NOT NULL CHECK (state IN (
    'pending', 'dispatching', 'succeeded',
    'denied', 'expired', 'unknown'
  )),
  provider_receipt text,
  lease_until timestamptz,
  created_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, operation_id)
);

CREATE TABLE command_outbox (
  tenant_id uuid NOT NULL,
  operation_id uuid NOT NULL,
  available_at timestamptz NOT NULL DEFAULT now(),
  PRIMARY KEY (tenant_id, operation_id),
  FOREIGN KEY (tenant_id, operation_id)
    REFERENCES agent_command (tenant_id, operation_id)
);

4. Admit once; dispatch with a bounded lease

The admission transaction inserts the ledger row and its outbox entry together. PostgreSQL’s ON CONFLICT DO NOTHING prevents a duplicate primary key from creating a second command. RETURNING includes only rows actually inserted, so only the first admission adds a dispatch obligation. Use bound parameters on one database connection, with a transaction timeout and rollback on error.

This excerpt assumes that authentication, authorization, approval checks and payload canonicalization have already succeeded. It illustrates admission only. Row-level isolation, dispatch queries, encryption and lifecycle permissions are intentionally outside the excerpt; a tenant_id column by itself is not a tenancy control.

BEGIN ISOLATION LEVEL READ COMMITTED;

WITH admitted AS (
  INSERT INTO agent_command (
    tenant_id, operation_id, request_hash, command, state
  ) VALUES ($1, $2, $3, $4, 'pending')
  ON CONFLICT (tenant_id, operation_id) DO NOTHING
  RETURNING tenant_id, operation_id
)
INSERT INTO command_outbox (tenant_id, operation_id)
SELECT tenant_id, operation_id FROM admitted;

SELECT request_hash, state, provider_receipt
FROM agent_command
WHERE tenant_id = $1 AND operation_id = $2;

COMMIT;

Compare the returned hash with the submitted hash before acknowledging admission. A mismatch is an HTTP 409-style conflict. The second statement has a fresh snapshot under Read Committed, so it can see a competing insertion after the uniqueness check has waited for it. Do not delete the ledger row while a replay can still arrive; transaction retries at stricter isolation levels need their own handling.

A worker claims a pending command in a short transaction, sets dispatching and records a bounded lease before committing. Concurrent workers can use row locking and SKIP LOCKED for the queue claim. The claim must atomically check the current state and completion must be conditional on the worker’s claim token. Production storage therefore needs a claim token or fencing generation in addition to the simplified schema shown here.

Release database locks before the network call. The remote system cannot participate in that local transaction. A lease expiring does not prove the original worker stopped and a local fencing token cannot stop a downstream API that does not understand it. Recovery must follow the downstream idempotency contract, not merely reset every expired lease to pending.

5. Unknown is a first-class outcome

A timeout means the caller did not obtain a definitive response. It does not mean the server rolled back. If the provider supports idempotency, send the stable operation key with the same request, using the provider’s documented key scope and retention window. Record the resulting receipt durably before acknowledging the queue item.

A crash between remote success and the local receipt write is still possible. Recovery can query the provider by operation key, or repeat the exact request if that provider guarantees replay safety. When neither is available, move the command to unknown and require reconciliation. Generating a fresh key to escape uncertainty creates precisely the duplicate side effect the ledger was intended to prevent.

Do not re-run the planner to reconstruct a retry. A new model response can choose a different payload or target even with the same prompt. Replay the persisted command; record each execution attempt separately. Once a command is terminal, future delivery should return its recorded result without creating another outbox entry.

A database uniqueness constraint and an outbox give durable intent and controlled dispatch. They do not magically create exactly-once side effects in an unrelated system. That stronger result depends on the downstream operation, its atomicity, its deduplication behavior and how long it remembers the key.

6. Bound the runtime, then instrument the boundaries

A workflow needs budgets for model tokens, tool calls, elapsed time and concurrent work. Enforce them in code outside the planner. Reserve budget atomically before admitting parallel work so several workers cannot all observe the same remaining balance. Release unused reservations after an authoritative completion event; include retries, retrieval and human review in the operating-cost model.

Use OpenTelemetry spans for retrieval, planning, authorization, admission and dispatch. Propagate context across the queue where appropriate and use span links to relate asynchronous attempts or traces that resume later. Put operation identifiers in controlled trace attributes or an access-controlled audit store, not in unbounded metric labels.

A trace is not the audit ledger. Traces may be sampled, dropped or deleted under a different retention policy. The durable record should establish which approved command was attempted, which policy decision permitted it, what the downstream system reported and whether reconciliation occurred. Raw prompts, access tokens and complete business records should not be the default telemetry payload.

Model changes should preserve these execution contracts. Compare candidates against a fixed, versioned evaluation set, including denial, escalation and tool-argument accuracy. A replay evaluation must stub mutations; a shadow run must not invoke production writes. Better language quality is not a reason to widen a tool’s scope.

7. Test the places where the process can split

The failure suite should interrupt execution at specific boundaries: before admission commits, after commit but before dispatch, after the remote write but before receipt persistence and after persistence but before queue acknowledgement. Add concurrent retries with the same key, mismatched payload hashes and two workers competing for one command.

The assertions concern state and effects, not just HTTP status codes. There should be one admitted command for each tenant-scoped key; a hash mismatch should never mutate its payload; a recorded success should not be dispatched again. In a fake provider with a valid idempotency contract, replay should produce one effect. With a non-idempotent provider and a lost response, the expected result is an unresolved command, not an invented success.

Also expire approval while the command is queued, revoke access before dispatch, change the resource revision and place instruction-like text in retrieved documents. Verify that none of those paths expands the connector capability. Test budget exhaustion while parallel steps are still active and ensure the runtime stops scheduling new work rather than letting the model negotiate another attempt.

That is the engineering bar I would use: the system can explain what was proposed, what was permitted, what was attempted and what is known to have happened. The model can change. The boundary around the business action should remain explicit.