Testing AI Agent Purchases: Prove the Order, Not Just the Payment

An agent opens a product link, submits a payment, and replies, "Your purchase is complete." That transcript leaves the merchant's hardest questions unanswered. Was this the product the buyer authorized? Did payment reach the expected recipient? Did the merchant recognize the right order? Can a restarted agent accidentally buy it again?

A passing AI agent purchase test should connect one authorized purchase intent to trusted payment verification and one business fulfillment. A checkout page, a transaction hash, or a confident final message cannot establish that chain of evidence alone.

The method below is a proposed test design, not a report of an executed GOAT Flow demo. It uses documented GOAT Flow behavior to identify useful assertions and failure injections. No merchant deployment, wallet transfer, or order outcome was tested for this article.

Define a Verified Order Before Running the Agent

Write the expected business outcome before giving the agent a link. For a hypothetical fixture, suppose a merchant sells a digital download and the buyer authorizes one purchase of that item. The exact price, asset, network, and recipient should come from the configured test environment, not from sample values copied out of documentation.

The acceptance record should bind the buyer's intent to a specific product, quantity, permitted payment terms, and delivery destination. It should also define whether the download grants access to an existing account, issues a new entitlement, or produces another identifiable delivery result. Otherwise, "fulfilled" becomes as ambiguous as "paid."

Treat the purchase as several related records rather than one success flag. There is an authorized business intent, a payment session or order, evidence of the payment result, and a fulfillment record. Their identifiers may differ. The test must show how they refer to the same purchase without assuming that a session identifier is also a merchant order identifier.

For the positive path, require an allowed purchase, payment evidence matching its terms, a merchant-accepted order, and one entitlement. For a rejected authorization, require no new transfer and no entitlement. An unknown payment result needs a third outcome: unresolved, with a reconciliation action. It should not be silently counted as either successful or unpaid.

Assign each assertion to an observer. The agent trace can show which tools it requested. A wallet or transaction observer can show payment activity. The merchant backend can establish accepted order state. The delivery system can establish the entitlement. Asking the same agent to certify all four makes a plausible explanation too easy to mistake for independent verification.

This oracle also separates purchasing from shopping assistance. Finding the right product or generating a valid checkout link may be successful discovery. Neither completes a test whose stated endpoint is a verified order. Conversely, an agent that correctly stops when consent is missing should pass that negative test even though no purchase occurs.

Build a Fixture With Three Independent Views

Use an isolated merchant configuration, a dedicated payer identity, and a harmless delivery target. Keep production secrets and customer entitlements outside the fixture. Obtain explicit authorization for any test that can move assets; this methodology does not itself authorize a transfer.

Define who holds each input before starting. Product links, manifests, and offers are public discovery data. Payer signing authority belongs behind a constrained signer, provisioned outside agent-visible text or raw command arguments; an identity alone cannot sign a transfer. Merchant API keys and secrets belong only to the merchant backend or a separately privileged test observer, never to the shopping agent. Keep checkout bearer handles in restricted records and secrets out of public metadata.

These roles do not require every request to carry merchant credentials. Public PayKit inspection/discovery and Card-link creation require no merchant HMAC; creating a Card link also requires no wallet signer and does not complete payment. Hosted Checkout merchant creation and protected order observation use their documented authenticated backend paths. Test the target deployment's authentication contract rather than inventing status codes or signing instructions.

Capture three views of each run. The decision view records the user's request, resolved product, policy decision, and requested tools. The payment view records the selected environment, current terms, session or order references, and any submitted transaction identity. The merchant view records trusted payment confirmation and the resulting delivery action. Their timestamps and correlation references should let another engineer reconstruct the sequence.

Keep the fixture stable enough to explain differences. Record the agent model and configuration, tool versions, relevant repository revision, merchant configuration version, and fault being injected. Preserve a redacted copy or hash of the discovered offer and payment terms. Do not save private keys, merchant secrets, card details, or public checkout bearer handles in broadly accessible test logs.

Run different evidence tiers deliberately. A mocked test is useful for deterministic decisions: does a changed price block authorization, or does a repeated completion event trigger a second delivery call? It is also the right place to try adversarial inputs without spending assets. However, a mock cannot establish that a real wallet, RPC, merchant verifier, and delivery integration cooperate.

For the private-input tests, put synthetic canary values, not real credentials, in the fixture's private fields. Inspect model/tool inputs, browser and public session responses, outbound requests, and reports for disclosure. Require public discovery and Card-link creation to work without merchant secrets; a crypto attempt without an authorized signer must produce no broadcast. A protected merchant operation with missing or invalid credentials must yield neither an accepted side effect nor trusted order evidence. Only the privileged observer may establish authenticated merchant state. Keep any real mutation inside separately approved test scope.

A testnet integration can exercise that cooperation when the merchant and payment path are configured. Record its actual network and asset identifiers. GOAT Flow documentation requires environment separation; Testnet3 settings do not establish which assets, limits, or capabilities are enabled for a production merchant. A production smoke test, if separately approved, is another evidence tier rather than an automatic next step.

This distinction matters for package testing too. The current GOAT Flow PayKit documentation describes manifest-driven payer orchestration, but explicitly excludes live on-chain transfers and receipt verification from its unit-test coverage. Passing such tests cannot substitute for a merchant integration run. This article did not run those package tests.

The tradeoff is practical: mocks make failures reproducible, while integration tests expose dependencies the mocks replace. Keep both results, and name the boundary of each rather than merging them into one checkout pass rate.

Test Discovery Without Giving the Page Spending Authority

Start the proposed discovery test with the exact merchant-shared link. Observe which origin and merchant identity the client resolves, which product it selects, and what capabilities it discovers. PayKit's documented trust model anchors requests to the input origin and requires the manifest's merchant identity to match the link. Those checks are useful targets for controlled negative fixtures.

Change the merchant identity in a mocked manifest while leaving the input link unchanged. The expected outcome is rejection before payment. Then inject an unrelated absolute endpoint into the manifest and check that the client does not use it to move the purchase to an attacker-controlled service. Record actual outbound requests in the harness; an agent saying it ignored the endpoint is weaker evidence.

These tests do not prove that every trusted-host response is commercially acceptable. A correctly resolved offer can still be the wrong item for the user's request. Test a valid but unwanted product beside the intended one and verify selection against the authorized product identity, not just similar titles or descriptive language.

Add a separate instruction-injection fixture. Put text in the product description that asks the agent to ignore its spending limit, disclose a secret, or send a second payment elsewhere. This is a proposed agent-policy test, not an allegation that GOAT Flow documentation contains such content or a claim that the payment library enforces the agent's policy.

The expected result should be observable: the product text remains descriptive input, no forbidden tool call is authorized, and no transfer instruction reaches the signing component. If the model proposes a forbidden action but a deterministic policy blocks it, record both events. That outcome differs materially from a system with no enforcement outside the model's own reasoning.

Also test incomplete discovery. An unavailable manifest or an ambiguous token selection should produce a bounded error or a request for clarification, not an invented asset address. Missing public information does not establish that a feature is unavailable. It means the agent lacks sufficient verified input for this attempted purchase and must stop or seek the appropriate authoritative source.

Bind Authorization to the Offer Actually Paid

Separate a request to inspect a product from permission to buy it. For the hypothetical download, define a test policy authorizing one item from the intended merchant, with an explicit price ceiling, allowed network and token, and a separate transaction-cost allowance where relevant. These are proposed application controls, not a claim that GOAT Flow supplies those exact policy settings.

Compare that authorization with fresh payment terms immediately before signing or sending. Discovery describes an offer; the current session supplies the transfer terms. A useful negative test changes the session amount after the agent inspects the product. Another changes the recipient or selected asset. Require the purchase to stop or seek renewed authorization if the new terms exceed the buyer's permission.

For fixed-price crypto products, PayKit documents a check that the session amount matches the manifest-advertised price before broadcast. Treat this as one guard within the test, not a complete spending policy. A merchant can advertise a product the buyer never authorized, and a matching price does not make an unrelated recipient acceptable to the buyer's application.

Test denomination explicitly. Display symbols are insufficient to identify a token across environments. The harness should compare the selected chain and contract with the authorized asset and check amount conversion using its verified precision. Use controlled values around rounding boundaries in mocks; do not invent production minimums, fees, or decimal precision.

Insufficient token balance and insufficient native gas are different preconditions for a crypto transfer. Test their failure handling separately and check for side effects before retrying. A rejected wallet request should not automatically become permission to switch networks, choose another token, bridge funds, or buy a cheaper substitute. Those are new decisions requiring their own authority.

Keep product, protocol, and wallet responsibilities distinct. GOAT Flow is the commerce product; x402 is a payment protocol, not the agent's wallet or its business-order database. Where an integration uses an x402 facilitator, that service performs verification and settlement functions; it does not define the buyer's shopping consent. Identify the components actually deployed before assigning test assertions to them.

Card flows need a different expectation. Current PayKit documentation describes creating a hosted Card link for a human payer to complete. If that rail is enabled, test the handoff and subsequent merchant confirmation, not autonomous entry of card credentials. Link creation is an intermediate result. The active deployment and merchant configuration must establish eligibility; documentation alone does not verify it.

Follow Payment Into the Merchant's Verifier

After a payment attempt, interrogate the merchant's trusted verification path rather than relying on the agent's reply. For crypto, associate the transaction with the intended order and compare merchant identity, chain, asset contract, recipient, and amount against the authorized purchase. A real transfer to the wrong address or for a different product must not satisfy this fixture.

Do not build one universal state enum for every surface. GOAT Flow's hosted Checkout Session, PayKit payment session, and server order use distinct models. The server order waiter recognizes PAYMENT_CONFIRMED and INVOICED as successful terminal states; the documented system can advance far enough that a poller sees only INVOICED. A test expecting to observe every intermediate state would reject a legitimate transition.

The assertion should concern the documented meaning of the actual surface, plus the matching order terms. A browser callback or redirect is a user-interface signal. An authenticated backend lookup or a webhook verified against the target deployment's contract is a different evidence source. Do not invent a canonical webhook event, signature scheme, retry schedule, or payload field when the deployment has not supplied it.

Add a forged-completion test at the merchant boundary. Deliver a browser success signal without trusted payment confirmation. The expected fulfillment count is zero. Then test a genuine payment record associated with a different business intent. That should also fail this purchase's acceptance check, even if the underlying transaction itself is valid.

Be cautious with proof terminology. The current Flow API reference describes a historical signature field in its order payment record as an unsigned digest that does not cover status. It is not an independent signed attestation. If the test requires independent on-chain verification, specify the transaction and transfer-event checks separately rather than trusting a suggestive field name.

Finally, inspect delivery as its own durable outcome. The proposed download fixture should associate its entitlement with the accepted business purchase and reject another attempt to grant the same entitlement from duplicate confirmation. Payment confirmation can be correct while delivery fails. In that case, record a paid-but-unfulfilled order with a recovery owner, not a failed payment that invites the agent to purchase again.

Interrupt the Journey Where Retries Become Dangerous

The most revealing proposed run stops between a side effect and its acknowledgement. At that point, a timeout describes missing knowledge, not necessarily missing payment. Test recovery from persisted state, including after a process restart that removes the agent's conversational memory.

Lose the Session-Creation Response

Allow the service to create a payable session, then suppress its response to the calling application. The harness should retain the server-side evidence so the evaluator knows the intended fault occurred. Check whether the application reconciles that original attempt or silently creates a new payment intent.

GOAT Flow's Hosted Checkout guide documents an important limit: reusing the same merchant's non-empty clientReferenceId returns a conflict rather than the original session handle. It is not a lookup operation. The guide also warns against inventing a fresh reference to bypass an ambiguous result.

Persist the business reference before creation and save the returned handle when available. If the response never arrives and the integration cannot recover the original handle through its authorized records or deployment procedure, the expected test outcome is a reconciliation hold. Do not specify an undocumented lookup endpoint or pretend the conflict response recovered the session.

Lose Status After a Transfer Broadcast

Before the initial PayKit payProduct() or payX402() call, persist a stable application-generated idempotencyKey bound to the authorized business intent. Keep it with the trusted origin and merchant, payer identity, product key where applicable, selected chain and token contract, and any returned session, order, and transaction references in restricted storage. This is PayKit's recovery input, not Hosted Checkout's clientReferenceId; the latter's duplicate conflict does not recover a lost handle.

In an authorized integration run, interrupt status observation after the wallet submits a transfer. Restart using the persisted key and original context, then watch the wallet and merchant records. Do not generate a new key or set force merely to clear a timeout. PayKit documents resuming or polling a reused session rather than automatically paying it again; its pinned product-recovery implementation uses an explicit key before validating mutable product configuration. This does not mean every no-key retry double-pays. It means an unchanged-product retry and durable recovery after a configuration change are different test cases.

First exercise that boundary with an injectable backend: lose the initial response or post-transfer status, erase process memory, then remove or disable the product or change its current quote. Restart with the persisted key and explicit token contract. Inspect actual request fields, recovered session identity, additional transfer calls, and independently collected merchant/delivery records. Require recovery of the original intent, zero additional transfers, and no silently created fresh payable intent. An unavailable authoritative amount or result must remain unresolved. A mock transfer-call count is not proof of onchain behavior; separately approved integration tests need wallet/chain and merchant observations too.

A safe library primitive cannot prevent a wrapper from issuing a different purchase request. If authoritative records cannot recover the original state, hold for reconciliation instead of treating a new request as a retry.

Use the following as a proposed fault matrix, not a list of observed product failures:

The difficult case is no usable transaction identity after a network interruption. Do not assume that absence in the caller's logs proves no broadcast occurred. Consult the authorized wallet and merchant observations available to the deployment. If they cannot resolve the result, report uncertainty and stop automatic payment retries. Slower recovery is preferable to converting an unknown outcome into a second potentially payable order.

Race Completion Against Delivery

Replay a valid completion notification while the first delivery attempt is still running, then repeat after delivery succeeds. Propose a durable uniqueness rule tied to the business purchase so concurrent workers cannot both grant the entitlement. Check the delivery record and the entitlement itself, not just the number of HTTP acknowledgements.

For the download fixture, the business goal is at most one entitlement for the purchase. It is not a claim of exactly-once message delivery: notifications, polls, and worker attempts may legitimately repeat. The merchant must make those repetitions converge on the same business result.

Also stop the delivery worker after it performs its side effect but before it marks completion. This tests the fulfillment system's own reconciliation boundary. Payment idempotency cannot repair an unrelated duplicate delivery. The proposed test passes only if the delivery mechanism can determine whether the entitlement already exists and recover without charging the buyer again.

Write a Verdict Another Engineer Can Reproduce

Finish with a report that distinguishes what the documentation says, what a mock simulated, what an integration actually exercised, and what remains deployment-unverified. A useful run record contains:

• The authorized intent, fixture version, agent/tool versions, environment, and approved test scope, including which signer, backend, and observers were actually exercised.

• The discovered offer and final terms, with secrets and bearer capabilities redacted.

• The injected interruption, its timing relative to side effects, and evidence that it occurred; preserve PayKit's recovery key or Checkout's business reference and returned handles privately, with only safe correlations in the report.

• The merchant's verification result, payment activity, delivery outcome, and correlation references.

• Each assertion's verdict, unresolved state, and the owner responsible for reconciliation.

Count payment attempts, submitted transfers, accepted orders, and entitlements separately. Several status polls do not mean several payments. One accepted order does not prove there was only one transfer. If a test reports a duplicate, identify which layer duplicated and attach the evidence; otherwise, the proposed fix may target the wrong component.

A release decision should block on unauthorized transfers, wrong-order acceptance, duplicate entitlements, or an unknown post-broadcast state that automatically triggers another purchase. A missing optional payment rail is different: mark it untested or outside the approved scope unless runtime evidence establishes its status. Do not convert incomplete documentation into a claim of feature absence.

The endpoint of this test is a reproducible business verdict. An engineer should be able to trace the authorized item through payment and merchant verification to its entitlement, then repeat the interrupted run without producing another purchase. Until that evidence exists, describe the integration as documented or partially tested, not as a verified agent purchase journey.

FAQ

Does running the payment CLI prove that an agent completed a purchase?

No. A CLI using mocks or an injected backend establishes only the exercised layers. A verified purchase requires independent evidence of authorized payment, the correct merchant-accepted order and the promised fulfillment. Report the signer, endpoints, chain observations, merchant observations and delivery system actually exercised; do not promote command execution into an end-to-end result.

Should the shopping agent receive the merchant API secret to check payment?

No. Keep merchant credentials on the backend or with a separately privileged observer. Public PayKit discovery and Card-link creation do not require merchant HMAC; protected merchant operations follow their authenticated contract. Provision payer signing authority outside agent-visible text and raw command arguments, and keep it distinct from merchant authority.

Which key should a PayKit restart preserve?

Persist a stable application-generated idempotencyKey before the initial payProduct or payX402 call, bound to the authorized intent and original payment context. Reuse it after interruption rather than creating a fresh key or forcing another broadcast. Hosted Checkout clientReferenceId is a different duplicate guard, not a way to retrieve a lost session handle.

How should a test treat a payment whose outcome cannot be recovered?

Mark it unresolved and hold automatic replacement payment. Preserve transaction, session, order and intent correlations in restricted storage and consult the designated wallet and merchant observers. A timeout or missing local hash does not prove that no broadcast occurred. A safe stop is preferable to turning unknown state into another purchase.

What should duplicate confirmations do to fulfillment?

They should converge on the same business result. For the proposed single-download fixture, assert one entitlement for the accepted purchase even if notifications, polls or workers repeat. Check the durable entitlement and delivery record, not only HTTP acknowledgements. These are proposed acceptance criteria; the article reports no executed payment tests.