Idempotent Agent Provisioning API Design
Retrying agent calls duplicates infrastructure and bills unless APIs are built idempotent by design.
Advertisement
LLM agents retry tool calls 15–30% of the time, driven by timeouts, validation errors, or model uncertainty. Whatever the exact figure, the pattern behind it is what matters: retries aren't rare exceptions in agent workflows, they're baked into how these systems operate. Most teams still build provisioning like a normal basic data-operations endpoint, and that choice is the actual bug. Once the retried action spins up infrastructure and bills a customer, the mistake stops being theoretical and starts showing up on an invoice.
The core mechanism is a PUT endpoint keyed on the customer identifier, not a POST. PUT /agents/{customer_id} is defined as idempotent by the HTTP specification, so a retry against the same URI with the same body is a no-op from the protocol's point of view: the server returns the existing resource rather than creating a second one. When PUT is not available due to platform constraints, the caller generates a stable UUID before the first attempt and sends it as an Idempotency-Key header on every retry; the server checks a persistent store for that key before running any provisioning logic, and on a cache hit returns the stored response unchanged. The server must also enforce a unique constraint on customer_id at the database level and wrap the check-and-create step in a single transaction or distributed lock, so two concurrent calls racing through the same existence check cannot both pass it at once. A fresh creation returns 201; a call that finds the resource already present returns 200 with the same body; if provisioning is still in progress, 202 with a job ID and polling URL gives every subsequent caller, retry or concurrent request, a single point to converge on.
Here's the mechanical reason agents retry so much more than a person would. A human hitting an error reads it, checks the docs, maybe pings a colleague, then decides what to do next. An agent has none of that. It operates in-band, meaning everything it knows about whether an action worked comes from the API response itself, per Pinecone's engineering analysis of agent behavior. No response, or an ambiguous one, and the agent's only rational move is to try again. There's no side channel for "let me go find out what actually happened."
Provisioning sits right in the blast radius of this problem. Reading a record twice is harmless. Provisioning an agent twice is not, since it spins up infrastructure, allocates credentials, and creates a resource that persists and costs money. A duplicated POST /provision call doesn't leave behind a confusing log line somebody notices next week. It leaves behind a second tenant, a second credential set, a second bill.
Picture the actual sequence: a network timeout hits mid-request, and the caller has no way to know whether the server finished the job before the connection dropped. Did the agent get created? Unknown, and unknowable from where the caller sits. So it retries, because retrying is the only move on the table. Multiply that by machine tempo, agents that retry aggressively, run in parallel, and act with nobody watching, and a glitch that would cost a human ten seconds of confusion turns into something that duplicates itself instantly and silently, at scale, before anyone notices.
Postman's 2025 State of the API Report found that 89% of developers use AI in their daily work, but only 24% design APIs with AI agents in mind. Most APIs in production were built on the assumption that a human reads the error message and decides what to do. Agents don't read; they act. That gap, between an API built for a reader and a client that only acts, is exactly where duplicate provisioning lives. No amount of client-side retry logic closes it. Only an API built to be replayed safely does.
What idempotency actually guarantees, and what it does not
Idempotency has a precise meaning: an operation is idempotent if running it once or running it ten times leaves the system in the same state. "Set agent status to ACTIVE" is idempotent. Run it a hundred times, the agent is still ACTIVE. "Increment the agent count by one" is not. Run it a hundred times, and now there are a hundred agents that shouldn't exist.
People lump three other ideas in with idempotency, and all three miss the mark. Safety isn't the same thing: a read operation is safe because it doesn't change anything, and reads happen to be idempotent too, but that overlap is a coincidence, not a rule. Atomicity is about whether a transaction completes fully or not at all; it says nothing about what happens on a retry. At-most-once delivery is a different strategy entirely, one that tries to stop the retry from happening in the first place instead of making the retry harmless once it does happen.
What idempotency actually promises is narrow. Call it twice and you get the same resource, the same ID, the same state, and the same response body as the first successful call.
What it does not promise is ordering, freshness, or eventual success. A timeout is still a timeout. Idempotency doesn't make the request succeed, it just makes it safe to try again without making things worse.
Skip this discipline and the failure looks exactly like what commerce systems learned the hard way years ago. A timeout on "create order" leaves the cart in an unknown state, the agent retries, and now there are two orders and two charges. Provisioning fails the identical way: duplicate resources created, duplicate billing records, duplicate credentials for the same customer.
The fix isn't a patch on top of the endpoint, it's a different model underneath it. Design around a state, not an event. Don't build "create agent," build "ensure agent X exists for customer Y." One is a command that fires once and can misfire twice. The other is a fact that's either already true or gets made true, and calling it again changes nothing.
Why does PUT work where POST creates duplicate provisioning problems?
POST is the wrong default here, and there's no case where it's the better one for provisioning. The HTTP specification says POST is not idempotent, and a second POST to /agents is a second creation as far as the protocol is concerned. Every proxy, cache, and load balancer in the request path treats it that way, whether or not that's what anyone intended when they wrote the handler.
PUT /agents/{customer_id} is the shape that actually works. PUT is defined as idempotent: send the same PUT to the same URI twice, get the same result both times. Putting the customer ID directly in the path does double duty, naming the resource and pinning down its identity so it stays stable across retries.
The contract is specific: a repeated PUT with the same body has to be a no-op from the caller's point of view. Whether the server just created the resource or found it already sitting there, it returns the same representation either way. That symmetry is the entire point.
There's a bigger design move available too. Instead of exposing every step of provisioning (spin up a sandbox, assign credentials, register integrations, write a billing record) as separate calls the agent has to orchestrate by hand, collapse it into one endpoint. PUT /agents/{customer_id}, with a documented postcondition: agent exists, is active, is reachable. The agent shouldn't need to know the internal provisioning steps, just what true looks like when the job's done.
On the response side, both outcomes deserve a clear signal. A fresh creation returns 201 with the full resource. A call that hit an already-provisioned agent returns 200 with the same resource. A header or field can flag which case just happened, giving the caller useful information without changing what the state actually is.
Sometimes POST can't be avoided, a platform constraint, a gateway that won't route PUT the way it needs to. Fine, but then the safety net has to move somewhere else. That's where idempotency keys come in.
Idempotency keys: the mechanism that makes POST safe for provisioning
The pattern, laid out well in fast.io's idempotency guide, comes down to four moving parts.
The caller generates a stable UUID at the moment it decides to provision, not at the moment it sends the HTTP request. That distinction matters: the key has to survive every retry of this same intent, so it can't get regenerated fresh each time the request goes out.
That key travels in a header, something like Idempotency-Key: <uuid>, on every attempt tied to this operation, retries included.
The server checks a persistent store for that key before it touches anything else. Key already seen? Return the stored response, don't run provisioning logic again. Key is new? Proceed, then store the result.
The server holds onto that key-to-response mapping for a documented window, long enough to outlast any retry that's likely to happen in practice.
Where the key gets made matters as much as the pattern itself. It has to come from the caller, before the first attempt goes out, never derived from a response and never generated by the server on receipt. A reasonable seed is something deterministic relative to the operation itself, customer ID plus operation type, so an agent generating keys programmatically doesn't collide with itself by accident.
What gets stored matters more than people usually assume, too. Not a flag saying "already done," the full response: status code, headers, body. A cached 201 with the original resource is what makes the second call trustworthy. Store anything less and the caller's just getting a shrug.
TTLs need real thought here, and they run longer than people expect from other domains. A payment retry might come seconds later. A provisioning retry might come hours later, an onboarding webhook firing twice isn't unusual, and if the key's already expired by then, the whole safety net is gone. Document the window as part of the API contract, not as an implementation detail buried in a runbook somewhere.
Stripe's idempotency key design gets cited constantly as the reference implementation, and for good reason: keys passed as headers, stored server-side, and on replay, even a replay of an error, the original status code and body come back exactly as they were the first time.
Scope the key to the operation, not the session. A key minted for "provision agent for customer 42" shouldn't get reused for "update config for customer 42." Document the namespacing rule explicitly, because an agent generating keys on its own logic will find every gap in that rule eventually.
State reconciliation when the first call's outcome is genuinely unknown
Idempotency keys solve the case where the server got the request. They don't solve the case where it never arrived, or where the key store itself failed somewhere between receiving the call and writing it down. The caller is still stuck not knowing, and no amount of header discipline fixes that gap.
One layer of defense: check before writing. The agent calls GET /agents/{customer_id} first, and only provisions if nothing comes back. That makes the whole workflow declarative, not just the individual HTTP call but the sequence of steps the agent follows.
It's not enough on its own, though. There's a gap between the GET and the write, a window where a second concurrent call can slip in and also pass the "doesn't exist yet" check. Two callers, same customer, both racing through the same gap. State-check-write has to sit alongside server-side duplicate detection, not replace it.
The deeper fix lives on the server: model provisioning as a desired-state write, not an event to record. "Ensure agent for customer_id=42 exists in state=ACTIVE." fast.io's idempotency guide describes this approach: the server reconciles what currently exists against what's supposed to exist, and it does that reconciliation atomically, regardless of how the HTTP layer above it is shaped.
Atomically here means something concrete: the check-and-create step happens inside a single transaction, or under a lock keyed on customer_id, so two concurrent calls can't both pass the same existence check at the same moment.
The response back to the caller also needs to distinguish three separate outcomes, because each calls for a different handling path. The agent was just created. The agent already existed, matching the target state. Or the agent already existed but in a conflicting state that needs actual resolution. Collapse those three into one generic "success" and the caller loses information it needs to act correctly.
Error response contracts that let agents recover without human intervention
An agent hitting an error can't file a ticket. It can't ask a coworker what the message means. Every round trip through ambiguity consumes real resources, so an error response that doesn't say anything useful isn't just bad API design, it has a measurable cost attached to it, per Pinecone's analysis.
A generic {"error": "validation failed"} gives an agent nothing to work with. Zuplo's readiness gap guide makes this point directly: that response can't be parsed into a retry decision, a fix, or a clear stop signal. It's a dead end dressed up as a response, and treating it as good enough is exactly the mistake to avoid.
RFC 9457 Problem Details gives a structural baseline worth adopting wholesale. A stable, machine-parseable error code instead of a prose sentence. A human-readable explanation of what went wrong and why. Field-level detail naming the exact field, the constraint it violated, and ideally the fix. And a recovery hint: retry with backoff, correct this field and resend, or stop, this one's terminal.
Provisioning has its own error shapes worth naming outright.
ALREADY_EXISTS reflects a safe outcome and should be handled in a way that lets the agent continue, not stop cold. PROVISIONING_IN_PROGRESS is transient, so the response should signal that clearly and give the agent a path to check back. QUOTA_EXCEEDED is terminal for now. The response should identify what limit was hit and how the caller can address it. INVALID_CUSTOMER_ID is terminal too. The response should name the field and what was wrong with it. No retry makes sense there, and the response should say so directly rather than leaving the agent to guess.
Whether an error is retryable should be communicated clearly in the response. Don't make the agent guess retry-worthiness from the status code alone; spell it out.
Idempotency key conflicts need their own handling too. If a key gets reused with a different request body than the one it was first bound to, the response should explain that the key is already bound to the original request and a fresh operation needs a fresh key.
And the lifecycle itself deserves explicit states, PENDING, PROVISIONING, ACTIVE, FAILED, rather than cramming undocumented meaning into a single overloaded status field.
Concurrency, locking, and the multi-caller provisioning scenario
Multiple callers hitting the same provisioning path at once isn't an edge case worth a footnote, it's routine. A webhook fires twice because that's just how at-least-once delivery works. A frontend and a backend both fire a provisioning call on a customer's first login. A retry sitting in a dead-letter queue gets replayed at the same moment a fresh call comes in from somewhere else. All of this happens in production, regularly, not as some rare storm of bad luck.
The fix is a lock, keyed on customer_id, acquired before any side effect runs: a distributed lock or a database transaction backed by a unique constraint on that same customer_id. Two concurrent calls need to serialize through that lock, not race each other to the finish line.
Even with locking in place, put a unique constraint on (customer_id, agent_type) at the database level as a backstop. If the locking logic has a bug somewhere, that constraint is what stops a duplicate row from ever landing.
What does the losing side of that race get back? Not a 500, and not a silent, confusing success either. A 200, with the resource that already got created by whichever call won. The loser in a provisioning race should look, from the outside, exactly like a normal idempotent retry. No special casing, no different response shape.
fast.io's advisory file-locking model for storage operations is a useful analogy here: a lease gets taken before a write, and anyone else trying to write gets a 409 back, then waits or retries. The same idea applies at the API layer, just with the lock sitting on the customer_id resource instead of a file.
And if provisioning genuinely takes more than a couple seconds, which spinning up infrastructure usually does, the first successful call should return 202 Accepted with a job ID and a polling URL. Every other call touching that same customer_id, retry or fresh concurrent request, gets back that same job ID. The job ID becomes the one place everything converges.
Credential and identity scoping that makes per-customer isolation durable
Idempotency and isolation are two separate contracts, and both have to hold at once. A provisioning call that's perfectly safe to retry but hands out shared credentials across customers is still broken. It's broken in a way that won't show up in a retry test, which is exactly why it slips past most reviews.
Every provisioning call should mint credentials scoped to exactly one customer's context, never a shared key with a tag slapped on identifying which customer it's "for." That distinction matters more than it sounds: the permission boundary itself needs to be the customer, not a label sitting next to a permission boundary that's actually much wider.
Those credentials also need a full programmatic lifecycle: issued via API, rotated without downtime, expired on a schedule that's written down somewhere. OAuth flows that need a human to click through a redirect screen don't belong here at all. An autonomous agent has no browser tab to redirect and no human standing by to approve it, a point Zuplo's guide raises directly.
Zuplo's consumer metadata pattern is a solid reference for what this looks like in practice: consumers tagged consumerType: "ai-agent", keys issued with expiration built in from the start, per-consumer rate limits, the whole lifecycle managed through a Developer API rather than a manual dashboard click.
There's a broader identity gap sitting underneath all of this, and it's worse than most teams assume. MCP and A2A define how agents talk to each other; they say nothing about who an agent actually is. A Knostic scan of roughly 2,000 MCP servers found every single one lacking authentication, according to the AIP paper (arxiv 2603.24775, March 2026). Provisioning an agent without solving identity means the thing that just got created can be reached by any caller, and nobody's verified whether that caller has any right to be there.
Least privilege belongs in the provisioning contract itself, not bolted on after the fact. The credentials handed out should cover exactly the tools and data this customer's agent needs, nothing wider, with RBAC applied at connection time and enforced again on every single tool call, a model Composio's approach reflects.
Credential issuance needs the same idempotency discipline as everything else here. Retry a provisioning call and the same key ID with the same scopes should come back, not a fresh key that leaves the old one orphaned somewhere with nobody tracking it.
Putting the contract into an OpenAPI specification the provisioning agent can read
None of this holds together if it only lives in a wiki page or a messaging-app thread from six months ago. An agent doesn't read prose documentation the way a person does. It reads, or gets fed, a machine-parseable contract, and OpenAPI is where that contract has to live if any of the above is going to get enforced rather than just remembered.
The PUT /agents/{customer_id} endpoint needs its full response set specified: 201 for a fresh create, 200 for an idempotent hit, 202 for provisioning still in progress, each with the exact schema of what comes back. The Idempotency-Key header belongs in the spec as a required parameter, not an optional afterthought buried in a changelog. Error responses should follow the RFC 9457 shape consistently across every failure mode named earlier: ALREADY_EXISTS, PROVISIONING_IN_PROGRESS, QUOTA_EXCEEDED, INVALID_CUSTOMER_ID, each with its own documented schema and a clear signal about whether retrying makes sense.
Lifecycle states deserve an explicit enum in the spec, not a free-text string field that drifts every time someone on the backend adds a new internal status without updating the docs.
Written this way, the specification stops being a reference for a developer reading it before writing code by hand. It becomes the input an agent parses at runtime to decide what a safe retry looks like, what a terminal error means, and when the job's actually done. Provisioning stays correct not because someone read the docs carefully once, but because the contract itself leaves no room for a duplicate tenant to slip through.
