Stateful vs. Stateless API Mocking in Integration Tests
Stateful simulators catch integration bugs that stateless mocks let slip through.
Integration testing against stateful APIs requires an environment that preserves changes across calls, so tests can verify a workflow’s cumulative result without hitting production. A stateful simulator provides that environment, while stateless mocks serve isolated request-and-response checks and live sandboxes provide final provider validation. The distinction is structural. Stateless mocks enforce consistency through configuration.
Why integration tests exist and what they need to prove
POST a customer, POST a subscription referencing that customer, then GET the subscription and verify that the customer is on it. Each step depends on the previous result. The create must succeed, the relationship must persist, and the read must materialize that relationship. A correct response from any one endpoint cannot establish that the whole sequence worked.
This is the standard an integration environment must meet. Unit tests can isolate a function and check its response to a known input, but integration tests examine dependencies between operations. If the subscription references a customer that was never stored, individually plausible responses conceal a broken workflow. The distinction changes what the test environment must do, because returning valid response bodies is not the same as enforcing the relationships those bodies describe.
What a stateless mock server actually does — and what it cannot do
A stateless mock server matches incoming requests to configured rules and returns the mapped responses. It holds no memory between calls. In the customer-and-subscription example, the customer returned by the first call and the subscription returned by the final call can agree because the author configured them to agree. That agreement proves nothing about whether a create operation persisted data or established a relationship.
A four-step payment flow provides a practical litmus test. First, POST /payments returns 201 with ID pay_xyz and status pending. Next, GET /payments/pay_xyz returns 200 with the same ID and status. Then, POST /payments/pay_xyz/capture returns 200 with status captured. What does a final GET /payments/pay_xyz return? A stateless mock returns whatever was configured for that route, which may still be pending, because capture did not change a stored payment.
Stateless mocks are useful for checking URLs, headers, specific error responses, and individual functions without involving a dependency. They also support frontend development against an agreed contract and stubbing dependencies without a public spec. Those tasks need controlled responses. The mismatch begins when a test treats a configured response as evidence that an earlier operation changed the system.
Why mock-based integration tests pass and then production fails
A pipeline mocks POST /v1/payment_intents, asserts 200, and permits a merge, but the first real checkout in staging fails because the handler never stored the PaymentIntent ID. The next step in the webhook path reads undefined. The mock supplied success without testing the dependency. Even careful test authors can miss this failure if their assertions stop before the state-dependent step.
A configured response cannot establish that a real API accepted a relationship, that a create changed a list endpoint, or that a downstream read reflects a mutation. A hardcoded capture response also cannot prove that authorization happened first. Such tests can remain green until deployment because the environment never enforces the skipped constraint. The production failure is therefore not a contradiction of the test result, but evidence that the test proved less than its name implied.
What stateful simulation adds and how it closes the gap
A stateful simulator stores resources and applies operations to them. A POST creates stored data, a subsequent GET retrieves it, and list endpoints include it. PATCH changes that resource. State machines check that transitions are valid, and state changes can trigger webhook events. In the payment litmus test, the final GET returns captured because the capture operation changed the stored payment.
The simulator can check that the referenced customer exists and make the relationship available through the subscription read. Consistency follows from behavior rather than matching fixtures. That lets the test exercise relationship integrity and create-then-read persistence, while pagination can be tested against the resources actually created. A missing resource or an incorrect relationship can now affect later calls.
The same mechanism supports checks on duplicate POSTs and webhook handlers. A test can compare the resource returned by a repeated operation with the existing resource and check that an event follows the corresponding state change. However, a “sandbox” label does not establish these capabilities. The four-step payment test distinguishes a stored mutation from a configured reply without relying on product terminology. Statefulness must be demonstrated through dependent operations.
What provider-supplied sandboxes actually give you — and where they stop
Provider sandboxes offer genuine stateful behavior, but provider-supplied testing products are not interchangeable. Stripe’s stripe-mock checks that POST data is valid and puts valid input parameters in the response to that call where names and types match. It does not persist that data for future requests. Stripe’s live sandbox is a different environment. The stateless behavior of stripe-mock should not be attributed to it.
PayPal’s webhook simulator has a separate limitation. PayPal states that its simulated events are for demonstration, are not associated with any app, and do not appear in the dashboard event viewer. A demonstration event is therefore not evidence of an app-linked resource transition. PayPal also retries failed webhook deliveries multiple times over several days, creating retry and deduplication responsibilities that a static event fixture does not exercise.
Live sandboxes have limits on how they can run in CI even when their resources behave as the test needs. They need networks and provider accounts, have rate limits, and may be shared between test runs. Provider credentials in CI runners widen the attack surface. The test cannot control event ordering or retry timing, and it cannot run offline or in an air-gapped environment. These constraints make live sandboxes useful complements to a local, deterministic simulator rather than substitutes for it as the primary CI environment.
How drift quietly invalidates mocks over time
Accuracy also has a time dimension. A hand-maintained stub can stop matching the real API as soon as that API changes, without producing any signal inside the test suite. Its responses still agree with each other because they come from the same configuration. Tests therefore keep passing against an obsolete contract, and the discrepancy may be visible only in production. Initial fidelity provides no ongoing protection.
Spec-derived mocks move part of this responsibility to the OpenAPI document. Their accuracy follows the specification, so a stale or incomplete document still produces an incomplete mock. An undocumented endpoint may yield a visible 404, but schema-valid generated data can remain semantically weak. A string placeholder is not a plausible customer name. Derivation reduces manual duplication without proving that the resulting environment represents current behavior.
The defense is continuous verification against the real API before every release, with the results published. One-time calibration is not enough for this purpose. A stateful simulator also needs this discipline, because retaining state does not establish that its transitions match the provider’s current rules.
Fault injection and resilience testing as a stateful concern
A successful workflow does not establish resilience under partial failure. Slow responses, dropped connections, 429s, and out-of-order webhooks change how operations unfold across time. An isolated error response can test classification, but recovery depends on what earlier attempts did. Retry logic, circuit breakers, fallback behavior, and error propagation therefore need tests that preserve the sequence’s context. A naive simulator that stores resources but cannot control failures leaves this dimension untested.
Idempotency makes the requirement especially clear in practice. A retry returning success is insufficient evidence if the test cannot check for a second resource created by the operation. A stateful fault-injection test can return a 429 with retry guidance, succeed on the next call, and preserve enough context to examine the outcome. The assertions cover status classification, delay bounds, attempt limits, the final result, and the absence of duplicate creation. Scripting an error followed by success cannot prove that no duplicate was created.
Stripe documents that events can arrive out of order and deliveries can be duplicated, so handlers must still work correctly when events are repeated or reordered. A configurable stateful environment makes those sequences reproducible in CI. It can also support pre-production load tests through artificial latency, variable payload sizes, and concurrent requests. Those controls exercise timeout cascades and throughput limits without directing load at rate-limited provider services.
How AI agents expose the same stateful gap at a larger scale
AI agents that call Slack, GitHub, payment processors, or databases face the same dependency problem. Their multi-turn workflows amplify it because later tool calls depend on resources created or changed earlier, while LLM behavior adds variability. A resource created in turn 1 must remain available when read in turn 3 and updated in turn 5. Fixed responses cannot establish that continuity. Live API calls avoid that particular illusion but introduce slow, expensive, non-deterministic execution and potentially irreversible actions, including real charges.
For deterministic middleware tests, a stub LLM can return tool calls keyed on prompt content while real middleware executes against a stateful simulator. This exercises tool execution without substituting canned results for the entire workflow. It does not evaluate the real model’s decisions. For MCP-based tests of actual tool-use behavior, the agent can use a real LLM and a real MCP server, with the server calling simulated third-party APIs.
That boundary allows pagination, rate limits, error envelopes, and auth refresh logic to be tested without live third-party credentials. Agent-training environments require the same persistent foundation at greater scale. They need deterministic, stateful environments that scale horizontally across parallel episodes, because reward functions evaluate the cumulative outcome of an action sequence. Surge AI’s EnterpriseBench includes Corecraft, an enterprise RL environment simulating a customer support organization across 14 entity types and 23 unique tools. That scope makes linked state a prerequisite for evaluating multi-step work.
How can you choose the right environment for integration testing against stateful APIs?
Does the test need to verify cumulative state across more than one call? If not, a stateless mock is correctly scoped for checking function behavior, request parameters, headers, or a known error response. It can also unblock frontend work or stand in for a dependency without a public spec. There is no benefit in requiring persistent resource behavior when the assertion does not examine it.
If later assertions depend on earlier mutations, the local environment needs a stateful simulator. That includes create-read-update flows, list membership, authorization before capture, state-triggered webhooks, and idempotency under retry. Agent workflows need it when resources must still be there for later turns. For these tests, the selection criteria are persistence, relationship integrity, valid transitions, controllable failures, and current verification against the provider. A successful HTTP response alone does not satisfy those criteria.
Live provider sandboxes serve a different purpose. They are appropriate for pre-release checks of behavior that only the provider can confirm, including Stripe webhook retry timing or PayPal event delivery within a controlled window. They complement local tests on a slower schedule.
The resulting CI structure assigns each environment a defined role. PR pipelines use stateless mocks for unit tests and local stateful simulation for integration tests, keeping those runs fast and free of live provider credentials. Merge or scheduled pipelines extend simulation to broader contract and multi-system checks. Pre-release pipelines use live sandboxes for final production-like validation. The point of graduation is precise, because a test needs stateful simulation when its result depends on what a previous call actually changed.
