A field journal of third-party API failures, rate limits, breaking changes, and the postmortems teams wish they'd written sooner.

Why Your Stateful Integration Tests Fail Nondeterministically and How to Own the State Machine

Flaky integration tests signal unowned state, and a simulator fixes it.

· 9 min read
Cover illustration for “Why Your Stateful Integration Tests Fail Nondeterministically and How to Own the State Machine”
Integration Testing · November 11, 2026 · 9 min read · 2,022 words

Integration testing against stateful APIs requires control over the state machine as well as predictable responses. Use an isolated stateful simulator that starts from known data, enforces legal operation sequences, and resets after each test, with behavior verified against the real API before each release. Nondeterministic failures usually expose a gap in that ownership, while fault injection and time control extend it to failure paths.

Why integration tests fail without any code changing

A test passes locally, fails in CI, then passes on rerun without a code change. Nothing in the assertion explains the different outcome. That lack of repeatability defines nondeterministic failure, and it is distinct from an assertion that consistently identifies a defect. The application may be stable while the test signal is not.

The usual response is to rerun the job. That gets a pipeline moving but leaves the cause in place, costs more to run, and teaches engineers to pay less attention to failures. Once that habit spreads, genuine regressions can be dismissed as flakiness. Teams then fall back on manual verification, surrendering the speed that automated integration tests were supposed to provide.

What CI environments do to test determinism

CI changes the conditions under which tests execute. Ephemeral runners reset local state between jobs, while drift comes from differences between environments. Parallel execution exposes hidden assumptions about which operation finishes first. Shared infrastructure adds resource contention and variable latency, and distributed dependencies amplify those differences across service boundaries.

These effects do happen, but they do not excuse poor tests. A sequence that appears reliable on a developer’s machine may execute differently when several jobs compete for resources. Timing can therefore explain why a failure appears in CI but not locally. It does not, by itself, explain why a different schedule changes the correctness of a stateful workflow.

Unowned shared state is the root cause, not the environment

Several tests or CI jobs read and write the same staging database, sandbox account, or external API surface, but none controls what the others leave behind. Their outcomes then depend on execution interleaving, making them unreliable indicators of application correctness. That dependency remains even with faster runners or another retry.

Consider Test A, which creates a record, and Test B, which expects no records to exist. If they share state, B’s result depends on whether A has already run. The assertion can be perfectly reasonable in isolation. It becomes unreliable because the suite does not provide the baseline that B assumes. Calling this just a timing issue hides the lack of isolation.

The distinction changes the direction of diagnosis. Looking at runner performance may explain the order in which operations ran, but looking at state ownership explains why that order changed the result. Retries change the schedule and may accidentally produce a passing interleaving. They do not give either test exclusive control over its assumptions.

The state machine mental model: what integration tests are actually exercising

A payment charge makes the model concrete. It moves through a lifecycle of created, authorized, captured, and refunded, with each operation depending on the state established by earlier calls. A refund before capture is not equivalent to a refund afterward. An integration test checks those relationships as well as whether the endpoints return valid response bodies.

The model has three essential properties. Named states and legal transitions describe how a resource changes, while guards reject operations that are invalid in its current state. Deterministic reset restores a known baseline before another test begins. Together, these properties make the sequence observable and repeatable. The response to a request must reflect the resource’s state, rather than a success payload selected independently of its history.

A static mock returns one response per endpoint, while a conditional mock chooses a response from the request’s shape. Neither necessarily remembers prior operations. A stateful simulator holds data between requests, so a created resource can subsequently be listed, updated, and deleted. Without that memory, a mock can return a captured charge before capture has occurred, or return a customer that an earlier creation call never actually established.

What happens when you test against a state machine you don't own

Order dependence is the first operational consequence. Tests get records and mutations left by earlier tests, so a change in execution order changes what their assertions find. This calls for isolated state spaces and initialization that sets a known starting state. A better assertion alone cannot supply the missing baseline.

Tests calling live APIs inherit rate limits, transient outages, changing response formats, and network delays. A failure may therefore describe the dependency’s condition independently of any regression in the application. Controlling the test’s state is necessary, but repeatable execution also requires control over the dependency behavior presented during that test.

Provider sandboxes keep more testing away from production, but they do not fully match it. Stripe’s sandbox payments do not go through real card networks or payment providers. PayPal’s Braintree GraphQL integration guide states that sandbox test values do not work in production. Apple’s sandbox simulates successful payments by returning transactions. These environments serve useful testing purposes, but their results do not prove that every production behavior was exercised.

Those limits can also constrain fixture collection. ArtisanPack-UI’s ecommerce-paypal project documented in issue #27, November 2026, what its live sandbox could not reach, including PayPal with a real buyer, while building a CI fixture corpus from live webhook deliveries. A sandbox can catch some contract drift. It does not substitute for a controlled stateful environment, and a controlled environment still needs verification against the API it represents.

How to test stateful APIs through isolation, initialization, mutation sequencing, and teardown

The practical response is to establish ownership at four points, beginning with isolation. Each test or job needs its own state space rather than concurrent access to one staging database or sandbox account. Ephemeral, per-job environments provide that boundary. Within a job, parallel branches also need independent state, because job-level isolation alone does not prevent tests inside the job from interfering with one another.

Initialization then sets a known starting condition for each test. That condition should include realistic data where the workflow requires it, rather than an empty environment that makes every list query return nothing. Empty results cannot show whether the application can handle realistic data shapes and volumes. A simulator with pre-seeded data can supply that baseline without requiring every test to construct the entire environment from scratch.

Mutation sequencing makes the workflow itself testable. The test should drive a resource through legal transitions and assert the documented rejection when it attempts an illegal one. A refund-before-capture test should check that the refund is rejected and fail if it gets a hardcoded success response. For mutable collections, assertions should check invariants such as record existence and correct totals rather than positions that change when insertion order shifts.

Teardown completes the ownership contract after each test. The beforeEach and afterEach hooks, or their equivalents, need to cover database records, file system artifacts, session state, and in-flight asynchronous operations. Cleanup cannot belong to a later test. If one test depends on another to remove its state, the two remain coupled even when they usually run in a convenient order.

At the API boundary, these requirements favor stateful simulation over stateless mocks or recorded cassettes. VCR-style cassettes preserve one execution path at one moment, but playback does not hold mutable state that enforces a resource lifecycle. A simulator can instead make what a test creates available to later reads and mutations. Isolation alone is not sufficient, however. Verification against the real API before every release is what exposes divergence before stale simulated behavior reaches the test suite.

A provider’s approved test environment is still useful for the tests it was built to support. Kalshi has a demo environment with mock funds and a separate Trade API root where order workflows can be tested before touching live positions. Its credentials are not shared with production. A stateful simulator gives tests isolation without relying on the network, while verification checks whether the order workflows it models still match the API.

Fault injection and time control as first-class ownership concerns

Ownership must extend beyond successful resource transitions. Rate-limit responses, service errors, and degraded latency are part of how an API behaves, so tests need a way to cause them on purpose. A simulator that stays healthy cannot by itself show that the application handles 429s or 503s correctly. At the simulator layer, a test can specify the fault, receive it reproducibly, and assert the application’s response without waiting for a provider or network to fail.

Behavior that depends on time needs the same level of control. Webhook retry windows, session expirations, and rate-limit resets cannot be exercised quickly against a live system merely by sending requests faster. PayPal retries failed webhook deliveries up to 25 times over three days. Stripe retries undelivered webhook events for up to three days. Testing those windows requires either waiting for real time to pass or using time-control tooling, making controllable time a practical integration requirement.

Agent testing exposes the same need for reproducible faults. AgentChaos, in August 2026, used an HTTP-layer wrapper to inject faults into LLM API responses across multiple agent architectures and found substantial drops in pass@1 rates. The result grounds fault tolerance in testable behavior. Resilience testing therefore belongs in the controlled environment itself, rather than being deferred until after successful workflows appear reliable.

AI agents and RL environments: the same ownership problem at higher stakes

An agent adds variability above the API state machine. Its tool-use decisions can change between runs even when the code and API baseline remain identical. If the external state also changes, a failed evaluation cannot cleanly distinguish model variance from an integration defect. Binary pass-or-fail rules built for deterministic software can conflate those two causes, obscuring diagnosis.

Each evaluation needs isolated state, explicit initialization, and deterministic reset between episodes. This holds one source of variation constant so that differences in behavior can be interpreted against a repeatable environment. Without that control, a change in the starting state may look like a change in how well the agent performs.

Reinforcement learning multiplies the isolation requirement across parallel rollouts. Each episode needs its own stateful environment because shared mutations corrupt the reward signal and produce untrustworthy policy gradients. Kubernetes-sigs’ agent-sandbox project targets high-throughput isolated execution for training and evaluation loops, including software engineering benchmarks, R2E-Gym, and Ray/RLlib. Execution isolation sets that boundary, but the workflows inside it still need stateful API ownership.

A practical audit: four questions to assess whether you own your state machine

The audit begins with a question about isolation. Can two CI jobs run this suite concurrently without interfering with each other? A no or an unsure answer points to shared state that the jobs do not have sole control over. The same question applies to parallel tests within each job.

The second question concerns the starting condition. Does every test begin from an explicitly defined, known state rather than inheriting whatever the previous test left behind? A test that requires an earlier test to run is order-dependent. Realistic seeded data can provide that baseline if each run gets the starting state it needs, without mutations left over from past runs.

Does the simulation return the errors documented for invalid state transitions? Success responses alone cannot establish that a workflow is correct. The environment must distinguish an operation that is legal now from one that would become legal only after an earlier transition.

Does every test clean up all the resources it creates, including session state and in-flight asynchronous operations, before the next test begins? Teardown is part of the contract. A test that leaves its cleanup to another test has not completed its ownership obligations.

For API simulators, one further dimension qualifies those answers. Is the simulator verified against the real API before each release, or is there an undated gap between modeled and live behavior? The slack-web-api-client weekly drift checks address that kind of gap. Four affirmative answers establish control over test state, while drift verification connects that controlled model to the API the application must actually use.

Sources

  1. Detect & Fix Flaky Tests in CI/CD Pipelines

More in Integration Testing