Isolating Stateful API Behavior Across Parallel Test Workers
Namespacing state per worker lets parallel tests verify stateful API behavior without interference.
Integration testing against stateful APIs requires a simulator that preserves call history without sharing test state across parallel workers. Each worker needs its own world, provided by a private namespace or a separate simulator instance, with deterministic resets between test cases. These boundaries make ordered workflows and fault recovery testable without hitting production, while continuous verification keeps the simulated behavior aligned with the real API.
Why stateful simulation breaks when tests run in parallel
Worker 2 resets shared data while Worker 5 is asserting that a newly created record appears in a list. The assertion fails, but the application may be correct. The simulator has allowed one test to erase another test’s prerequisites. Without isolation, adding realistic state creates failures that are neither reproducible nor attributable to the code under test.
Yet removing state would remove the behavior the test needs. A stateless mock returns a canned response regardless of call history, so it cannot establish that a created record appears in a subsequent list or that a deleted message disappears on refresh. Those assertions require a history of changes.
Parallel execution changes who can alter that history. A shared simulator gives every worker access to the same world, even though each test assumes control over its own fixtures. Forcing the CI worker count to one avoids concurrent interference, but sacrifices parallel throughput without repairing the design. The necessary boundary is around the state.
What "state bleeding" actually looks like across parallel workers
The msw-terrarium case study documents this failure in the nextcov-example Next.js Todo app. Every Playwright worker used a single json-server instance, and the README acknowledged that state-aware tests “can only be run serially.” CI therefore forced a single worker. The original app also skipped “list when empty tasks” because an empty-bucket assertion could not safely coexist with other workers’ fixtures.
The test asserts that no tasks exist in the world under test, while another worker may be populating that same world. One worker’s valid write invalidates another worker’s valid read. The failing output no longer distinguishes an application defect from interference in the test infrastructure.
Retries can conceal the same loss of control. The first attempt may change state before it fails and, by accident, set up the fixture that lets the second attempt pass. The resulting “passed on retry” status looks reassuring while hiding a sequencing bug. In a shared-state environment, that result is evidence to examine.
Why namespacing state per worker is the right primitive
The general solution is one isolated state namespace per concurrent unit of execution. This separates the simulator’s lifecycle from the lifecycle of its data. A single long-lived server can hold many short-lived worker partitions, so workers get the startup benefit of a shared process while keeping their records to themselves. The process is shared. The test worlds are not.
To keep that boundary in place, every request carries a worker identity in a header, path prefix, or query parameter. The simulator uses that identity to select a private in-memory partition. Isolating creates while leaving list requests pointed at shared storage still allows one worker to observe another worker’s data.
A header alone cannot provide that guarantee. Unless the simulator uses the identity when choosing its key space, the request remains attached to shared state regardless of what the test runner sends. The case study supplies a concrete implementation of this principle. Its Playwright adapter automatically attaches the x-mock-bucket header with the value w0, and the mock server routes every read and write to that worker’s in-memory World.
Forked simulator instances as an alternative when namespacing is insufficient
Namespacing separates records, but it does not separate everything inside a process. Workers still share the process memory, the clock state, and the global configuration. A separate simulator instance per worker provides a harder boundary, at the cost of startup overhead and port management. The choice depends on what the test must isolate.
Forking is appropriate when the simulator contains global state that cannot be namespaced, including connection pools, schema caches, or intentionally global rate-limit counters. It also fits faults that must affect the simulator process. Another reason to fork is when workers need different configurations, such as different pre-seeded data shapes or API version stubs. Namespacing remains the efficient choice when private partitions cover the required boundary.
In practice, a setup fixture starts each worker’s simulator on a deterministic port derived from a base port plus the worker index. The worker targets that port, and its instance is torn down when the worker completes. In container-based CI, each worker container can have its own simulator sidecar to keep the same ownership. Horizontal scaling then adds containers that run on their own, with no need to coordinate access to shared state. This setup also works for RL training environments that grow to thousands of containers.
Scoped state resets: keeping each worker’s world clean across test cases
Worker isolation does not prevent contamination between sequential tests inside one worker. A test that crashes midway can leave records that make its successor pass or fail incorrectly. Each test therefore needs to start from a deterministic state. A reset at the beginning protects correctness even when the previous test never reached teardown, while worker initialization establishes the starting world for the session.
The reset strategy follows from the assertion. If the assertion requires absent records, a full partition reset is appropriate. In msw-terrarium, given.fresh() restores the worker’s bucket to its baseline, which is empty when no baseline is configured. That is the condition the skipped empty-list test needed. A configured baseline, however, means that “fresh” does not necessarily mean empty.
If the assertion instead depends on a predictable populated world, given.load('seed-name') resets the bucket and applies a named seed fixture. Realistic workflow tests need this restoration path. If the test needs a world with only some of its state changed, given.patch({...}) upserts the chosen rows without wiping the partition. A patch is consequently not a substitute for a clean baseline.
The controls for reset, seed, and patch belong outside the application’s normal API surface. They are test infrastructure. Keeping them separate prevents state-management controls from contaminating ordinary request handling.
Fault injection in an isolated-state environment
Each worker needs its own fault queues, just as it needs its own records. If Worker 3 arms a one-shot fault and Worker 7 consumes it, neither worker can interpret the outcome correctly. Reliable fault testing therefore depends on isolation.
The given.failNext(key, spec, times = 1) pattern demonstrates scoped fault ownership. A fault is armed for a bucket and, by default, fires once against the next matching request in that bucket. This makes the injected condition attributable to the test that requested it.
Error codes are only part of the problem. A response may return 200 with a stale or corrupted record, a latency spike may expose an intermediate state to a polling loop, or a webhook may arrive out of order relative to the state it describes. Canned stateless responses cannot establish these history-dependent faults, while shared state makes their effects non-deterministic.
The tests should probe the guarantees that storage alone does not supply, including atomic updates, transaction isolation, and idempotency. A repeated create can test idempotency, while a controlled transition can test a read that temporarily observes older state. A repeated request can also establish whether a terminal result remains terminal. Isolation makes these workflow assertions reproducible across workers with different timing.
How simulator drift undermines isolation guarantees over time
If its behavior diverges from the real API, passing tests establish only agreement with the simulator. Isolation controls who can change state. It cannot establish that the modeled state and responses are faithful.
Stripe’s 2025-03-31.basil release illustrates the distinction. Within a Checkout Session, the release moved shipping_details from the top level to collected_information.shipping_details. A simulator that kept the old field path could let integration tests pass while the integration failed in production. A registered webhook endpoint with a null api_version introduces another drift risk because a later account upgrade can change payload shapes without a code deployment.
Continuous verification checks how the simulator behaves against the live API before every release and publishes the fidelity delta. That makes divergence visible in the simulator’s release notes. This calls for automated verification rather than updates to response fixtures made by hand now and then. Parallel tests can then rely on both independent state and a model whose differences from the API are exposed.
Applying these patterns to Slack, GitHub, Stripe, and Kalshi integrations
These boundaries must match each API’s behavior. SSE and NDJSON endpoints, where applicable, need simulated chunked delivery with configurable delay per bucket. One worker’s streaming schedule must not interfere with another’s polling behavior, but a simulator must not substitute a generic streaming protocol for the API it represents.
GitHub webhooks are not automatically redelivered on failure. A receipt test must therefore arm delivery and assert receipt within the same worker namespace, rather than replay a missed delivery into another partition. Signature verification needs an equally explicit boundary. Exercising HMAC-SHA256 verification through X-Hub-Signature-256 with a per-worker secret keeps signature tests independent of a shared global signing key.
Stripe’s idempotency behavior needs a stateful key store because keys stay in it for a documented minimum window before they can be removed. Namespacing that store prevents Worker 3’s key from satisfying Worker 7’s duplicate-detection assertion. The test must still detect duplicates within one worker and keep out unrelated requests from other workers. Isolating records but sharing idempotency keys would leave that boundary incomplete.
Kalshi combines RSA-PSS request signing with clock-drift sensitivity. Timestamps too far from server time produce the same opaque authentication error as a wrong key, so frozen or skewed container clocks can resemble credential failures. The simulator should reproduce that distinction per worker, using separate instances when shared clock state prevents isolation. Kalshi’s demo environment has credentials separate from production, so a simulator with isolated worker credentials can be checked against it without touching production authentication.
Structuring a CI pipeline that enforces per-worker isolation end to end
State ownership should be assigned at the pipeline level. Before tests begin, the runner assigns every worker a stable identity, such as an index, UUID, or derived token. Request handling injects that identity automatically as a namespace header. Simulator startup then follows one pipeline-wide choice, either a shared process with enforced namespace routing or a separate sidecar for each worker container.
Every test begins with an explicit fresh() or load(seed) call in a beforeEach fixture. Teardown may clean up afterward, but correctness does not depend on it. Faults are armed in the test body within the worker’s namespace, and the test verifies that they fired before asserting recovery. A separate fidelity stage compares the simulator against the live API and fails the build when the delta exceeds a defined threshold.
The case study shows the throughput consequence of making this change at the mock layer. Its CI configuration moved from one worker to four, and the previously skipped empty-list test ran correctly alongside four concurrent workers. No application code changed.
A “passed on retry” result should remain a mandatory investigation trigger. It may reflect a genuine transient fault, or it may reveal contamination that the isolation setup failed to close. In the latter case, the defect belongs to the test infrastructure. A pipeline audit should check whether the failed attempt set up the state that let the retry pass before accepting that result as green.
