A field journal of third-party API failures, rate limits, breaking changes, and the postmortems teams wish they'd written sooner.

Running a Stateful REST API Simulator as a Long-Lived Test Dependency

Lifecycle, seeding, and reset rules keep a stateful simulator reliable.

· 9 min read
Cover illustration for “Running a Stateful REST API Simulator as a Long-Lived Test Dependency”
Integration Testing · November 10, 2026 · 9 min read · 1,926 words

Integration testing against stateful APIs requires a simulator that remembers resources, enforces valid transitions, and produces webhooks from those transitions without hitting production. When tests rely on that simulator for a long time, they need clear rules for its lifecycle, initialization, reset, and isolation. Those rules let tests run through ordered workflows without their results depending on earlier tests.

Why a simulator that works per-test breaks down when it runs all day

A simulator can behave correctly within one test run and still become unreliable across an entire pipeline or workday. Resources accumulate, initialization becomes stale, and one test inherits changes made by another. A response may match the simulator’s current state even though that state differs from what a test expects at the start.

That tension comes directly from the behavior integration tests need. A POST creates a resource that the next GET reads, a subscription moves between states, and a mutation triggers a webhook. These operations depend on one another in a chain. Remembering that chain makes realistic testing possible, but keeping it past the intended test boundary can make tests fail because of the order they run in.

How do you test APIs where order of operations matters?

A stateless mock maps a request shape to a configured response without remembering earlier calls. A stateful simulator instead maintains a world. After a customer is created, subsequent calls in the session can retrieve that customer. After a subscription is canceled, a GET returns its post-cancellation representation rather than a response configured independently of the cancellation.

The state machine defines which changes are valid. A subscription might move through trialing, active, paused, and canceled, with invalid transitions rejected. Webhooks should follow those transitions wherever the real API would emit them. This lets the application’s webhook-receiving code respond to an actual simulated change. The test exercises the relationship between a mutation and its consequences.

A useful capability check is deliberately simple. Create a resource, retrieve it, mutate it, and retrieve it again to observe the mutation. A tool that cannot preserve that sequence does not provide stateful simulation. Passing that check establishes the foundation for multi-step integration tests, but it also establishes the need to manage the resulting state when tests share a simulator.

Deciding when to start and stop the simulator relative to your test suite

Starting and stopping the simulator for each test provides maximum isolation because no simulator state survives into the next test. That simplicity has a recurring cost. Across a large suite, the time spent on startup latency adds up, and pre-seeded data must be loaded each time. This approach suits tests that need enough isolation to justify that cost on every run.

Starting once per suite, CI job, or test file cuts the work spent on repeated startups. Individual tests then need explicit resets so that one test’s mutations do not influence the next. A process kept alive across an entire pipeline or local development session reduces total startup cost further. It also allows more state to accumulate. The choice depends on how large the suite is, how much startup costs, and how the team works, while reset and isolation keep tests working correctly under that choice.

The dangerous shortcut is to make test order supply initialization. One test creates a resource, another changes it, and a third assumes that both predecessors succeeded. That suite may pass in sequence but fail when tests run alone, change order, or execute concurrently. Ordered API calls belong inside the workflow being tested. The practical diagnostic is whether a test produces the same result when run alone.

State initialization: pre-seeded data vs. building state inside the test

Lifecycle placement determines when the simulator becomes available, but not what tests find inside it. An empty account is appropriate for checking a list endpoint’s behavior when no resources exist. It cannot meaningfully exercise pagination or relationships that require existing data. Initialization must match the behavior under test. Otherwise, a clean environment can still be an inadequate one.

Pre-seeded data gives tests a fixed world to start from, with customers, historical charges, existing subscriptions, and prior webhooks. Tests can query relationships without constructing every prerequisite first. With programmatic setup, each test instead uses API calls to create exactly what it needs before it makes assertions. Dependencies become explicit, but setup adds runtime and can obscure the tested behavior when it grows large.

The practical combination is a stable pre-seeded baseline for read-heavy tests and programmatic setup for write-heavy tests that need a known clean starting state. Neither strategy removes the need for isolation. Pre-seeding also shortens setup code and supports coherent workflows across services. For example, the same client IDs can appear in payment, messaging, and mailing simulators, allowing a cross-service test to refer to the same customer throughout.

Reset strategies and what each one actually guarantees

Restarting a container destroys the process and its in-memory state, providing a clean start at the cost of startup and pre-seed loading. It is appropriate for CI runs that require bit-for-bit reproducibility. An explicit reset endpoint instead restores the baseline in-process through a synchronous call such as POST /reset. It avoids restarting the container, but its guarantee extends only to the state its implementation actually reaches.

Partial reset leaves a particularly difficult ambiguity. A successful call can appear to establish a clean baseline while untouched in-memory state survives. Soft delete has a different limitation because it retains resources marked inactive. GET and list operations must filter by status, or tests may find resources they thought were gone. Logical reset is cheap, but it is not equivalent to an empty namespace.

Namespace or tenant isolation changes the problem from erasing shared state to separating it. Each run or CI job gets its own unique context key, and calls in that context can access only that context’s state. A fresh namespace has no changes from another session to reset. This fits parallel CI directly, where one engineer’s pipeline must not collide with another’s on the same simulator host. Tests that reuse a namespace still need to manage changes within it.

Concurrency and parallelism: where shared stateful infrastructure races

Isolation becomes more demanding when tests execute concurrently. Two tests can POST the same resource identifier at the same time, producing a conflict or overwrite depending on the simulator’s handling of identifiers and idempotency. Another test may read a list while a mutation is underway. Scheduling then affects what the assertion observes. Sequential correctness does not establish that these interactions are safe.

Reset itself can participate in a race. If a reset call and a test request arrive together, the test may encounter either the pre-reset or post-reset state, depending on processing order. A reset strategy that works between sequential tests can therefore disrupt parallel tests sharing the same context. The boundary must keep competing work apart as well as offer a way to delete its results.

Horizontal scaling introduces a related visibility problem. With per-instance in-memory state, a POST handled by one instance is invisible to a GET routed to another unless both use a shared backing store. Per-session namespacing at the data layer keeps one session’s state from affecting another’s, however many instances there are, while concurrency within a session still depends on the simulator’s consistency model. This matters in RL environments, where thousands of agent containers each need their own isolated simulator state. Horizontal scalability and session isolation are minimum requirements for that use case.

Fidelity drift and how a long-lived simulator becomes a liability without continuous verification

Internal consistency does not establish fidelity to the real API. Fields can be renamed, response shapes can change, new parameters can become required, and endpoints can be deprecated. A simulator may continue serving its old behavior without revealing any of those changes. Persistent operation increases the opportunity for that gap to grow, especially when updates depend on manual maintenance.

A per-test fixture regenerated from a specification or recording on each CI run is at least as fresh as that input. A persistent simulator that someone updates by hand may be weeks or months behind. Restarting alone does not improve its accuracy. The verification-based alternative is to probe the real API on a schedule with fixed requests and compare its responses with the simulator’s responses to the same inputs. A score and diff make divergences visible rather than leaving them to emerge during production integration.

A fidelity score expresses the fraction of probe-based comparisons that match the real API’s responses. Its scope is limited to those comparisons. When probes reveal drift, the appropriate model is explicit opt-in migration. The simulator retains pinned behavior while the team reviews the diff and deliberately adopts the change, as it would when updating a dependency version.

Fault injection as a first-class concern in a long-lived simulator setup

Matching successful API behavior tests integration correctness, but resilience requires deliberate failures. A simulator should produce 429 responses with realistic Retry-After headers so that tests can establish whether backoff logic actually backs off. Transient 503 responses and connection drops exercise circuit breakers and retry caps. Failure behavior belongs in the test model. It should not depend on an external service becoming unreliable at a convenient moment.

Latency injection addresses a separate part of that model. Fixed delays, variable lognormal delays, and timeout-triggering delays reveal whether the application’s timeout budgets are appropriate. Dropped or delayed webhooks test polling fallbacks and eventual delivery handling. These scenarios extend the earlier state-machine tests by changing how the application receives the consequences of a transition.

Silent semantic failures need attention even when transport-level behavior looks healthy. A 200 response with a renamed field or missing optional property can expose weak parsing without triggering surface-level health checks. Probe-based comparisons can show these differences in the payloads. In a long-lived CI setup, the lesson is to define fault scenarios declaratively and version-control them with the suite. A fault profile is another kind of session initialization state that is loaded for resilience tests and cleared before the next work starts.

Applying these patterns to a concrete CI pipeline with a persistent simulator

At pipeline start, the simulator can already be running or can start from a Dockerized image that loads pre-seeded data from a versioned fixture file in the repository. The CI job then requests a namespaced session and receives a session key and base URL. Subsequent calls use that session rather than a shared, unqualified state space. This establishes the job’s isolation boundary before functional tests begin.

Within that boundary, read-heavy tests use the seeded baseline, while write-heavy tests construct their prerequisites programmatically. A job-level namespace prevents interference from other jobs, but does not excuse dependencies between tests inside the job.

The fault injection phase then applies the declared profile to the session and runs resilience tests against it. The profile is cleared before the next phase, so deliberate failures do not leak into unrelated assertions. Cleanup frees the session namespace so its state does not carry over to the next CI run. The simulator process can remain available throughout. Its lifetime is separate from the lifetime of any job’s test state.

A developer starts the simulator once, runs individual tests against named namespaces, and resets or rotates those namespaces when a clean slate is needed. Other in-flight work need not be discarded. Once the simulator is running locally, test phases require no external network calls, live credentials, or dependence on third-party uptime. The same pipeline can run in a laptop’s airplane mode or a locked-down enterprise CI environment, while scheduled fidelity verification takes place outside that pipeline.

More in Integration Testing