A distributed system on Azure fails in small ways constantly. A connection resets mid-request, a dependency returns a 503 for two seconds while it scales, a database node fails over and rejects writes for a heartbeat. Most of these faults are transient: they clear on their own within milliseconds or seconds, and a second attempt would succeed. The retry and circuit breaker patterns exist to turn those brief stumbles into invisible recoveries rather than user-facing errors, and to stop a genuinely sick dependency from dragging the rest of the system down with it. Engineers who skip them ship code that works in the demo and pages someone at 3 a.m. when a regional blip ripples into a full outage.
The trouble is that resilience is not a single switch. It is a small family of patterns that each handle a different failure shape, and they only work when composed correctly. Retrying a transient fault is the right move; retrying a dependency that is genuinely down just piles more load onto something already collapsing. A retry without a backoff turns a recoverable hiccup into a self-inflicted denial of service. A retry on a non-idempotent operation, such as charging a card, can bill a customer twice. The patterns are simple to name and easy to get subtly wrong, which is why so many production incidents trace back to a retry loop that should have been a circuit break, or a missing timeout that let one slow call exhaust a thread pool.
This guide lays out the resilience patterns an Azure engineer should reach for, in the order they compose: retry with exponential backoff and jitter, the circuit breaker that stops hammering a failed dependency, the timeout that bounds how long any single call may take, and the bulkhead that isolates one failing component so it cannot sink the whole process. It then ties them together with the requirement that makes retries safe at all, idempotency, and walks a reference design that an order service would actually run in production. By the end you should be able to look at a call to any Azure service or downstream API and know which patterns it needs, how to configure them, and where each one stops helping.
Why transient faults are the default condition, not the exception
The first mental shift is accepting that failure in a distributed system is routine. On a single machine, a function call either returns or the process crashes. Across a network, between services, and over managed platforms that scale, patch, and fail over underneath you, a request has dozens of independent ways to fail without anything being broken. A load balancer drains a node. A platform-as-a-service instance recycles. A throttling limit kicks in for one second. None of these is a bug, and none of them justifies surfacing an error to the caller if a brief, well-behaved second attempt would have worked.
Azure documents this directly. The official guidance on transient fault handling separates errors into two buckets: faults that are self-correcting, where a short wait and another attempt resolves the problem, and faults that are persistent, where retrying changes nothing except the load you add. The entire discipline of resilience engineering comes down to telling those two apart at runtime and responding to each correctly. The retry pattern handles the first bucket. The circuit breaker handles the second. The other patterns exist to keep the cost of being wrong about which bucket you are in from spreading.
What counts as a transient fault on Azure?
A transient fault is a temporary condition that clears without intervention: a brief throttling response such as HTTP 429, a 503 while a service scales, a connection reset, or a database failover that rejects writes for a few seconds. It is distinguished from a persistent fault, such as a 401 or a 404, where retrying is useless because the cause will not change.
The practical test is whether a second attempt, after a sensible pause, has a meaningful chance of succeeding. A 429 throttling response is the textbook case: the service is telling you it is busy right now and to come back shortly, which is an explicit invitation to retry with a delay. Our deep dive on this exact signal, fixing Azure 429 throttling across services, walks the throttling case end to end. A transient database condition behaves the same way; the SQL error that says the database is currently unavailable during a failover, covered in Azure SQL error 40613, resolves on its own once the failover completes, so a retry after a short wait usually connects.
By contrast, an authentication failure, a malformed request, or a missing resource is persistent. The server will reject the second attempt exactly as it rejected the first, and every retry against it is wasted work that consumes a connection, a thread, and a slice of the dependency’s capacity. A resilience layer that retries persistent faults is worse than no resilience layer at all, because it amplifies load precisely when a dependency can least afford it. The skill is not retrying; it is retrying selectively, and knowing when to stop.
Retry, circuit breaker, and the supporting patterns defined plainly
Before composing anything, each pattern needs a precise definition, because the value lies in their differences. The patterns are often discussed as a bundle, but each one answers a distinct question: how long do I wait before trying again, when do I stop trying entirely, how long do I let a single call run, and how do I keep one sick dependency from starving the rest.
Retry with exponential backoff and jitter
The retry pattern re-issues a failed operation a bounded number of times, in the expectation that a transient fault will have cleared. The naive version retries immediately, which is the single most common resilience mistake in production. Immediate retries do two harmful things at once. They give the dependency no time to recover, since the second attempt lands while the condition that caused the first failure is still present. And when many clients fail at the same moment, which is exactly what happens during a shared outage, they all retry in lockstep, producing a synchronized wave of traffic that hits the recovering dependency harder than the original load did. This is the retry storm, and it is how a brief blip becomes a sustained outage.
Exponential backoff fixes the timing. Instead of a fixed delay, the wait between attempts grows multiplicatively: roughly one second, then two, then four, then eight. This gives a struggling dependency progressively more room to recover and caps the total number of attempts within a reasonable window. The base delay, the multiplier, and the maximum number of attempts are the three knobs, and the right values depend on how long the dependency typically takes to heal and how long the caller can afford to wait.
Backoff alone is not enough, because backoff is deterministic. If a thousand clients all fail at the same instant and all apply the same backoff schedule, they retry together at one second, together at two seconds, and together at four. The wave is delayed but not dispersed. Jitter solves this by adding a bounded random component to each delay, so the thousand clients spread their attempts across the window rather than colliding at the same moments. The combination of exponential backoff with jitter is the production-grade form of the retry pattern, and it is what every mature Azure SDK and resilience library implements by default. The key fact to keep exact: backoff with jitter avoids synchronized retries, and synchronized retries are what turn a recovering dependency back into a failing one.
The circuit breaker
A retry, even a well-behaved one with backoff and jitter, assumes the fault is transient. When a dependency is genuinely down, hard down, returning errors for minutes rather than milliseconds, retrying is no longer helping. Every retried call ties up a thread, holds a connection, and adds load to something that cannot serve it. The circuit breaker is the pattern that recognizes this state and stops trying.
The circuit breaker borrows its model from an electrical breaker. It has three states. In the closed state, calls pass through normally and the breaker counts failures. When the failure rate crosses a configured threshold within a sampling window, the breaker trips to the open state. In the open state, calls fail immediately without even attempting the dependency, which protects both the caller, who fails fast instead of blocking, and the dependency, which gets a chance to recover without the added load. After a configured break duration, the breaker moves to the half-open state, where it allows a small number of trial calls through. If those succeed, the dependency has recovered and the breaker closes again. If they fail, it returns to open and waits longer.
The circuit breaker is what prevents the cascading failure, the mode where one slow or dead dependency causes its callers to back up, which causes their callers to back up, until the entire chain is saturated and an outage in one leaf service has become an outage in the whole system. By failing fast when a dependency is genuinely unavailable, the breaker contains the failure at the boundary rather than letting it propagate upstream. The fact to keep exact: a circuit breaker prevents cascading load by stopping calls to a dependency that is down, which is precisely the case where a retry would make things worse.
Timeout
A timeout bounds how long a single call may run before the caller gives up. It sounds trivial, and it is the pattern engineers most often omit, which is unfortunate because a missing timeout is the quiet cause of a large share of production hangs. A dependency does not have to return an error to hurt you. It can simply be slow, holding the connection open while your caller waits indefinitely. Without a timeout, a single slow dependency can consume every thread and connection in the calling service, at which point the caller is down even though nothing technically failed. The dependency is slow; the caller is dead.
The timeout converts an unbounded wait into a bounded, controllable failure. It also feeds the other patterns: a timeout that fires is itself a failure signal that the retry can act on and that the circuit breaker can count toward its threshold. Setting the value is a judgment call. Too short and you abandon calls that would have succeeded; too long and the timeout does not protect you from the slow-dependency hang it exists to prevent. A common approach is to set the per-attempt timeout near the dependency’s high-percentile latency, so normal calls always complete but a genuinely stuck call is cut loose quickly.
Bulkhead
The bulkhead pattern takes its name from a ship’s hull, which is divided into watertight compartments so that a breach in one does not flood the entire vessel. Applied to software, it partitions resources so that one failing or saturated dependency cannot consume all of a shared resource and starve the others. The classic implementation limits the number of concurrent calls, or the size of the thread pool or connection pool, allocated to each dependency.
The scenario it prevents is resource exhaustion by one bad actor. Suppose your service calls three downstream dependencies from a shared thread pool. One of them gets slow. Without isolation, calls to the slow dependency pile up and occupy more and more of the shared pool, until there are no threads left for calls to the two healthy dependencies. The slow dependency has now taken down two services it has nothing to do with. A bulkhead caps how much of the pool any single dependency may hold, so the slow one can saturate its own slice and fail, while the healthy dependencies keep serving from theirs. The bulkhead trades a little throughput, since each dependency gets a fixed allocation rather than the whole pool, for the guarantee that one failure stays contained.
Idempotency: the requirement that makes retries safe at all
Idempotency is the property that performing an operation more than once has the same effect as performing it once. Reading a record is naturally idempotent: fetch it twice and nothing changes. Setting a value to a fixed state is idempotent: assign status to “shipped” twice and the outcome is identical. Appending, incrementing, charging, and sending are not idempotent by default: do them twice and you get two rows, two charges, two emails.
This matters because retry is built on repetition, and repetition is only safe when the operation tolerates being repeated. The dangerous case is the one where the operation succeeded on the server but the response was lost on the way back. From the caller’s point of view the call failed, so the retry fires, and now a payment that already went through is submitted again. The customer is charged twice, and the retry pattern that was supposed to improve reliability has instead produced a financial defect.
The standard remedy is an idempotency key. The caller generates a unique identifier for the logical operation and sends it with every attempt, including retries. The server records which keys it has already processed and, on seeing a repeat, returns the result of the first execution rather than performing the work again. This turns a non-idempotent operation into an idempotent one at the protocol level, which is what makes it safe to retry. The same discipline underpins reliable event processing, where consumers must handle duplicate deliveries; our guide to event-driven architecture on Azure develops the idempotent-consumer pattern in that context. The fact to keep exact: retries require idempotency, and an operation that is not idempotent must be made so before it is ever retried.
The backoff-and-break rule
These patterns compose into a single organizing principle, the rule this guide is built around. Call it the backoff-and-break rule: resilience comes from retrying transient faults with backoff and jitter while a circuit breaker stops retrying a dependency that is genuinely down, so that the two patterns together prevent both flakiness and cascading overload. Retry handles the brief, self-correcting failure. The breaker handles the sustained, persistent failure. Backoff and jitter make the retries themselves polite rather than aggressive. The timeout bounds each attempt so a slow call cannot hang the caller. The bulkhead isolates the blast radius so one sick dependency cannot starve the others. And idempotency is the safety contract that lets any of the retrying happen without corrupting state.
The reason the rule pairs retry and break so tightly is that each pattern covers the other’s blind spot. Retry on its own is dangerous, because it cannot tell a two-second blip from a ten-minute outage; it will keep hammering a dead dependency until its attempt budget runs out, and across many clients that is a retry storm aimed at something already on the floor. The circuit breaker on its own is too blunt, because it has no mechanism to recover from the brief faults that should never have been surfaced as errors at all. Together they form a control loop: retry absorbs the transient, the breaker detects when the transient has become persistent and stops the bleeding, and the half-open probe detects when the dependency has recovered and resumes normal flow. Neither pattern is sufficient alone, and using one without the other is the most common way teams end up with resilience code that makes outages worse instead of better.
The InsightCrunch resilience pattern table
The patterns are easiest to hold in one view, with what each protects against and how it composes with the rest. This is the reference to keep next to any service that calls a dependency over the network.
| Pattern | Failure it protects against | Key configuration | How it composes |
|---|---|---|---|
| Retry with backoff and jitter | Transient faults that clear on their own (429, 503, connection reset, brief failover) | Base delay, multiplier, max attempts, jitter, and a predicate that retries only transient errors | Wraps the call; should sit inside the circuit breaker so the breaker counts exhausted retries as failures |
| Circuit breaker | A persistently failing or down dependency that retries would only overload further | Failure-rate threshold, sampling window, break duration, half-open trial count | Wraps the retry; trips when retries keep failing, fails fast while open, probes in half-open |
| Timeout | A slow dependency that never errors but holds the connection and hangs the caller | Per-attempt timeout near the dependency’s high-percentile latency | Sits innermost around each attempt; a fired timeout is a failure the retry and breaker both act on |
| Bulkhead | One saturated dependency consuming a shared pool and starving healthy dependencies | Max concurrent calls or pool size per dependency | Caps concurrency per dependency so a failure stays in its own compartment |
| Idempotency | Duplicate side effects when a succeeded-but-unacknowledged call is retried | Idempotency key recorded server-side, deduplicated on repeat | The precondition that makes retrying any write safe at all |
Read the table top to bottom and the composition order falls out. The timeout is innermost, bounding each individual attempt. The retry wraps the timeout, re-issuing attempts that fail or time out. The circuit breaker wraps the retry, so that when the retries keep exhausting against a dead dependency, the breaker trips and stops them. The bulkhead caps how much concurrency the whole stack may consume for that one dependency. And idempotency is not a layer in the stack at all but a property the operation must already have for the retry layer to be safe. Get the order wrong, for example by putting the breaker inside the retry, and the retry will simply keep re-opening the breaker on every attempt, defeating its purpose.
Why does the order of the patterns matter?
Order determines behavior because each pattern wraps the next. The timeout must be innermost so it bounds a single attempt, the retry wraps the timeout so it can re-attempt, and the breaker wraps the retry so it trips only after the retries have genuinely failed. Reverse retry and breaker and every attempt re-trips the breaker, which makes it useless.
This is the most consequential detail in any resilience configuration, and it is the one most often gotten wrong. The mental model that keeps it straight is to read the stack from the outside in as the call travels down, and from the inside out as the result travels back. A request enters through the bulkhead, which decides whether there is capacity for it at all. It passes the circuit breaker, which decides whether the dependency is currently considered healthy. If so, it enters the retry, which will manage attempts. Each attempt is wrapped by a timeout that bounds how long that single try may run. The actual call happens at the center. On the way back, a failure or timeout bubbles up to the retry, which decides whether to try again; if the retries exhaust, that exhaustion bubbles up to the breaker as a failure, which counts toward tripping it. The bulkhead, throughout, has been holding the concurrency slot.
The Azure services and libraries that realize the patterns
These patterns are not abstractions you must build from scratch on Azure. They are baked into the official SDKs and into the first-party resilience libraries, and a large part of doing this well is knowing what the platform already gives you so you do not reinvent it, or worse, fight it.
Retry is already built into the Azure SDKs
The modern Azure SDKs ship with retry behavior on by default, configured through a common options surface. The Azure.Core library, which underlies the .NET data-plane SDKs for Storage, Key Vault, Service Bus, and the rest, exposes a RetryOptions object. By default the clients retry three times using an exponential strategy, with an initial delay of about 0.8 seconds and a maximum delay capped at one minute. That default exists so that the common transient faults, the 429s and the 503s and the brief failovers, are absorbed without the calling code writing a single line of retry logic.
The defaults are tunable per client. The Storage SDK, for example, raises the default maximum retries from three to five, on the reasoning that storage calls are often background batch work that can afford to wait longer for a transient condition to clear. You configure the behavior through the client options:
// Tune the built-in retry on an Azure SDK client (Azure.Core options)
var options = new BlobClientOptions
{
Retry =
{
Mode = RetryMode.Exponential, // grow the delay between attempts
MaxRetries = 5, // total retry attempts after the first
Delay = TimeSpan.FromSeconds(1), // initial backoff
MaxDelay = TimeSpan.FromSeconds(30), // cap on any single backoff
NetworkTimeout = TimeSpan.FromSeconds(100) // per-attempt network timeout
}
};
var serviceClient = new BlobServiceClient(accountUri, new DefaultAzureCredential(), options);
The messaging and data SDKs follow the same shape with service-specific names. Service Bus exposes ServiceBusRetryOptions with a ServiceBusRetryMode of Exponential, and surfaces an IsTransient flag on its exceptions so the caller can distinguish a fault that was retried and exhausted from one that was never retryable. The Cosmos DB SDK has its own retry mechanism with explicit handling for rate-limited requests, automatically respecting the retry-after hint that the service returns on a 429 rather than guessing at a backoff. The lesson is consistent across the platform: for calls to Azure services through their official SDKs, the retry pattern is present and sensible out of the box, and your job is to tune it, decide what counts as transient for your workload, and avoid disabling it by accident.
Where the SDK retry stops and a resilience library begins
The SDK retry handles the call to the Azure service itself. It does not handle the circuit breaker, the bulkhead, or retries against your own downstream HTTP APIs, third-party services, or any call that does not go through an Azure SDK. For those, the .NET ecosystem standardized on Polly, which received a complete rewrite in version 8 built in collaboration with Microsoft. Polly v8 replaced its old policy-and-wrap API with a unified ResiliencePipeline that composes strategies in a consistent, allocation-conscious way.
Microsoft wraps Polly v8 in two first-party packages. Microsoft.Extensions.Resilience provides the general resilience pipeline with dependency-injection integration, and Microsoft.Extensions.Http.Resilience adds an HTTP-focused layer that attaches resilience directly to an HttpClient through the client factory. The older Microsoft.Extensions.Http.Polly package is now deprecated in favor of these. Building a pipeline looks like this:
// Compose retry (with backoff + jitter), then circuit breaker, then timeout
// using Polly v8 via Microsoft.Extensions.Http.Resilience
var pipeline = new ResiliencePipelineBuilder()
.AddRetry(new RetryStrategyOptions
{
ShouldHandle = new PredicateBuilder().Handle<HttpRequestException>(),
BackoffType = DelayBackoffType.Exponential, // multiplicative delay
UseJitter = true, // spread synchronized retries
MaxRetryAttempts = 4,
Delay = TimeSpan.FromMilliseconds(500) // base delay
})
.AddCircuitBreaker(new CircuitBreakerStrategyOptions
{
FailureRatio = 0.5, // trip at a 50% failure rate
SamplingDuration = TimeSpan.FromSeconds(30),
MinimumThroughput = 10, // need enough calls to judge
BreakDuration = TimeSpan.FromSeconds(15)
})
.AddTimeout(TimeSpan.FromSeconds(10)) // bound the whole pipeline attempt
.Build();
var result = await pipeline.ExecuteAsync(async token =>
{
var response = await httpClient.GetAsync("/api/inventory", token);
response.EnsureSuccessStatusCode();
return await response.Content.ReadAsStringAsync(token);
});
The strategies execute in the order configured, with the outermost added strategy running first, which is why the composition order from the pattern table is enforced by the sequence of the builder calls. For HTTP specifically, the library offers a standard handler that bundles a sensible retry, circuit breaker, and timeout set in the correct order, so a team that wants the defaults can attach the whole pipeline to a named client with a single call rather than hand-assembling it. The reason to understand the assembly anyway is that the defaults are a starting point, and the values that matter, the failure ratio, the break duration, the per-attempt timeout, are workload-specific and worth setting deliberately.
Do I need a library if the SDK already retries?
Yes, for anything beyond plain calls to an Azure service through its SDK. The SDK retry covers the transient-fault retry for that one call. A circuit breaker, a bulkhead, retries against your own APIs or third-party endpoints, and pipeline composition across all of them require a resilience library such as Polly v8 through the Microsoft resilience extensions.
The division of labor is worth stating plainly so you do not double-wrap or leave a gap. When your code calls Blob Storage, Key Vault, Cosmos, or Service Bus through the official client, lean on the SDK’s built-in retry and tune it; wrapping that call in another retry layer usually just multiplies the attempt count in confusing ways. When your code calls a microservice you own, a partner API, or anything over a raw HttpClient, there is no built-in resilience, and that is where the Polly pipeline earns its place. The circuit breaker and bulkhead almost always come from the library regardless of what is being called, because the SDKs implement retry but generally not breaking or isolation. A clean architecture uses the SDK retry for Azure service calls and a named, well-tuned Polly pipeline for everything else, with the two kept distinct so each call has exactly one resilience owner.
A reference design walked through
Patterns make the most sense applied to a concrete system, so consider an order service that, to place an order, must call three dependencies: a payment provider over HTTP, an inventory microservice the team owns, and a notification queue on Service Bus. Each dependency has a different failure profile, and each therefore needs a different resilience configuration. Treating them identically is a common mistake; the right design tunes each path to the dependency behind it.
The payment provider is a third-party HTTP API with strict rate limits and a per-charge side effect. It needs a retry on transient failures, a circuit breaker because an outage at the provider should not back up the whole order pipeline, a timeout because a slow payment call must not hold a request thread indefinitely, and, above all, idempotency, because retrying a charge that already succeeded would double-bill the customer. The order service generates an idempotency key per order and sends it on every attempt; the provider deduplicates on that key, so a retried charge that already went through returns the original result rather than charging again. The retry predicate is narrow: it retries 429 and 5xx responses and connection failures, and it explicitly does not retry a 402 declined-payment response, because a declined card is a persistent fault that no number of attempts will fix.
The inventory microservice is owned by the team, runs on the same platform, and has no irreversible side effect on a read. Its reservation call does have a side effect, so it too uses an idempotency key, but its reads can be retried freely. It gets a retry, a circuit breaker, and a timeout, plus a bulkhead, because the order service calls inventory on the hot path and must not let a slow inventory deploy consume every thread and take down order placement along with it. The bulkhead caps inventory calls at a fixed slice of the concurrency budget, so an inventory slowdown degrades inventory-dependent features while order intake that does not need a fresh inventory check keeps flowing.
The notification step is different again. It publishes a message to Service Bus and does not block the order on delivery; the order is already placed by the time the notification fires. Here the SDK’s built-in retry on the Service Bus client handles the transient case, and because the notification is asynchronous and the consumer is built to be idempotent, a duplicate publish during a retry is harmless. There is no need for a circuit breaker on the publish path, because a notification failure does not threaten the order; if Service Bus is unavailable, the order still completes and the notification is reconciled later. This is the discipline the design teaches: resilience is applied per dependency according to what that dependency can do to you, not painted uniformly across every call.
How do I decide which patterns a given call needs?
Start from what the dependency can do to you. If transient faults are likely, add retry with backoff. If a sustained outage would back up the caller, add a circuit breaker. If the call could hang, add a timeout. If one slow dependency could starve a shared pool, add a bulkhead. If the call has a side effect, make it idempotent before retrying.
Putting the design into one pipeline per dependency keeps the configuration legible. The payment client carries a retry-breaker-timeout pipeline with a narrow transient predicate and an idempotency key on every charge. The inventory client carries the same three patterns plus a bulkhead. The notification path leans on the SDK retry and an idempotent consumer with no breaker at all. Three dependencies, three deliberately different resilience profiles, and a reviewer can read each one and see exactly which failure modes it guards against and which it consciously accepts. That legibility is itself a resilience property, because the configuration that nobody can reason about is the configuration that drifts into being wrong.
Trade-offs and the failure modes the patterns must handle
Every pattern carries a cost, and the failures the patterns are meant to prevent reappear, in subtler forms, when the patterns are misconfigured. A resilience layer is itself a system that can fail, and the ways it fails are predictable enough to enumerate.
The first and most damaging failure mode is the retry that amplifies load. This is what happens when a retry is configured without backoff, or with backoff but without jitter, or with an attempt budget too large, or with a predicate so broad that it retries persistent faults. In all of these, the retry adds traffic to a dependency that is already struggling. The canonical scenario is a downstream service that gets slow under load; its callers time out and retry immediately; the retries add to the load; the service gets slower; more callers time out and retry; and the system has built a positive feedback loop that takes the dependency from degraded to dead. The fix is the discipline already described: exponential backoff with jitter, a bounded attempt count, a narrow transient-only predicate, and a circuit breaker sitting outside the retry to cut it off when the retries stop helping. The throttling guide on Azure 429 responses shows the same dynamic from the dependency’s side, where a flood of un-backed-off retries is exactly what trips the rate limiter harder.
The second failure mode is the non-idempotent retry that duplicates a side effect. A retried charge double-bills. A retried “create order” makes two orders. A retried “send email” sends two emails. The cause is always the same: a write operation that succeeded on the server but whose acknowledgment was lost, retried by a layer that assumed the failure meant the write did not happen. The only durable fix is to make the operation idempotent before any retry touches it, using an idempotency key the server deduplicates on, so that a repeated attempt returns the first result instead of repeating the work. Teams that add retries to a write path without first auditing its idempotency are shipping a latent double-execution bug that will surface the first time a response is lost in flight.
The third failure mode is the absent circuit breaker, which leaves retries to run unbounded against a dead dependency. Without a breaker, a hard outage in one dependency turns into a chain of backed-up callers: requests to the dead dependency hold threads while they exhaust their retries, those threads are unavailable for other work, the caller saturates, its callers saturate, and the failure cascades upstream until a problem in one leaf has consumed the whole tree. This is the exact cascade the circuit breaker exists to stop, and omitting it is the difference between a contained failure and a system-wide outage. The breaker fails the calls to the dead dependency fast, frees the threads, and lets the rest of the system keep serving the requests that do not depend on the failed component.
What happens if I retry without a circuit breaker?
Retrying without a circuit breaker means nothing stops the retries when a dependency is genuinely down. Each retried call holds a thread and a connection while it exhausts its attempts, those resources back up, callers saturate, and the failure cascades upstream. A breaker fails fast once the dependency is judged dead, freeing resources and containing the outage.
The fourth failure mode is subtler: the timeout that is mis-set. A timeout set too long does not protect against the slow-dependency hang, because by the time it fires the caller’s threads are already exhausted; a timeout set too short abandons calls that would have succeeded, manufacturing failures out of normal latency variance and feeding them to the retry and breaker, which then trip on what was never a real fault. The fix is to set the per-attempt timeout from measured latency, near the dependency’s high percentile, so that healthy calls always complete and only genuinely stuck calls are cut. A timeout chosen by guess is a coin flip between not protecting you and actively harming you.
The fifth failure mode is the bulkhead omitted, which lets one saturated dependency consume a shared pool. The scenario, again, is concrete: three dependencies sharing one connection or thread pool, one of them slows, its calls pile up and occupy the pool, and the two healthy dependencies are starved of the resources they need, so a slowdown in one unrelated component takes down two others. The bulkhead caps each dependency’s share of the pool, so the slow one saturates only its own allocation and the healthy paths keep their slices. The cost is that no single dependency can burst into the whole pool when it is the only one busy, which is a real throughput trade in exchange for the isolation guarantee.
The sixth failure mode is the synchronized retry storm, which is what backoff without jitter produces under a shared outage. When many clients fail at the same instant and all apply the same deterministic backoff, they retry together at each step, producing waves that hit the recovering dependency in unison and can re-trigger the very failure they are recovering from. Jitter is the fix, spreading the retries across the window so the load arrives smoothly rather than in spikes. This is why every mature implementation enables jitter by default, and why disabling it to make retry timing predictable is a false economy that trades a small operational convenience for a large reliability risk.
When the patterns fit and when they are overkill
Resilience patterns are not free, and applying all of them everywhere is its own kind of mistake. Each pattern adds configuration, latency under failure, and a new way for the system to behave surprisingly. The judgment is matching the pattern to the risk rather than reaching for the full stack reflexively.
Retry with backoff and jitter is close to universally appropriate for any call that crosses a process or network boundary and can experience a transient fault, which is almost all of them. The cost is low and the upside is high, so the default answer for a remote call is to retry transient faults politely. The exception is an operation that is both non-idempotent and impossible to make idempotent, where a retry risks a duplicate side effect with no safe deduplication; for those, the right move is to fix the idempotency first or accept that the call fails fast without retry rather than risk the double execution.
The circuit breaker fits any dependency whose sustained failure would harm the caller, which means any synchronous call on a hot path. It is overkill on a fire-and-forget path where a dependency failure does not threaten the caller, such as the asynchronous notification in the reference design, where there is nothing to protect because the order already completed. Adding a breaker there buys nothing and adds a failure mode. The test is whether failing fast on this dependency actually helps the caller; if a slow or failed call to it cannot back up the caller, the breaker has no work to do.
The timeout is appropriate on essentially every synchronous remote call, because the slow-dependency hang it prevents is both common and severe, and the cost of a timeout is only that you must choose a sensible value. There are very few synchronous network calls that should be allowed to run unbounded, and the ones that genuinely need long ceilings, such as a deliberate long-poll, should set a long timeout rather than no timeout. The bulkhead, by contrast, earns its place only when a shared resource pool serves multiple dependencies and the failure of one could starve the others. A service that calls a single dependency does not need a bulkhead, because there is nothing to isolate it from; the isolation matters only when there is contention to manage.
When is a circuit breaker overkill?
A circuit breaker is overkill when a dependency’s failure cannot harm the caller, such as on an asynchronous or fire-and-forget path where the result is not awaited. If failing fast on the dependency does not free anything or protect anything upstream, the breaker adds configuration and a new failure mode without preventing a real one.
The honest summary is that retry and timeout belong on nearly every synchronous remote call, the circuit breaker belongs on every synchronous call whose failure can back up the caller, idempotency belongs on every retried write, and the bulkhead belongs wherever multiple dependencies contend for a shared pool. That is not the full stack on every call, and resisting the urge to apply every pattern to every path is part of using them well. The Azure Well-Architected Framework treats this kind of deliberate, risk-matched design as the core of its reliability pillar; our walkthrough of the Well-Architected Framework places resilience patterns inside that broader reliability discipline.
How to evolve a resilience strategy over time
A resilience configuration is not set once and forgotten. The values that matter, the attempt counts, the failure thresholds, the timeouts, are estimates at first and should be tuned against real behavior. The evolution path moves from sensible defaults, to measured tuning, to active validation, to the more advanced strategies that only make sense once the basics are solid.
The first stage is observability. A resilience layer that you cannot see into is a resilience layer you cannot tune or trust. Every retry, every breaker state transition, every timeout, and every bulkhead rejection should emit a metric and a log, so that you can answer questions like how often the payment circuit breaker trips, what the retry rate to inventory looks like under load, and whether timeouts are firing on healthy calls. Polly v8 surfaces telemetry for exactly this, and the first improvement most teams make after shipping the patterns is wiring that telemetry into their metrics so the configuration stops being a guess. Without it, you discover your timeout is too short only when it manufactures an incident.
The second stage is tuning from data. Once you can see the behavior, the timeout should be set from the dependency’s measured latency distribution rather than a round number, the breaker’s failure threshold and break duration should reflect how that dependency actually recovers, and the retry’s attempt count and backoff should match how long transient faults typically persist. A timeout at the dependency’s 99th-percentile latency, a breaker that samples enough calls to judge a real failure rate rather than tripping on a couple of unlucky requests, and a backoff whose total window fits inside the caller’s own deadline: these are the adjustments that move the configuration from plausible to correct.
The third stage is validation through fault injection. The only way to know your resilience patterns work is to make the failures happen on purpose and watch the system respond. Inject latency into a dependency and confirm the timeout fires and the bulkhead contains it. Take a dependency down and confirm the breaker trips, the caller fails fast, and the rest of the system keeps serving. Azure Chaos Studio exists to run exactly these experiments in a controlled way, turning resilience from an assumption into a tested property. A pattern that has never been exercised under real failure is a pattern you are merely hoping works.
The fourth stage brings in the more advanced strategies, which the Polly v8 pipeline supports alongside the core four. Hedging issues a second attempt in parallel after a short delay rather than waiting for the first to fail, trading extra load for lower tail latency on dependencies where the occasional slow call dominates the latency budget. Rate limiting caps the outbound call rate to a dependency to stay within its quota and avoid triggering its throttling in the first place, which is the proactive complement to retrying the 429 after the fact. Fallback supplies a degraded but acceptable response when the primary path fails, such as a cached value or a default, so that a dependency outage produces reduced function rather than a hard error. These are not starting points; they are refinements to add once the retry, breaker, timeout, and bulkhead are in place, measured, and validated.
How the circuit breaker behaves under the hood
The breaker rewards a closer look, because its three states and the parameters that govern the transitions between them are where most misconfigurations hide. The closed state is the normal operating mode, where calls pass to the downstream component and the breaker observes their outcomes. What it observes is not a single failure but a rate, measured over a sampling window, which matters because a single unlucky error should not trip a breaker any more than a single coin landing heads proves a biased coin. The breaker accumulates outcomes across the window and trips only when the proportion of failures crosses the configured ratio and enough calls have occurred to make that ratio meaningful.
That minimum-throughput requirement is the parameter teams most often overlook, and overlooking it produces a flaky breaker. If the breaker is allowed to trip on a 50 percent failure ratio without a floor on call volume, then two requests where one fails will trip it, even though one failure out of two proves nothing about the health of the downstream component. A sensible floor, ten or twenty calls within the window before the ratio is even evaluated, prevents the breaker from reacting to noise. The window length, the failure ratio, and the minimum throughput together define how sensitive the breaker is, and tuning them is a balance: too sensitive and the breaker opens on normal variance, cutting off a healthy component; too insensitive and it stays closed through an outage it should have caught.
The open state is the protective mode. Once tripped, the breaker rejects calls immediately for the configured break duration, without touching the downstream component at all. This is the fail-fast behavior that frees the caller’s threads and gives the struggling component room to recover unburdened. The break duration is a guess about how long the downstream component needs to recover, and it interacts with the half-open probe that follows. A break that is too short sends probes before the component has healed, so the breaker flaps between open and half-open and never settles; a break that is too long keeps rejecting calls after the component has already recovered, extending the outage from the caller’s side past the point where the downstream is healthy again.
What is the half-open state for?
The half-open state is the breaker’s recovery test. After the break duration elapses, the breaker permits a small number of trial calls to reach the downstream component. If they succeed, the component has recovered and the breaker closes back to normal operation. If they fail, the component is still unhealthy, so the breaker returns to open and waits another break duration before testing again.
The half-open behavior is what makes the breaker a control loop rather than a one-way switch. Without it, a tripped breaker would either stay open forever, requiring manual intervention to reset, or reopen the floodgates blindly after the timer, sending full traffic at a component that may not have recovered. The half-open probe threads the needle by sending just enough traffic to test health without risking a fresh overload. A common refinement limits the half-open state to a single concurrent trial, so that if the downstream is still sick, only one probe is wasted rather than a burst. Understanding this state is what separates an engineer who configures a breaker from one who can diagnose why a breaker is flapping, which is almost always a break duration mismatched to the downstream component’s real recovery time.
There is a second, often missed subtlety: the breaker should be keyed correctly. A single breaker guarding a downstream component with several endpoints will trip on the failure of one endpoint and reject calls to the healthy ones, which is too coarse. Conversely, a breaker per individual request is too fine to ever accumulate enough throughput to judge a failure rate. The right granularity is usually one breaker per logical downstream component, or per endpoint group that shares a failure mode, so that the breaker’s view of health matches the unit that actually fails or recovers together. Getting the key wrong produces a breaker that either over-trips, taking down healthy paths, or never trips, never seeing enough traffic to act.
Retry budgets, deadlines, and honoring retry-after
A retry configuration that looks correct in isolation can still be wrong once you account for the request’s overall deadline. Every retry consumes time, and the sum of the attempts plus their backoff delays must fit inside the time the caller is willing to wait. A four-attempt retry with exponential backoff starting at one second can easily span fifteen seconds before it gives up, and if the user-facing request has a five-second budget, those retries will blow the budget long before they exhaust, leaving the user staring at a spinner while the system dutifully re-attempts a call that can no longer help them. The retry window must be sized against the caller’s deadline, not chosen in a vacuum.
This is the deadline-propagation problem, and it compounds across a call chain. If service A calls service B with a five-second budget, and B retries its own call to C for eight seconds, then B is spending time A has already given up waiting for. The discipline is to propagate the deadline down the chain, so each layer knows how much of the original budget remains and sizes its retries to fit. A call that arrives with two seconds left on the clock should not embark on an eight-second retry sequence; it should make one attempt, or none, and fail fast so the caller can move on. Treating the deadline as a first-class input to the retry decision, rather than retrying a fixed number of times regardless of remaining time, is what keeps a retry layer from becoming a latency amplifier.
The notion of a retry budget extends the idea from a single request to the aggregate. A common refinement caps retries as a percentage of total traffic, for example allowing retries to add at most ten percent to the request volume against a downstream component. This budget acts as a circuit-breaker-like guard on the retry layer itself: under normal conditions the budget is never approached, but during a partial outage when many calls are failing and retrying, the budget caps how much additional load the retries may add, preventing the retry layer from contributing to a retry storm even before the breaker trips. It is a smoother, always-on complement to the breaker’s hard cutoff.
Should I honor the retry-after header?
Yes. When a downstream component returns a retry-after value, typically on a 429 throttling response, it is telling you exactly how long to wait before re-attempting, and that hint is better than any backoff you would compute yourself. Honoring it avoids re-attempting before the component is ready and avoids tripping its rate limiter again. The Cosmos DB SDK does this automatically.
Honoring the server’s own timing guidance is the single most reliable way to retry a throttled call, because the component knows its own state better than the caller’s backoff formula does. A computed exponential backoff is a guess at how long to wait; a retry-after header is the component telling you the answer. When both are present, the header wins. This is why a well-built retry layer reads the retry-after value from a 429 or 503 response and uses it as the delay for the next attempt, falling back to computed backoff only when no header is present. Several Azure SDKs, the Cosmos client among them, already do this internally, which is one more reason to lean on the built-in retry for Azure service calls rather than wrapping them in a generic retry that ignores the hint and re-attempts too soon, prolonging the throttling it was meant to ride out.
Resilience across specific Azure services
The patterns are general, but each Azure service presents transient faults in its own way and offers its own built-in handling, so applying resilience well means knowing the per-service specifics rather than treating every dependency identically.
Cosmos DB has the most developed built-in resilience of the data services. Its SDK retries throttled requests automatically, reading the retry-after hint the service returns on a 429 and waiting exactly that long before the next attempt, up to a configurable cap. It also handles transient connectivity and, in multi-region accounts, can fail reads over to another region. The practical guidance is to raise the SDK’s retry-on-throttle settings for workloads that experience bursts rather than to wrap Cosmos calls in an external retry, because an external layer that does not understand the retry-after hint will re-attempt too aggressively and make the throttling worse. Our guide to Azure 429 throttling across services covers the throttling signal and the budget-aware response in depth.
Azure SQL surfaces transient conditions as specific error numbers, the database-unavailable-during-failover case being the canonical one, and the data-access libraries provide first-class transient-fault handling for them. In Entity Framework Core, enabling retry on failure turns on a built-in execution strategy that re-attempts the recognized transient SQL error numbers with backoff, so application code does not catch and re-issue them by hand. The detail that catches teams out is that this execution strategy interacts with explicit transactions and with user-initiated retries, so you enable the built-in strategy and let it own the retrying rather than layering your own loop on top. Our walkthrough of Azure SQL error 40613 explains why the database-unavailable condition is transient and clears once the failover completes, which is exactly why the built-in strategy resolves it without surfacing an error.
Service Bus and the other messaging services build retry into their clients and expose an IsTransient flag on their exceptions, so the caller can tell a fault that was retried and exhausted from one that was never eligible for retry. Because messaging is frequently asynchronous, the resilience emphasis shifts from the publish side, where a retried publish during a lost acknowledgment is usually harmless, to the consume side, where the consumer must be idempotent to tolerate the at-least-once delivery that the broker guarantees. A message can be delivered more than once, on redelivery after a processing failure or a lock expiry, so the consumer that updates state must deduplicate on a message identifier exactly as a retried write deduplicates on an idempotency key.
How is resilience different for asynchronous Azure services?
For asynchronous services such as Service Bus or Event Hubs, the resilience burden moves to the consumer. The platform guarantees at-least-once delivery, so messages can arrive more than once, which means the consumer must be idempotent to tolerate duplicates. A circuit breaker on the publish path is usually unnecessary because a publish failure does not block the originating request, which has already completed.
Storage, Key Vault, and the broader set of Azure.Core-based SDKs share the common retry surface described earlier, with sensible exponential defaults that you tune per client. At the edge and the platform layer, the resilience story is different again: Azure Front Door and Application Gateway offer health probes and automatic failover across backends, App Service and Azure Kubernetes Service provide readiness and liveness probes that take an unhealthy instance out of rotation, and these platform-level mechanisms work alongside, not instead of, the application-level patterns. A health probe removing a sick instance is a coarse-grained breaker at the infrastructure layer; the circuit breaker in your code is the fine-grained one at the call layer. A complete resilience design uses both, the platform handling instance and backend health while the application handles the per-call retry, break, timeout, and isolation. Placing these patterns inside the broader reliability discipline of the Azure Well-Architected Framework is what turns a collection of techniques into a coherent strategy.
An operational playbook for the recurring scenarios
The patterns become muscle memory when you can match a production symptom to the pattern that addresses it. The following scenarios recur often enough across incident reviews that recognizing them on sight is worth the practice.
The symptom of an overloaded downstream that keeps getting worse, where latency climbs and error rates rise in a way that does not level off, frequently traces to immediate or un-jittered re-attempts amplifying the load. What you observe is a feedback loop: as the component slows, callers give up and re-issue, the re-issues add load, the component slows further. The change is to slow the re-attempts with exponential backoff, disperse them with jitter, narrow the predicate so only genuine transient faults trigger a re-attempt, and place a circuit breaker outside the re-attempt logic so that when the re-attempts stop helping, they stop entirely. The fix is rarely fewer total attempts alone; it is politer attempts plus a hard cutoff.
The symptom of duplicate side effects, a customer reporting two charges or two confirmation messages, traces to a non-idempotent operation being re-attempted after a lost acknowledgment. What you observe is that the duplicate appears only intermittently, under network stress, which is the tell that an operation succeeded server-side but the response was lost, triggering a re-attempt. The change is to introduce an idempotency key the server deduplicates on, so a repeated attempt returns the original result. No amount of tuning the re-attempt timing fixes this; the operation itself must be made safe to repeat, and until it is, the re-attempt should be disabled on that path.
The symptom of one failing component taking down unrelated features traces either to a missing circuit breaker, which lets re-attempts against a dead component back up the caller and cascade, or to a missing bulkhead, which lets the dead component’s calls consume a shared pool and starve the healthy paths. What you observe is that an outage in component X coincides with degradation in features that have nothing to do with X, which is the signature of resource contention or cascade. The change is to add a breaker so calls to X fail fast and free their resources, and a bulkhead so X’s calls cannot occupy more than its allotted slice of the shared pool. The two together turn a system-wide outage into a localized degradation confined to the features that genuinely need X.
How do I tell a retry problem from a circuit breaker problem?
If the symptom is a struggling component that re-attempts keep hammering and never recovers, the re-attempt configuration is the problem: add backoff, jitter, and a narrow predicate. If the symptom is re-attempts running unbounded against a component that is fully down, while resources back up and the failure spreads, the missing circuit breaker is the problem. Re-attempt issues are about politeness; breaker issues are about knowing when to stop.
The symptom of a caller that hangs without any error, where threads are consumed and the service stops responding although nothing has thrown, traces to a missing or over-long timeout letting a slow component hold connections open. What you observe is thread-pool exhaustion or connection-pool exhaustion with no error logs from the component itself, because it never failed, it just never answered. The change is to set a per-attempt timeout from the component’s measured high-percentile latency, so a stuck call is cut loose before it can occupy a thread indefinitely. This scenario is insidious precisely because there is no error to point at; the component is healthy by its own metrics while the caller dies waiting, which is why the timeout, the least glamorous pattern, prevents a disproportionate share of real outages.
Where resilience code belongs in an architecture
A resilience strategy fails in practice not only through bad parameters but through bad placement. Retry loops scattered through business logic, breakers instantiated inline, timeouts hardcoded in three different methods: this is how a codebase ends up with resilience behavior nobody can audit, where one call to a downstream component retries five times and an adjacent call to the same component retries none, for no reason anyone remembers. The patterns are a cross-cutting concern, and treating them as one is what keeps them consistent and correct as the system grows.
The clean placement on .NET is the resilience pipeline registered once in dependency injection and attached to a named client, so the business logic that calls the downstream component contains no resilience code at all. It calls the client and gets a result or an exception; the retry, the breaker, the timeout, and the bulkhead all live in the pipeline configured at startup. This separation has three payoffs. The resilience behavior is defined in one place, so it is consistent across every call through that client and changeable in one edit. The business logic stays readable, expressing what it does rather than how it tolerates failure. And the pipeline becomes independently testable, because you can exercise the configured pipeline against a fake that fails on command and confirm the patterns behave, without dragging the business logic into the test.
Naming matters here too. A pipeline named for the downstream component it guards, the payment pipeline, the inventory pipeline, makes the configuration legible and prevents the accidental reuse of one component’s tuning for another whose failure profile is different. The payment provider with strict rate limits and irreversible charges needs a different pipeline from the internal inventory service, and giving each a named pipeline keeps that distinction explicit rather than letting a single shared configuration paper over it. When a reviewer opens the startup configuration, the named pipelines read as a map of every downstream dependency and exactly how each is protected, which is documentation that cannot drift out of sync with the code because it is the code.
Should resilience logic live in the application or the platform?
Both, at different granularities. The platform layer, through health probes, load balancer failover, and service mesh policies, handles coarse-grained health: removing a sick instance or backend from rotation. The application layer, through the retry, breaker, timeout, and bulkhead in code, handles fine-grained per-call behavior the platform cannot see. They are complementary, not alternatives, and a complete design uses each for what it does best.
A service mesh deserves a mention because it changes where the patterns can live. A mesh sidecar can apply retries, timeouts, and breaking at the network layer, outside the application process entirely, configured by policy rather than code. This is appealing because it makes resilience uniform across services regardless of language, and it removes the burden from application teams. The limit is that a sidecar sees bytes and status codes, not intent: it cannot know that a particular write is non-idempotent and must not be retried, and it cannot generate an idempotency key. So even with a mesh handling the transport-level patterns, the application remains responsible for idempotency and for any resilience decision that depends on understanding what the call means. The mesh handles the general case; the application handles the cases that require knowing the semantics of the operation, and idempotency is always one of those.
Beyond the core four: hedging, rate limiting, and graceful degradation
Once the four core patterns are in place, measured, and validated, three further strategies earn consideration, each addressing a problem the core four do not. They are refinements rather than foundations, and reaching for them before the basics are solid is a common way to add complexity without adding reliability.
Hedging attacks tail latency rather than failure. On a downstream component where most calls are fast but a small fraction are slow for no reason you can fix, hedging issues a second, parallel attempt after a short delay rather than waiting for the first slow call to finish or time out. Whichever attempt returns first wins, and the other is abandoned. This trades extra load, since some fraction of calls now happen twice, for a tighter latency distribution, cutting the slow tail that a single attempt would suffer. Hedging pays off on read-heavy paths to components with high latency variance, and dangerous on writes, because a hedged write is by definition a duplicated operation and demands the same idempotency guarantee as a retried one. The decision to hedge is a decision to accept more load for less tail latency, and it only pays where the tail actually hurts.
Rate limiting is the proactive complement to retrying a throttle. Rather than calling a downstream component as fast as the code can and re-attempting the 429s that result, a rate limiter caps the outbound call rate to stay within the component’s known quota, so the throttle is avoided rather than recovered from. This is gentler on the downstream and more predictable for the caller, and it is the right pattern when you know a component’s limit and your own traffic could exceed it. It composes with retry rather than replacing it: the rate limiter keeps you under the quota in the normal case, and the retry handles the occasional throttle that slips through when traffic spikes faster than the limiter smooths it. Pairing the two is how a high-volume caller stays a good citizen against a metered dependency.
What is graceful degradation and how does fallback enable it?
Graceful degradation means a system continues to provide reduced but useful function when a dependency fails, rather than failing entirely. The fallback strategy enables it by supplying an alternative response when the primary path fails: a cached value, a default, or a simplified result. A product page that cannot reach the recommendations service shows the page without recommendations rather than returning an error.
Fallback is the pattern that turns a dependency failure into a degraded experience instead of an outage, and it is the strategy that most directly serves the user during a partial failure. When the circuit breaker is open and calls to a component are failing fast, the fallback decides what the caller does with that fast failure: return a cached value if one is available, return a sensible default, omit the optional feature the failed component powered, or queue the work for later if it does not need to happen synchronously. The art of fallback is identifying which parts of a response are essential and which are enhancements, so that the loss of an enhancement degrades the experience rather than breaking it. A checkout that cannot reach the loyalty-points service should still complete the purchase and reconcile the points afterward, because the purchase is essential and the points are an enhancement. Designing these degradation paths is what separates a system that survives a dependency outage with a slightly poorer experience from one that hands the user an error page, and it is the human-facing payoff of all the machinery beneath it.
A short example shows the idempotency contract that underpins safe retrying and hedging alike, expressed at the call site:
// An idempotency key makes a retried or hedged write safe.
// The server records processed keys and returns the first result on a repeat.
var idempotencyKey = order.Id.ToString(); // stable per logical operation
using var request = new HttpRequestMessage(HttpMethod.Post, "/api/payments");
request.Headers.Add("Idempotency-Key", idempotencyKey); // sent on every attempt
request.Content = JsonContent.Create(new { order.Id, order.Amount });
// The resilience pipeline may retry; the key guarantees one charge, not many.
var response = await pipeline.ExecuteAsync(
async token => await httpClient.SendAsync(request, token));
The point of the example is not the syntax but the discipline: the key is stable for the logical operation, it is sent on every attempt including retries, and the server is what enforces the single execution by deduplicating on it. Without the server side honoring the key, the header is decoration; with it, the same retry that would have double-charged now collapses to a single charge no matter how many attempts the pipeline makes. This is the contract that lets every other pattern in this guide operate without fear of corrupting state, and it is worth confirming on every write path before that path is ever retried.
Common misconceptions that produce fragile systems
Several beliefs about resilience are widespread enough to be worth naming and correcting, because each one leads teams to ship code that looks resilient and behaves worse than no resilience at all.
The first misconception is that more retries mean more reliability. Past a small bounded count, additional attempts buy almost nothing on a transient fault, which has usually cleared within the first one or two re-attempts, while costing a great deal on a persistent fault, where every extra attempt is wasted load on a component that will not recover. A high attempt count also blows the caller’s deadline and delays the circuit breaker from tripping, because the breaker cannot register a clean failure until the long re-attempt sequence finally exhausts. Reliability comes from the right small number of attempts paired with a breaker, not from grinding through a large one.
The second misconception is that the circuit breaker replaces the retry. They solve different problems and the breaker without re-attempts surfaces every brief transient fault as an error, defeating the purpose of resilience, which is to make brief stumbles invisible. The breaker is the upper guard that stops re-attempts when they have become futile; it is not a substitute for the re-attempts that absorb the faults that should never reach the user. A system with a breaker and no retry fails on every blip; a system with retry and no breaker melts down under a sustained outage. The backoff-and-break rule pairs them precisely because each is incomplete alone.
The third misconception is that resilience is a library you add rather than a design you reason through. Installing a resilience package and attaching its defaults to every client produces motion without thought: breakers on fire-and-forget paths that have nothing to protect, retries on non-idempotent writes that now double-execute, uniform timeouts that fit no component’s actual latency. The library supplies the mechanism, but the decisions, which patterns each call needs, in what order, with what values, and which writes are safe to repeat, are engineering judgments the library cannot make for you. A thoughtful configuration with three patterns beats a thoughtless one with all six.
The fourth misconception is that idempotency is a database concern rather than a protocol concern. Teams often assume that because their database uses transactions, their operations are safe to repeat, which confuses atomicity with idempotency. A transaction guarantees that one execution of an operation completes wholly or not at all; it says nothing about what happens when the same logical operation is executed twice because a response was lost. Two transactions, each atomic, can still produce two charges. Idempotency is a property of the operation as seen across attempts, enforced by deduplicating on a key, and it sits above the database, not inside it. Until a write path carries and honors an idempotency key, it is not safe to re-attempt, transactions notwithstanding.
The fifth misconception is that resilience can be added at the end, after the features work. Resilience touches the shape of the operations themselves, most of all through idempotency, which is far cheaper to design in than to retrofit. Bolting retries onto a write path that was never built to be repeated is how the double-charge bug ships; the idempotency key has to be threaded through the operation from the start. Treating resilience as a property to design alongside the feature, rather than a wrapper to add afterward, is what makes it cheap and correct rather than expensive and leaky.
Is adding a resilience library enough to make a system resilient?
No. A library supplies the mechanisms, but resilience is the set of decisions about which patterns each call needs, how they are ordered, what values fit each component, and which writes are safe to repeat. Attaching default policies everywhere produces breakers that protect nothing, retries that double-execute writes, and timeouts matched to no real latency. The judgment is yours; the library only carries it out.
The thread running through every one of these misconceptions is that resilience rewards thought over reflex. The patterns are few and their definitions are simple, but their value comes entirely from applying the right ones, in the right order, with values drawn from how a specific component actually behaves, and from auditing the operations they wrap for the idempotency that makes repetition safe. A team that internalizes the backoff-and-break rule and the composition order, and that resists the urge to spray every pattern across every call, will build systems that absorb the constant low-grade failure of a distributed platform without the operator ever noticing, which is the entire point.
The verdict
Resilience on Azure is not a feature you enable but a set of patterns you compose deliberately, matched to what each dependency can do to the caller. The backoff-and-break rule captures the heart of it: retry transient faults with exponential backoff and jitter so brief stumbles heal invisibly, and let a circuit breaker stop the retries the moment a dependency goes from briefly flaky to genuinely down, so that the two patterns together prevent both the surfaced error and the cascading overload. Around that core, the timeout bounds every single attempt so a slow dependency cannot hang the caller, the bulkhead isolates one failing component so it cannot starve the rest, and idempotency is the contract that makes any of the retrying safe in the first place.
The platform does a great deal of this for you. The Azure SDKs retry transient faults out of the box, the Cosmos client honors retry-after on throttling, and the first-party resilience extensions wrap Polly v8 so that a production-grade pipeline of retry, breaker, and timeout is a few lines of configuration rather than a hand-rolled framework. The engineering judgment that remains is yours: deciding which patterns each call needs, ordering them so the timeout is innermost and the breaker wraps the retry, setting the values from measured behavior rather than guesswork, auditing every retried write for idempotency, and validating the whole thing under injected failure before you trust it in front of users. Do that, and a regional blip becomes a recovery nobody notices instead of the incident that defines the quarter. The patterns themselves are few and their definitions fit on a single reference card, but the difference between a system that shrugs off the constant low-grade failure of a distributed platform and one that amplifies it into an outage lies entirely in the care of the application: the right patterns chosen per call, ordered correctly, tuned from measured behavior, and built on operations that are safe to repeat. To put the patterns into practice against real Azure dependencies, VaultBook provides hands-on labs where you can implement and test a full retry, circuit breaker, timeout, and bulkhead pipeline and watch each one behave under controlled failure.
Frequently asked questions
What are retry and circuit breaker patterns on Azure?
The retry pattern re-issues a failed operation a bounded number of times in the expectation that a transient fault, such as a 429 throttle, a 503, or a brief failover, will clear on its own. The circuit breaker pattern monitors the failure rate of a dependency and, when failures cross a threshold, stops sending calls to it for a period so that a dependency which is genuinely down is not hammered by retries. The two are complementary: retry absorbs brief, self-correcting faults, while the breaker detects when a fault has become persistent and stops the retries from piling load onto a dependency that cannot serve it. On Azure they are realized through the SDK retry built into the official clients and through resilience libraries such as Polly v8, wrapped by the Microsoft resilience extensions, for circuit breaking and for calls outside the Azure SDKs.
How do I implement backoff with jitter on Azure?
Exponential backoff grows the delay between retry attempts multiplicatively, for example one second, then two, then four, giving a struggling dependency progressively more time to recover. Jitter adds a bounded random component to each delay so that many clients failing at the same moment do not retry in synchronized waves. In the Azure SDKs, the exponential retry mode is the default and the delay and cap are set through the client’s RetryOptions. With Polly v8 through the Microsoft resilience extensions, you set BackoffType to Exponential and UseJitter to true on the retry strategy options, along with the base delay and maximum attempts. The combination matters because backoff alone is deterministic, so without jitter a thousand clients still collide at the same retry moments; jitter spreads them across the window and is what actually prevents the synchronized retry storm.
How do I handle transient faults in an Azure application?
First distinguish transient faults, which clear on their own, from persistent faults, which do not. Transient faults include 429 throttling, 503 service-busy responses, connection resets, and brief database failovers; persistent faults include 401 authentication errors, 404 not-found, and 400 bad-request. Retry only the transient ones, using a narrow predicate that matches the specific transient status codes and exceptions rather than retrying everything. For calls to Azure services through their SDKs, the built-in retry already handles the common transient cases, so tune it rather than replacing it. For your own APIs and third-party endpoints, configure a retry with exponential backoff and jitter through a resilience library. The discipline is selective retry: retrying a persistent fault wastes work and amplifies load on a dependency that cannot benefit from another attempt.
How do timeout and bulkhead patterns help resilience?
A timeout bounds how long a single call may run before the caller gives up, which prevents the quiet failure where a slow dependency holds connections open and exhausts the caller’s threads without ever returning an error. Set the per-attempt timeout near the dependency’s high-percentile latency so normal calls complete but a stuck call is cut loose quickly. A bulkhead partitions a shared resource pool so that one saturated dependency cannot consume all of it and starve the others; it caps the concurrent calls or pool slice any single dependency may hold. Together they contain two distinct failure shapes: the timeout stops a slow dependency from hanging the caller, and the bulkhead stops a saturated dependency from spreading its failure to unrelated dependencies sharing the same pool. Both are containment patterns that limit the blast radius of a single component’s problem.
How do the Azure SDK retries and resilience libraries work together?
The Azure SDKs build retry into their official clients, so calls to Storage, Key Vault, Service Bus, Cosmos, and similar services retry transient faults by default, configured through a common options surface such as Azure.Core’s RetryOptions. That built-in retry covers the transient-fault case for the Azure service call itself, and you tune it rather than wrapping it in another retry layer. A resilience library, specifically Polly v8 through Microsoft.Extensions.Resilience and Microsoft.Extensions.Http.Resilience, handles everything the SDK retry does not: the circuit breaker, the bulkhead, retries against your own microservices or third-party APIs, and the composition of multiple patterns into one ordered pipeline. The clean division is SDK retry for Azure service calls and a named Polly pipeline for everything else, with each call having exactly one resilience owner so the two never double-wrap or leave a gap.
How do I make retries idempotent and safe?
Idempotency means performing an operation more than once has the same effect as performing it once, which is the precondition for retrying any write safely. Reads are naturally idempotent; appends, increments, charges, and sends are not. The danger is a write that succeeds on the server but whose acknowledgment is lost, so the caller retries and duplicates the side effect, such as charging a card twice. The standard fix is an idempotency key: the caller generates a unique identifier for the logical operation and sends it on every attempt including retries, and the server records processed keys and returns the original result on a repeat rather than performing the work again. Audit every retried write for idempotency before adding retry to its path. An operation that cannot be made idempotent should fail fast without retry rather than risk a duplicate execution.
What is a retry storm and how do I prevent one?
A retry storm is a synchronized wave of retry traffic that hits a recovering dependency hard enough to re-trigger the failure it was recovering from. It happens when many clients fail at the same moment, during a shared outage, and all retry on the same schedule, either immediately or with deterministic backoff that has no jitter. The aligned retries arrive in spikes that can push a barely-recovering dependency back into failure, producing a self-sustaining loop. Prevention has three parts: exponential backoff so attempts spread out over time, jitter so clients do not align on the same retry moments, and a circuit breaker that trips when retries keep failing so the storm is cut off rather than running until every client exhausts its attempt budget. Jitter is the specific ingredient that disperses the wave; backoff alone only delays it.
Should every call have a circuit breaker?
No. A circuit breaker fits any synchronous call whose sustained failure could back up the caller, which is most hot-path calls, but it adds nothing on a fire-and-forget or asynchronous path where the dependency’s failure cannot harm the caller. In the reference order service, the synchronous payment and inventory calls get breakers because an outage there would back up order placement, while the asynchronous notification publish does not, because the order has already completed and a notification failure threatens nothing. The test is whether failing fast on this dependency actually frees or protects something upstream. If a slow or failed call to the dependency cannot saturate the caller, a breaker just adds configuration and a new failure mode without preventing a real one. Match the pattern to the risk rather than applying it uniformly.
How do I set the right timeout value?
Set the per-attempt timeout from the dependency’s measured latency distribution, near a high percentile such as the 99th, rather than picking a round number by intuition. A timeout set too long fails to protect against the slow-dependency hang, because by the time it fires the caller’s threads may already be exhausted. A timeout set too short abandons calls that would have succeeded, manufacturing failures out of ordinary latency variance and feeding them to the retry and circuit breaker, which then trip on faults that were never real. The correct value lets healthy calls complete reliably while cutting genuinely stuck calls quickly, which means you need observability on the dependency’s latency before you can choose it well. A timeout chosen by guess is a coin flip between not protecting you and actively harming you, so measure first.
What is the difference between retry and circuit breaker?
Retry and circuit breaker handle opposite ends of the same failure spectrum. Retry assumes the fault is transient and will clear, so it re-issues the call a bounded number of times with growing delays. The circuit breaker assumes that once failures persist past a threshold the dependency is genuinely down, so it stops sending calls entirely for a period and fails fast instead. Retry is optimistic and reattempts; the breaker is protective and stops. They compose: the retry sits inside the breaker, so the breaker counts exhausted retries as failures and trips when the retries keep failing, indicating the fault is no longer transient. Used together they cover the full range, with retry absorbing brief stumbles and the breaker containing sustained outages. Using either alone leaves a gap, which is why the backoff-and-break rule pairs them.
Does the bulkhead pattern reduce throughput?
A bulkhead trades a measure of peak throughput for isolation. By capping the concurrent calls or pool slice any single dependency may hold, it prevents one dependency from bursting into the entire shared pool even when it is the only busy one, which is a real throughput cost. In exchange, it guarantees that a saturated dependency cannot consume all of a shared resource and starve the others, so one component’s failure stays in its own compartment instead of spreading. The trade is usually worth it on a service that calls multiple dependencies from a shared pool, because the isolation prevents a far more expensive failure than the throughput it gives up. On a service that calls only one dependency, the bulkhead has nothing to isolate from and the throughput cost buys nothing, so it should be omitted there.
How do I validate that my resilience patterns actually work?
Validate them by injecting failures on purpose and observing the response, because a pattern that has never been exercised under real failure is only a hope. Inject latency into a dependency and confirm the timeout fires and the bulkhead contains the slowdown without spreading it. Take a dependency offline and confirm the circuit breaker trips, the caller fails fast rather than hanging, and the rest of the system keeps serving requests that do not depend on the failed component. Azure Chaos Studio runs these experiments in a controlled way against real infrastructure. Alongside fault injection, wire telemetry from the resilience pipeline into your metrics so you can see retry rates, breaker state transitions, and timeout firings in normal operation. Together, injected failure tests confirm the patterns work and continuous telemetry confirms they stay correctly tuned as the system changes.
How many times should I retry a failed operation?
There is no universal number, but a small bounded count, typically three to five attempts, is the usual range for interactive paths, with the exact value driven by how long the transient fault persists and how long the caller can afford to wait. The total retry window, base delay multiplied out across all attempts with backoff, must fit inside the caller’s own deadline, or the retries will blow the request budget before they help. Background or batch work can afford more attempts and longer windows than an interactive request a user is waiting on, which is why the Storage SDK raises its default from three to five. The attempt count interacts with the circuit breaker: you want enough attempts to ride out a genuine transient fault but few enough that a dead dependency trips the breaker quickly rather than each caller grinding through a long retry budget against something that will never recover.
Where does the SDK retry stop and where do I need Polly?
The Azure SDK retry covers transient-fault retry for the specific call made through that official client, and nothing more. It does not provide a circuit breaker, a bulkhead, retries against APIs that are not Azure services, or composition of multiple patterns into an ordered pipeline. For those you need a resilience library, which on .NET means Polly v8, usually consumed through Microsoft.Extensions.Resilience for general pipelines and Microsoft.Extensions.Http.Resilience for HttpClient calls. So the boundary is: use and tune the built-in SDK retry for calls to Azure services through their clients, and use a Polly pipeline for circuit breaking, bulkheading, and any call to your own services or third parties. Keep the two distinct so each call has exactly one resilience owner, which avoids the confusion of a retry inside a retry and the gap of a call with no breaker where it needed one.
Can resilience patterns make an outage worse?
Yes, and misconfigured resilience is a common root cause of incidents rather than a guard against them. A retry without backoff or jitter amplifies load on a struggling dependency and can drive it from degraded to dead. A retry on a non-idempotent operation duplicates side effects such as double charges. A missing circuit breaker lets retries run unbounded against a dead dependency until the failure cascades upstream. A timeout set too short manufactures failures from normal latency, and one set too long fails to protect against hangs. The patterns prevent outages only when configured with the discipline they require: narrow transient predicates, exponential backoff with jitter, bounded attempt counts, a breaker outside the retry, timeouts set from measured latency, idempotency on every retried write, and validation under injected failure. Applied carelessly, the same patterns become the mechanism by which a small fault becomes a large one.