Queues and backoff: how retries and deferrals actually work

Exponential backoff: the standard, and why

When a message is deferred, your server does not retry immediately and endlessly; it uses exponential backoff, widening the gap between attempts as failures accumulate. The first retry might fire after a few minutes, the next after fifteen, then thirty, then an hour, then longer, up to the point where the server gives up. The logic balances two competing needs: retrying soon enough to clear a brief problem quickly, and waiting long enough not to hammer a receiver that needs time. The table shows an illustrative schedule; the exact intervals are configurable, but the shape — short at first, lengthening steadily — is near-universal because it is what well-behaved senders do and what receivers expect to see.

Greylisting: the deliberate delay

Greylisting deserves its own explanation because it is the deferral most senders meet most often, and it alarms people who do not understand it. A greylisting receiver temporarily rejects mail from a combination it has not seen before — the triplet of sending IP, sender address and recipient address — and accepts it on a later attempt. The trick rests on a behavioural difference: a real mail server, on receiving a temporary failure, dutifully retries, while a great deal of spam tooling fires once and moves on. So the delay filters out the lazy spammers at the cost of a few minutes for everyone else. In practice a greylisted message delivers on the second or third attempt, usually within fifteen to thirty minutes, and you need do nothing — your queue handles it. The lasting fix, where you want to skip the delay, is authentication: properly aligned SPF, DKIM and DMARC let many receivers recognise you and bypass greylisting altogether.

When a deferral becomes a bounce

A deferral is not forever. Your server retries for a configured maximum — commonly a day to several days — and if the receiver is still refusing when that window closes, the server stops trying and generates a permanent bounce of its own back to the original sender. This is how a purely temporary failure at the far end can surface, eventually, as a hard bounce in your records even though the receiver never returned a 5xx. The practical lesson is to treat persistent deferrals as a problem rather than as something that will sort itself out. A message that has been deferred for hours, climbing the backoff ladder with no acceptance, is on its way to becoming a bounce, and repeated soft failures from the same provider mean something systemic — reputation, authentication, throttling — needs attention before the retry window quietly runs out.

Patterns receivers watch

It is worth remembering that retrying is itself a behaviour the receiver observes and judges. A sender that uses standard, widening backoff looks like a legitimate, well-configured mail system and earns goodwill over time. A sender that retries aggressively — the same message slammed at the receiver every few seconds — looks broken or hostile and can earn a harder block for the trouble. The metaphor that fits is a knock at a door: a polite knock, repeated at sensible intervals, is fine; the same knock hammered continuously becomes a reason to stop answering. Worse, retrying before a stated greylist window has elapsed often just resets the receiver's clock, so impatience literally extends the delay it was meant to shorten. Respectful spacing is not only courteous; it is faster.

Throttling versus greylisting: different problems

Two of the most common deferral causes call for opposite responses, and confusing them wastes effort. Greylisting is about identity — the receiver does not recognise you yet — and the fix is to retry normally and authenticate well, after which it stops. Throttling is about rate — you are sending too fast for what your reputation with that receiver currently allows — and the fix is to slow down, pacing your sends to that destination rather than simply retrying the same flood. Tuning your retry backoff does nothing for a throttling problem, because the issue is not when you retry but how fast you send overall; conversely, pacing your send rate does nothing for greylisting, which only wants a second attempt. Diagnosing which one you face — usually from the code and from whether the deferrals scale with your volume — decides whether you reach for the backoff knob or the rate knob.

How providers signal throttling: explicit and silent

Throttling shows up in two ways, and the silent kind catches people out. The explicit kind is honest: the receiver returns a 4xx, commonly a 421, that says in so many words that you have hit a rate limit and should slow down. Your queue sees the code, backs off, and retries, and the signal is easy to read. The silent kind is subtler: instead of a clear rate-limit reply, the receiver simply accepts your connections more slowly, drags out responses, or quietly routes more of your mail to spam as you push volume. There is no obvious error to alert on, only a creeping slowdown and a softening of placement. Detecting silent throttling takes watching the right metrics — connection times stretching, acceptance rate dipping, queue depth rising without a matching spike in any one code — rather than waiting for an explicit complaint. The receivers that throttle silently are telling you to slow down just as firmly as the ones that say so; you simply have to listen harder.

Connections, concurrency and how they feed deferrals

Volume is not the only thing receivers limit; they also care how many connections you open at once and how many messages you push down each. Open too many simultaneous connections to one provider, or hold them too long, and you trip concurrency limits that surface as deferrals even when your total volume is modest. This is why two senders pushing the same daily count can see very different deferral rates: one spreads its mail across sensible, steady connections, while the other hammers the receiver with a burst of parallel sessions that looks aggressive. Tuning concurrency — how many connections per destination, how many messages per connection, how quickly you ramp parallelism — is part of pacing, and getting it wrong produces deferrals that no amount of backoff tuning will cure, because the problem is the shape of your traffic, not the timing of your retries. Receivers reward traffic that looks calm and steady over traffic that arrives in spikes.

Queue depth as a health metric

The single most useful number for spotting deferral trouble early is the depth of your retry queue. In a healthy system, messages enter retry and clear at roughly the rate they arrive, so the queue stays shallow and stable. When deferrals start outpacing successful retries, the queue grows, and a steadily rising queue — especially toward one provider — is an early, measurable warning that something systemic is wrong: you are being throttled, your reputation has slipped, your authentication is failing, or a receiver is down. Watching queue depth per provider alongside the 4xx rate turns a vague sense that mail is slow into a specific, timely signal. It is one of the metrics we put front and centre when we run managed deliverability, precisely because a growing queue caught early is a problem solved before it becomes a wave of bounces.

Tuning retries on your own MTA

Operators who run their own infrastructure have direct control over backoff, and a few principles keep that control from backfiring. Set the first retry at or above a receiver's stated greylist window, because retrying earlier just earns another temporary failure. Keep your queue-scan interval no longer than your minimum backoff, so the schedule you configured is the schedule that actually runs. Use destination pacing — controlling how fast you send to a specific receiver — for throttling symptoms, and reserve backoff changes for genuine deferrals, since they fix different problems. Avoid forcing repeated queue flushes, which reset the receiver's wait window and extend delays, and avoid very short backoff timers, which create noise and more temporary failures. These are exactly the controls a clean PowerMTA setup configures deliberately, and the kind of tuning that turns a deferral storm back into smooth delivery when we are called in to troubleshoot a delivery drop.

Deferrals during warm-up

A new IP gets deferred heavily, and that is normal rather than alarming. Receivers are cautious with unfamiliar identities, so they defer mail from a fresh sender while they decide whether to trust it, and the deferrals ease as reputation accumulates. During warm-up, those 4xx responses are feedback: a provider starting to defer you more as you raise volume is telling you to slow the ramp, and respecting that signal — holding volume, letting acceptance recover — is how a warm-up stays on track. Pushing harder against rising deferrals is how it derails. The interplay between deferrals and the ramp is close enough that we treat reading them as part of warming an IP at all, which we cover in our guide to how IP warm-up works.

Deferrals and time-sensitive mail

Some mail cannot wait, and deferrals are a particular problem for it. A password reset, a one-time code, an order confirmation — these lose much of their value if they arrive twenty minutes late because a greylisting receiver deferred the first attempt. The instinct is to retry such mail faster, but as we have seen that backfires. The better answers are structural. Send transactional mail from infrastructure with strong, established reputation and clean authentication, so it is deferred less often in the first place; segregate it from bulk and promotional traffic so a throttling problem on the marketing stream never delays a password reset; and, for the most time-critical messages, consider that email is an inherently best-effort medium and that truly instant delivery may need a second channel. Separating the streams so that urgent mail rides its own well-warmed path is one of the clearest reasons to segment sending by type rather than pushing everything through one queue.

The cost of mishandling deferrals

Deferrals handled badly are expensive in ways that compound. The immediate cost is delay: mail that should arrive in seconds sits in a queue for hours, and time-sensitive messages lose their purpose. The next cost is bounces, as deferrals that never clear age out of the retry window and become permanent failures, so a throttling problem you ignored turns into undelivered mail and a worse sender history. Then comes reputation: aggressive retrying, growing backlogs and rising soft-failure rates are themselves signals to receivers that you are not a well-run sender, which earns you still more deferrals — the loop feeding itself. And underneath all of it is the opportunity cost of flying blind: a team that does not watch its queue and deferral metrics learns about each of these only after it has already hurt delivery. None of this is dramatic in any single moment, which is exactly why it is dangerous; deferral problems erode quietly until someone notices the mail is slow and the bounces are up.

What not to do

A few habits reliably turn a manageable deferral problem into a worse one. Do not retry faster to "force" delivery; aggressive retry loops earn suspicion and can harden a soft block into a real one. Do not force repeated queue flushes hoping to push mail through, because each flush can reset the receiver's wait window and lengthen the delay. Do not set tiny backoff timers, which generate a storm of premature attempts and more temporary failures. Do not ignore a growing queue on the assumption that retries will sort it out, because a deferral that never clears becomes a bounce. And do not tune your retry timers when the real problem is rate — or pace your sends when the real problem is greylisting — because applying the wrong fix leaves the actual cause untouched. Patience and the right knob beat panic and the wrong one every time.

What a deferral log entry tells you

Every deferral leaves a line in your logs, and that line is a small diagnostic packet if you read it. A useful entry records the recipient domain, the sending IP used, the SMTP reply the receiver gave — both the three-digit code and the enhanced code and text — the time of the attempt, and when the next retry is scheduled. Reading them together answers the questions that matter: which provider is deferring, with what code, how old the message is, and whether the backoff is behaving. A greylisting deferral shows a 451 4.7.1 clearing on a later attempt; a throttling one shows a 421 that scales with your volume; a reputation slide shows deferrals widening across messages to one provider with no acceptance. Comparing the queue age, the reply code and the next-retry behaviour before changing anything is the discipline that stops you from tuning the wrong knob. A sender who reads the log entry knows whether to wait, slow down or investigate; one who reads only the aggregate "delivery is slow" is guessing.

Reading deferrals per provider

Deferrals only make sense provider by provider, because each receiver has its own limits, its own greylisting policy, and its own reputation view of you. A deferral rate that is fine for one provider may signal trouble at another, and a backoff that suits one receiver's greylist window may be wrong for another's. The senders who handle deferrals well watch them segmented by destination: this provider is greylisting first sends and clearing them on retry, that one is throttling because volume rose too fast, a third is deferring on a reputation wobble. That per-provider view, combined with the queue-depth and 4xx-rate signals, is what lets you respond specifically — pace the one that is throttling, keep retrying the one that is greylisting, investigate the one whose deferrals look like a reputation slide — instead of treating a varied set of problems as one undifferentiated "mail is slow."

In short

A deferral is a conversation, not a verdict. The receiver returns a 4xx to say "later," your server queues the message and retries on a widening backoff, and the exchange resolves into delivery if you answer well — or into a bounce if you do not, once the retry window runs out. Handle it by reading the code to learn the cause, backing off politely rather than hammering, distinguishing throttling from greylisting and reaching for the right knob, watching queue depth per provider as your early-warning signal, and authenticating cleanly so you are deferred less in the first place. Do that and deferrals stay what they are meant to be: a brief, self-resolving pause on the way to the inbox. Mishandle them and they curdle into delays, bounces and a reputation that earns you still more deferrals — the loop worth staying out of. The senders who never seem to fight their queue are not lucky; they authenticate cleanly, pace their traffic, back off politely, and read their logs — and so the deferrals they do get clear quietly, the way the system intends.