DNS in Production

When a DNS change is slow to take effect, the usual answer is “wait for the TTL”. That answer is incomplete. The TTL is one clock among several: the resolver may raise or cut it, a miss is cached on its own timer, a failed lookup can be cached too, and the client may hold the address for longer than any of them. In our lab, a Java client given a record with a 2 s TTL kept using the old address for about 30 s ^[1]M.

So this paper measures the clocks, one at a time, and then adds them up. The lab is small: two resolvers, Unbound 1.22.0 and BIND 9.20.30, four client runtimes (Python, Go, curl and Java) and a custom scriptable authoritative server. That limits what we can say. Other resolvers appear only through their documentation, and we label those figures as sourced, not measured. Every measured number is indicative, from one laptop with emulated images, and a lab figure is a lower bound for a real cutover.

We wrote it for the engineer who owns a zone or a failover design for a small or mid-sized service, and for whoever is paged when DNS is slow to switch. You need dig and a shell. It is not aimed at registry or large public-resolver operators.

The question: Why does a change you made at the authoritative server not reach every user at the same moment?

AWS’s summary of its DynamoDB disruption in October 2025 gives the cause as “a latent race condition in the DynamoDB DNS management system” that left the regional endpoint with no IP addresses. DNS failures began at 11:48 PM PDT on 19 October. DNS information was restored at 2:25 AM, and DynamoDB recovered at 2:40 AM, after cached DNS records had expired ^[2]S. The fix was in place at 2:25. The service was not back until 2:40.

Those 15 minutes are the subject of this paperD. Everything between your zone file and a user’s connection holds a copy of your answer, and each copy has its own expiry. Editing the zone changes one copy. Users keep seeing the others until they run out.

1.1 Where the 15 minutes went§

The arithmetic is short. From 11:48 PM to 2:40 AM is 2 hours 52 minutes of DNS-caused errors. From the restored DNS information at 2:25 AM to recovery at 2:40 AM is 15 minutes, which is about 9 percent of the totalD. The postmortem does not break that tail into resolver caching and client behaviour ^[2]S. We will not allocate it either: nobody outside AWS can, and any split would be a guess.

InfoQ’s report of the postmortem quotes the engineer Yan Cui saying that DNS was the first symptom, not the root cause ^[3]S. That is his view, and AWS’s own account fits it, since the failure was an automation fault in the system that writes DNS recordsO. Chapter 8 returns to that. This chapter is about the other half of the story: once the records were right again, how long did it take for everyone to see them?

Fixing the zone fixes one copy of the answer.

1.2 What the standards promise§

The standards are consistent on the basic idea and loose on the rest. RFC 1034 says the TTL is a time limit on caching, set by the zone administrator, and that the administrator may lower it before an anticipated change ^[4]S. RFC 1035 says a TTL of zero means the record is used only for the transaction in progress ^[5]S. RFC 2181 limits a TTL to the range 0 through 2147483647, says a value with the most significant bit set is treated as zero, and describes the TTL as a maximum time to live, not a mandatory one ^[6]S.

Three consequences follow, and the rest of the paper depends on them.

• A TTL is a ceiling. The limit is on how long a copy may be kept, and the RFC wording does not promise that every cache keeps it that long.
• The records of one set expire together. RFC 2181 requires equal TTLs inside an RRSet and says to use the lowest if they differ ^[6]S. Mixing TTLs in one set buys nothing.
• The zone owner is not the only author. RFC 2181 also ranks authoritative answers above data from the additional section, so what a resolver keeps depends on where it learned the record ^[6]S.

The RFCs are not a perfect guide either. RFC 1035 treats the TTL as signed in one section and unsigned in another ^[5]S. A TTL is one number in the zone file and a family of behaviours on the wire. Chapter 2 separates them.

1.3 The path of copies§

The figure draws the path a lookup takes. Every box that can answer from memory is a copy, and a miss moves one hop onward. The drawing is a model. It attaches no number to any hop, because chapters 2 to 6 measure the clocks one at a time and some of them we did not measure.

Not every hop exists in every deployment, and some hops are several. A stub resolver on the host may be a caching service: the systemd-resolved manual describes a caching and validating stub whose caches are flushed on a network configuration change and on SIGHUP ^[7]S. The manual page says little about how it treats TTLs, so we make no claim about its clamping. The OS layer is a cache with its own lifecycle, and you may have to flush it.

1.4 One change, four runtimes§

We wanted to see whether the layer closest to the application matters in practice. The setup was small. A record had a TTL of 2 s at a caching resolver in the lab. About 3.5 s after the first lookup we changed the answer at the authoritative server. A client then sent one request every 0.5 s, each on a new connection, and we logged when the new address first appeared ^[8]M. The resolver side was Unbound 1.22.0 in this test, and the client runs were Python 3.12.16, Go 1.23.12 (both resolver modes), curl 7.88.1 and OpenJDK 21.0.13.

The first row is the baseline. With no client-side cache, the switch followed the 2 s resolver TTL plus the 0.5 s request interval, and the 31 requests of each run produced no errors ^[8]M. The other rows show a second clock laid on top. The Java default held the old address for about 30 s from the first lookup, although the record said 2 s. The long-lived curl process held it for about 60 s from the first lookup, which is the size of its DNS cache, and curl 7.88.1 has no command line option to change that. With the Java cache property at −1 it did not switch in 45 s. We do not claim that it never would, only that the run ended first ^[8]M.

Two limits matter. The Java default is a property of this build, not of Java in general: the Java SE 21 documentation leaves the caching period to the implementation, so read our 26.66 s as a measurement of OpenJDK 21.0.13 in this labM. And every request used a new connection. A client that keeps one connection open does not look the name up again while it stays open, which is a different failure that we describe in chapter 6 from vendor documentation only, without measuring it.

1.5 Cutover is the remaining TTL§

The resolver in the middle, in our lab, did what the RFC wording suggests. Both Unbound 1.22.0 and BIND 9.20.30 served a record for about its TTL: 5 s and 20 s records were cached for about 5.3 s and 20.9 s (Unbound) and 4.9 s and 20.1 s (BIND) ^[8]M. After we flipped the authoritative answer, a resolver kept serving the old address for about the time left on the cache entry. Across TTLs of 3, 6 and 12 s, flipped at 10, 50 and 90 percent of the TTL, measured minus expected stayed within 0.95 s for Unbound and 0.72 s for BIND, and a TTL of 0 was visible at once ^[8]M.

So a user’s wait depends on when the user’s resolver last filled its cache, which you do not know. A record with a 12 s TTL flipped 1.2 s after the fill took about 11.6 s to show in Unbound and 10.1 s in BIND. The same flip at 10.8 s took 1.8 s and 1.1 s ^[8]M. The unit of cutover is a range, from nothing to one TTL, per resolver in the path.

The figures come from a lab: one laptop clock, loopback-class latency, a synthetic authoritative server, linux/amd64 images emulated on an arm64 host, so every timing carries a few hundred milliseconds of overhead. Two runs agreed to within 0.86 s at most. Treat the lab numbers as a lower bound for a real cutoverM. What we did not measure here is the detection side: the lab has no health checker, so how long a failure takes to be noticed is not in these numbers. Chapter 5 builds that term from provider documentation and marks it as derived.

1.6 Prove it on your own zone§

Before the next change, look at one copy yourself. Ask your resolver for a record you own without letting it recurse:

dig +norecurse www.example.com A @your-resolver

If the resolver has the answer, the TTL column is the time it has left, not the value in your zone. Run it twice a few seconds apart and watch it count down. Write down the remaining TTL, the value in the zone, and the resolver you askedO. If you can name the hops in the figure for one real client, you can say where your own change will wait.

The question: Which timers decide when a new DNS answer becomes visible, and which of them do you control?

In the model of chapter 1: every cached copy has its own expiry. This chapter lists the expiries a resolver keeps.

The TTL is the clock people know. It is also the only one the zone owner sets directly. A resolver runs at least four more, and each has its own default, its own cap and its own failure mode. When someone says “DNS is slow to switch”, the answer is usually one of the other four.

This chapter names the five clocks, prints what the standards and four resolver manuals say about each, and then shows what two resolvers did with them in our lab. The two were Unbound 1.22.0 and BIND 9.20.30, both in Docker 29.8.2 on one arm64 laptop, with linux/amd64 images emulated ^[8]M. We did not run PowerDNS, CoreDNS or dnsmasq. Their numbers below come from their documentation, read on 2026-11-11, and are labelled that way.

2.1 The clocks§

1. Positive TTL, then the clamp. How long an existing answer is reused. The authoritative server sets the TTL, and the resolver may raise it to a floor or lower it to a cap before it counts.
2. Negative TTL. How long “this name, or this type, does not exist” is reused.
3. Failure cache. How long “we could not get an answer at all” is reused, so a broken server is not asked again at once.
4. Stale window. How long an expired copy may still be handed out when a refresh fails.
5. Parent or child NS TTL. How long the resolver believes the delegation, which decides whose servers it asks next.

Only the first and, through the SOA record, the second are set in your zone. The failure cache and the stale window are resolver policy. The fifth depends on a copy of your data that lives in your parent zone and which you may not control at all.

2.2 What the standards and the manuals say§

RFC 2308 section 5 sets the negative TTL from the lower of the SOA MINIMUM field and the TTL of the SOA record itself ^[9]S. It suggests capping negative caching at 1 to 3 hours and says values over one day have been found to be problematicS. Section 8 makes negative caching mandatory for caching resolversS.

RFC 9520 requires resolvers to cache resolution failures for at least 1 second and at most 5 minutes, and says they should back off ^[10]S. RFC 8767 requires a stale TTL above zero and recommends 30 seconds. It suggests a 1.8 second client response timer and a maximum stale time of 1 to 3 days, and says not to retry a failed refresh more often than every 30 seconds ^[11]S. Those are RFC values. Nothing in this chapter measured them.

For the TTL itself, Moura et al. (IMC 2019) and RFC 9199 both recommend at least 1 hour for general zones. They say about 5 to 15 minutes only suits DNS load balancing or DDoS redirection ^{[12, 13]}S. In the same paper a ccTLD saw its median latency fall from 183 ms to 28.7 ms after lengthening its TTL, and a controlled test compared 60 s with 86,400 s from about 15,000 Atlas probes ^[12]S. Moura et al. do not recommend a TTL of 0S. They also report that many resolvers applied a minimum caching time of tens of seconds in their 2019 measurements, so a short TTL is a request, not a guaranteeS.

The implementations then fill in what the standards leave open. Table 2.1 sets four manuals side by side.

Table 2.1. Documented defaults, read from each project’s manual on 2026-11-11. Sourced, not measured: the versions in the manuals were the current online ones, not necessarily the versions anyone runs. n/l: not listed in the part we read. n/r: section not read. The PowerDNS settings are `minimum-ttl-override` and `packetcache-servfail-ttl`. dnsmasq keeps its floor under 1 h unless recompiled; `--neg-ttl` applies only to replies without an SOA. Sources: Unbound ^[14], PowerDNS ^[15], CoreDNS ^[16], dnsmasq ^[17]S.

Read across a row and the problem shows. The positive cap is 86400 s in two resolvers and 3600 s in a third. The negative cap is 3600 s in two and 1800 s in oneS. The floor is 0, 1 or 5 seconds. Per the manuals we read, a record set to 10 seconds is cached for 10 seconds by Unbound, PowerDNS and CoreDNS, and a record set to 2 seconds is held for 5 seconds by CoreDNSS. CoreDNS documents a stale TTL of 0 for stale answers, where RFC 8767 requires a value above zero, so check the version you run before relying on eitherS. dnsmasq documents --neg-ttl as a fallback for replies with no SOA, so an operator who sets it and expects it to bound negative caching in general has read it wrongS.

We have no table row for BIND, on purpose. The BIND reference lists the option names (max-cache-ttl, max-ncache-ttl, min-cache-ttl, stale-answer-enable, max-stale-ttl and others) ^[18]S. We read only the 9.20.0 text for max-ncache-ttl and min-ncache-ttl (chapter 3). We did not read the stale options or the TTL cap and floor, so Table 2.1 has no BIND row. Where BIND appears below, the number was measured.

2.3 Clock 1: the TTL and its clamp§

Both resolvers honored the TTL we set, and both honored a floor and a cap when we configured one (cache-min-ttl 15 and cache-max-ttl 8 in Unbound, min-cache-ttl 15 and max-cache-ttl 8 in BIND)M. With the floor, the first reply carried TTL 15. In each case the authoritative server was asked twice, once at the first lookup and once after expiry. Readings are “about”: poll step 0.2 s, whole-second TTL counting.

This is a narrow result. We tested two TTL values and two clamp settings, not a grid of TTLs from 0 to 2147483647, and we did not test any default floor of another resolver. Each resolver landed within 0.9 s of the expected value, and the two differed from each other by up to 1.7 s (the 8 s cap: 8.9 s in Unbound, 7.2 s in BIND)M.

The floor is a policy of whoever runs the resolver, and a client behind a resolver with a 5 second floor does not get your 2 second TTL. Moura et al. saw the same thing at internet scale in 2019, so we treat the floor as a fact of life, not a misconfigurationS.

2.4 Clock 2: the negative TTL§

The negative TTL is the clock that catches people creating a new record. A lookup that returns NXDOMAIN is cached, and a record created a second later stays invisible for the negative TTL. RFC 2308 says NXDOMAIN is cached by name and class, and NODATA by name, type and class ^[9]S. The duration is the lower of the SOA TTL and the SOA MINIMUM.

In our lab, with an SOA TTL of 30 s and a MINIMUM of 6 s, and an authoritative that put the lowered value on the wire as RFC 2308 section 5 expects, BIND cached the NXDOMAIN for about 5.9 s and Unbound for about 6.9 s, and both showed TTL 6 on the first replyM. With SOA TTL and MINIMUM both 8, Unbound cached for about 8.6 s and BIND for about 7.8 sM. A resolver cap on negative answers was honored too: with a 3 s cap, a 20 s negative answer lasted about 3.6 s in Unbound and about 2.3 s in BINDM. In every case the name we created right after its first NXDOMAIN stayed invisible for roughly the negative TTLM. Chapter 3 works the rest.

The lab also produced a finding that needs care. When our test authoritative sent the SOA with its full TTL of 30 s (or 60 s) in the authority section and a MINIMUM of 6 s, a reply a compliant server would not send, Unbound still answered with TTL 6 and cached for about 6.9 s, while BIND cached for about 29.9 s (about 59.9 s)M. That is lab-mode behavior: it depends on the shape of a reply built by a custom scriptable server, and with a compliant reply BIND gave 5.9 s. It is not a claim that BIND ignores the MINIMUM, and not a claim about any production authoritative. What it does show is that “the negative TTL is min(TTL, MINIMUM)” is something two implementations can disagree about when the wire data is not clean, and you only discover which one you have by looking at the TTL on the first NXDOMAIN reply.

2.5 Clock 3: the failure cache§

When a resolver gets no usable answer, RFC 9520 requires it to remember that, for at least 1 second and at most 5 minutes ^[10]S. RFC 2308 allows SERVFAIL to be cached too, with the same 5 minute ceilingS. The point is to stop a resolver from hammering a server that is down. RFC 9520 cites.COM and.NET query rates rising from about 7,000 to 900,000 per second during the October 2021 Facebook outage, which are the RFC’s numbersS.

Documented defaults are short: 5 seconds for SERVFAIL in the CoreDNS cache plugin and 60 seconds for the PowerDNS packet cacheS. We measured none of this. Our authoritative outage test (next section) showed what a client sees, not how often a resolver retries, and retry volume was not measured. For your own resolver’s failure cache, take the number yourself.

2.6 Clock 4: the stale window§

Serve-stale is the one clock that makes an outage shorter instead of longer. Without it, both resolvers stopped answering when the 5 second TTL ran out and the authoritative was unreachable: the last success was at about 5.0 s in Unbound and 4.0 s in BINDM. With serve-stale configured for a 15 second window, Unbound kept answering until about 20 s after the cache fill, which is the 5 s TTL plus 15 s of windowM. BIND’s last success was at about 19.0 sM.

Two limits apply. First, BIND answered only 11 polls and failed 2, and its first failure appeared at 5.0 s, so its answers were intermittent in the window, not continuous. Say “last success”, not “continuous service” (the poll pattern was not examined further)M. Second, we tested one outage mode. We did not compare an unreachable server with a REFUSED one, we did not measure retry volume, and the lag between the authoritative coming back and the resolver recovering is not citable from this run.

The stale reply TTL matched Unbound’s documented serve-expired-reply-ttl default of 30 s ^[14]S. BIND also returned 30 sM, which we report as a measurement only: We did not read the BIND option text that would say whether that is a default.

Serve-stale has a cost. RFC 8767 notes that it can extend the time an attacker’s forged data stays reachable, and that it has DNSSEC implications ^[11]S. A long stale window is also a second delay on top of your TTL when you want clients to see a change, and it is the resolver owner’s decision.

2.7 Clock 5: parent or child§

A delegation exists twice: as NS records in the parent zone and as NS records in your own zone. They have separate TTLs, and a resolver chooses one. RFC 9199 section 3.6.2 says resolvers may use the parent TTL and that operators should keep old infrastructure running for the larger of the parent and child TTLs ^[13]S. It also advises that address records of in-bailiwick name servers carry a TTL no longer than the NS TTLS. Moura et al. found most resolvers child-centric and some parent-centric, in 2019 data from the open internetS.

We tested the two resolvers we had. The lab was three tiers (a root, a parent and a child, all custom scriptable servers), the resolver started from root hints, and we polled once a second with fresh names after moving the delegation at the parent to a new server. Unbound kept the old server for about the child NS TTL: 30.2 s with a parent TTL of 6 s and a child TTL of 30 s, and 6.0 s with the values swappedM. BIND switched after about the parent NS TTL: 5.1 s with parent 6 s and child 30 s, and 29.1 s with the values swappedM.

The conclusion is as narrow as the lab. Two resolvers, two TTL pairs, a synthetic three tier setup, no real TLD servers. In our lab they differed, and that is enough for the advice: when you move a zone between providers, wait for the larger of the parent and child NS TTLs before you switch off the old servers, and do not assume your resolvers all follow the same oneO.

2.8 A worked example: why you lower the TTL early§

The clocks add up, and the simplest sum is one you can do yourself. You have a record with a 20 second TTL and you want to move it. Suppose you lower the TTL to 2 seconds and change the address at once. The resolver already holds a copy that expires up to 20 seconds from when it fetched it, and a new, shorter TTL on the authoritative does nothing to that copy.

We measured exactly this, with an old TTL of 20 s and a new TTL of 2 s. With no TTL change, the stale window was about 19.7 s in Unbound and 18.1 s in BIND. Lowering the TTL right before the change gave 18.9 s and 17.9 s, which barely helped. Lowering the TTL a full old TTL ahead cut the window to about 2.3 s in Unbound and 0.9 s in BINDM. In the ahead case the lab waited out the old entry and made one query first, so that row confirms the arithmetic and does not test resolver behaviour. All six cases were inside tolerance of what the arithmetic predicts.

The same arithmetic shows up when the TTL is left alone. After the authoritative answer changed, each resolver kept serving the old address for about the remaining TTL. Measured minus expected stayed within 0.95 s for Unbound and within 0.72 s for BIND across TTLs of 3, 6 and 12 s flipped at 10, 50 and 90 percent of the TTL, and a TTL of 0 was not reused at allM. In our lab Unbound ran 0.4 to 1 s later than the arithmetic and BIND equal or slightly earlier. We did not investigate whyM. RFC 1034 section 3.6 already advises lowering the TTL before an anticipated change ^[4]S. The lab says how far ahead: a full old TTL.

That is the cache term only. The health check, the change reaching the authoritative and the client caches are not in this lab, so a real cutover is longer than anything in this chapter. Chapter 5 adds them.

2.9 What this chapter did not measure§

We ran two resolvers. A TTL grid beyond 5 and 20 s, TTL 0 in the clamp test, NODATA and SERVFAIL negative caching, retry volume, REFUSED outages and the DS cache were not run. All timings come from one laptop with emulated CPUs and a custom authoritative. They are lower bounds. Two full runs agreed on all 91 cases, with a maximum difference of 0.86 s ^[8]M.

The question: Why is a record you just created still NXDOMAIN for some users?

In the model of chapter 2: the negative clock, the one most operators never set on purpose.

You deploy a service, ask a colleague to open its new hostname, and it fails. You create the record a moment later and it still fails. Nothing is wrong with the record. The resolver asked once, got an answer that said the name does not exist, and kept that answer.

3.1 What is cached, and for how long§

RFC 2308 says an NXDOMAIN answer is cached by name and class, and a NODATA answer (the name exists, the type does not) by name, type and class ^[9]S. So a failed lookup for new.example.org hides the name, and a failed AAAA lookup for an existing name hides only the AAAA. RFC 2308 section 8 makes negative caching mandatory for caching resolvers S.

The duration comes from the zone’s SOA record, not from any record you created. RFC 2308 section 5 sets the negative TTL from the lower of the SOA MINIMUM field and the TTL of the SOA record itself ^[9]S. It suggests capping negative caching at 1 to 3 hours and says values over one day have been found to be problematicS. Most zones carry whatever SOA the provider’s template wrote, and few operators have looked at its last field. The resolver may add its own bounds on top, which chapter 2 listed.

3.2 What we measured§

We tested the rule on Unbound 1.22.0 and BIND 9.20.30 in a Docker lab with one internal network and a scriptable authoritative server we wrote for the experiments. The lab ran on one laptop, with the image CPU emulated on an arm64 host, so allow a few hundred milliseconds of overhead. Each case created the name right after the first NXDOMAIN and timed how long the resolver kept hiding it. A case counts as a match when the measured time is within −1.2 s to +1.5 s of min(SOA TTL, SOA MINIMUM), or of the resolver cap where one was set. The harness ran twice and the two runs gave the same verdicts.

The authoritative has two reply modes. In capped mode it lowers the SOA TTL in the authority section to min(SOA TTL, MINIMUM) before sending, which is what RFC 2308 section 5 expects of the responding server. In uncapped mode it sends the zone’s SOA TTL unchanged. The first round used uncapped mode, which we had not intended, and the second round added capped mode. This matters below. In capped mode the lab server had already applied the minimum, so those rows show that the resolvers honor the wire TTL, not that they compute the minimum.

Behind a compliant reply the two resolvers agree. With SOA TTL 30 s and MINIMUM 6 s, BIND hid the name for about 5.9 s and Unbound for about 6.9 s, and both reported a first reply TTL of 6M. With TTL and MINIMUM both 8 s, Unbound took 8.6 s and BIND 7.8 sM. The one-second spread is inside the tolerance and is mostly whole-second TTL counting plus lab overhead. A configured negative cap won over the SOA in both: a 20 s negative answer was cut to about 3.6 s in Unbound and 2.3 s in BINDM. In our lab, then, a name created right after a failed lookup stayed invisible for roughly the negative TTLM. Every case created the name right after the first NXDOMAIN. We did not run a control that creates the name before the first lookup, so the lab shows the hidden window, not its absence.

3.2.1 The two rows that missed, and what they do not show§

An earlier round of this experiment found BIND hiding the name for about 30 s where Unbound took about 6 s. That round used the lab authoritative in uncapped mode, which we had not intended. The first BIND result was a lab artifact: with a compliant reply, BIND gave 5.9 s, the same as Unbound.

The two rows that remain are the uncapped ones. When the lab authoritative sent an authority SOA with TTL 30 (or 60) and a MINIMUM of 6, a reply a compliant authoritative would not send, BIND 9.20.30 cached the NXDOMAIN for about 29.9 s (about 59.9 s), while Unbound 1.22.0 cached it for about 6.9 sM. The measurements are the same in both runs (29.84 and 29.869 s, 60.01 and 59.904 s). Read this as lab-mode behavior: it is what each resolver did when handed a reply that the lab authoritative built that way. It is not evidence that BIND ignores the SOA MINIMUM, it is not a BIND defect, and it does not contradict RFC 2308.

We looked for an explanation in the BIND manual. The text for max-ncache-ttl and min-ncache-ttl in the 9.20.0 reference manual, which we read for this paper, does not say how the negative TTL is derived from the SOA recordS. That is a statement about those two option texts, not about the whole manual or the source code, and the lab ran 9.20.30 while the page is the 9.20.0 manual of the same stable branch. The manual gives max-ncache-ttl a default of 10800 s (3 hours), with a ceiling of 7 days, and min-ncache-ttl a default of 0 s with a ceiling of 90 s ^[19]S. A 10800 s cap is far above 30 s, so it does not explain the 29.9 s result.

3.3 A worked example§

Take a zone with SOA TTL 3600 s and MINIMUM 300 s. By RFC 2308 the negative TTL is min(3600, 300) = 300 sD. A client resolves api-2.example.org at 12:00:00, before you create the record, and gets NXDOMAIN. You create the record at 12:00:10. Any resolver that behaves as RFC 2308 describes keeps the NXDOMAIN until 12:05:00, so that client’s resolver hides your new record for 4 minutes 50 seconds, and every other client behind the same resolver shares the entryD. Raise MINIMUM to 3600 and the window is nearly an hour. Lower it to 30 and it is 20 seconds. The cost of a low MINIMUM is more authoritative queries for names that do not exist, and we did not measure that cost.

A negative cap in the resolver is a second dial, and it is the resolver operator’s, not yours. With the cap at 3 s, both resolvers we ran hid a 20 s negative answer for about 3 s (3.6 s in Unbound, 2.3 s in BIND). If your users sit behind a resolver you do not run, assume the SOA is the only dial you hold.

3.4 Other places the same answer appears§

Failures: RFC 2308 allows caching SERVFAIL for at most 5 minutes, and RFC 9520 repeats the 1 second to 5 minute bounds for resolution failures ^[10]S. We did not run a SERVFAIL case, so we cannot say how either resolver behaves inside that window.

Runtimes: the Java SE 21 documentation says unsuccessful lookups are cached for 10 seconds by default through networkaddress.cache.negative.ttl, and that a value of −1 means forever ^[1]S. That is a second negative cache stacked on the resolver’s, inside the application. Our Java runs in chapter 6 did not change this setting, so the 10 s figure is the documentation’s, not a measurement.

Small resolvers: the dnsmasq manual says --neg-ttl is used only when the reply carries no SOA ^[17]S. On a normal reply, the SOA you publish still decides. We did not run dnsmasq.

A real case: Slack’s account of its September 2021 DNSSEC rollout says Route 53 signed an NSEC record for a wildcard that omitted type information. Some resolvers then cached “no records of any type”, and A lookups for under 1 percent of users failed for about 24 hours ^[20]S. We report it as Slack’s account. We did not reproduce it, and our lab ran no NODATA, wildcard or DNSSEC case.

3.5 What we did not measure§

We did not measure NODATA against NXDOMAIN scope, an AAAA NODATA hiding an A answer, SERVFAIL windows, replies with no SOA, or any resolver other than Unbound 1.22.0 and BIND 9.20.30. We did not run the Java negative cache setting. The lab authoritative is a custom server, so every claim about what an authoritative sends is lab-mode.

The question: What does a resolver do when it cannot reach your servers, and what does that do to you?

On 4 October 2021 Meta’s authoritative DNS servers withdrew their own BGP advertisements when they could not reach the data centers, and so “became unreachable even though they were still operational” ^[21]S. The servers were healthy. Nobody could ask them anything. Meta also reports that the loss of DNS broke internal toolsS.

For a resolver, a dead authoritative and a withdrawn route look the same: no answer. What the resolver does next decides whether your users see old data, an error, or a retry storm aimed at the servers you are trying to bring back. This chapter covers the three behaviours, then our measurements of the first two.

4.1 Three things a resolver can do§

When a refresh gets no reply, a resolver has three options, and most deployments use a mix.

Answer from cache until the TTL runs out. Before expiry nothing is wrong. The authoritative is not asked, so its absence is invisible. This is the only protection a default configuration gives you, and it is as long as the TTL you chose. A 60 second failover TTL buys one minute of grace, a 24 hour TTL buys a day.

Serve the expired answer. RFC 8767 lets a resolver return an expired record when it cannot get a fresh one ^[11]S. It requires the stale TTL on the reply to be above zero and recommends 30 seconds. It suggests a client response timer of 1.8 seconds, so the client does not wait out a full resolution attempt before getting the stale answer. It suggests keeping stale data for 1 to 3 days at most, and says a failed refresh should not be retried more often than every 30 secondsS. We call the period after expiry the stale window.

Remember the failure and retry. RFC 9520 requires resolvers to cache resolution failures for at least 1 second and at most 5 minutes, and says they should back off ^[10]S. The reason is load. The RFC cites query rates for.COM and.NET rising from about 7,000 to 900,000 per second during the October 2021 Facebook outage, and a 2021 test by Verisign with roughly 50 to 60,000 queries per second for one domain ^[10]S. These are the RFC’s figures, not Meta’s. Meta’s own post says nothing about query load.

4.2 The cost of stale§

Serving stale is not free, and the RFC says so. Section 10 of RFC 8767 notes that serving stale data can extend the time an attacker’s forged data stays reachable, and that it has DNSSEC implicationsS. The second point matters most for your own records: if a stale answer is wrong, or a signature on it has lapsed, the resolver may keep handing it out for the whole window. A long window is a longer exposure to whatever was true when the record was last fresh.

The documented defaults differ widely. The table below sets the RFC values beside what each resolver’s documentation says and what we measured. Documented values are from pages read on 2026-11-11, and not all of every page was read.

CoreDNS documents serve_stale with a default of 1 hour and says stale responses carry TTL 0 ^[16]S. That differs from the RFC’s requirement of a stale TTL above zero, and we did not run CoreDNS, so we cannot say what a deployed CoreDNS does. For BIND we confirmed only that the option names exist (stale-answer-enable, stale-cache-enable, max-stale-ttl, stale-refresh-time, stale-answer-client-timeout) ^[18]S. We did not read the 9.20 stale option text, so no BIND stale default is printed. For PowerDNS Recursor and for Unbound’s serve-expired options beyond those in the table, we did not read the relevant manual text, so we make no claim.

4.3 What we measured§

Times are seconds after the authoritative stopped answering, which is about the moment of the cache fill, from the later of two runsM. The earlier run gave 5.0 s, 20.03 s, 4.0 s and 19.02 s for last success, the same poll counts for the two serve-stale rows and one fewer failed poll for Unbound at default, so the pattern heldM. Both defaults stopped answering when the 5 second TTL ran out. Unbound’s last success was at 5.02 s and BIND’s at 4.03 s, a second earlier, which fits a TTL counted in whole seconds letting a cache drop an entry up to a second earlyM. Neither served a stale record.

Unbound with serve-expired kept answering until 20.06 s. That is the 5 second TTL plus the 15 second window, and the match is the whole result: the configured window is the extension you getM. The poll log had 21 successes and one failure. The stale replies carried a TTL of 30 seconds, which equals the documented serve-expired-reply-ttl default ^[14]M.

BIND needs a more careful reading. With serve-stale enabled its last success was at 19.05 s, close to the same 20 seconds. But its first failure came at 5.02 s, the moment the TTL ended, so the log shows 11 successes and 2 failures in the window, not a clean run of answersM. The answers were intermittent. We did not examine why, and we do not claim it is a fault: it may be how BIND balances refresh attempts against the stale answer, but we did not test that. What the figures support is narrower. A BIND with serve-stale on can still hand a client a failure inside the window, so do not size anything on the assumption of continuous serviceM. Its stale replies carried a TTL of 30 seconds. We did not find a BIND default that documents that value, so it is a measurement onlyM.

4.3.1 The lab settings§

Only the lines that differ from the lab’s base resolver configuration (Appendix A) are listed. These are the exact settings, and a 15 second window is a test value, not advice.

# Unbound 1.22.0
serve-expired: yes
serve-expired-ttl: 15

# BIND 9.20.30 options
stale-answer-enable yes;
stale-cache-enable yes;
max-stale-ttl 15;

4.3.2 Worked example: how long does one record survive?§

Take a record with a 300 second TTL, a resolver whose stale window is 3600 seconds, and an authoritative that goes dark 100 seconds after a cache fill. The cached copy has 200 seconds left, so clients see fresh data until second 300 of its life. From then the resolver serves stale for up to 3600 seconds more. The last stale answer goes out at 300 + 3600 = 3900 seconds after the fill, that is 3800 seconds after the outage began, a little over an hourD. This follows the pattern in our measurement (5 + 15 = 20), scaled up. We did not run these numbers. Add the caveat from the BIND log: where a resolver is intermittent, some clients see failures well before 3900 seconds.

The same arithmetic shows why a short TTL is a smaller buffer than it looks. With the default off, a 60 second TTL gives at most one minute of survival, and a cache that filled 50 seconds earlier has 10 seconds left. Serve-stale is the setting that turns that into hours, and it is off by default in the Unbound manual we readS.

4.4 Incidents: what DNS tails do and do not show§

The Meta outage is the clean case of this chapter: authoritative servers that are up but unreachable ^[21]S. The post gives no TTL or cache figures, so it cannot tell you how long caches covered the gap.

On 14 July 2025, Cloudflare’s 1.1.1.1 resolver service was down for 62 minutes, from 21:52 to 22:54 UTC. A configuration change made on 2025-06-06 stayed dormant until a refresh on 2025-07-14 withdrew the resolver’s prefixes. DNS over HTTPS on a separate hostname and addresses was largely unaffected, and client retries produced a spike after the fix ^[22]S. The post does not say anything about caching or stale answers, and we draw no inference about them.

AWS’s summary of the October 2025 DynamoDB disruption gives a timeline in Pacific time. DNS failures began at 11:48 PM on 19 October. AWS says DNS information was restored at 2:25 AM, that cached DNS records expired between 2:25 and 2:40 AM, and that DynamoDB recovered at 2:40 AM ^[2]S. The summary does not split that tail into resolver caches and client reconnects, and we do not either. It is one statement of one source. What it does show is that after the cause was repaired, caches held the old state for up to about 15 minutesS.

The three incidents are different failures: unreachable authoritative servers at Meta, an unreachable public resolver at Cloudflare, and a regional endpoint with no addresses at AWS. None of the three posts turns its DNS caching into TTL guidance. We read four incident reports for this paper (AWS, Meta, Cloudflare and Slack) and none gave oneS.

4.5 What public resolver documents say§

If your users sit behind a large public resolver, you have no control over its stale policy and little documentation of it. The Cloudflare 1.1.1.1 FAQ states that it validates DNSSEC, sends no EDNS Client Subnet and has a cache purge tool, and says nothing on TTLs, negative caching or serve-stale ^[23]S. The Google Public DNS introduction, read the same day, carries no TTL or serve-stale detail either ^[24]S. Its performance page says only that zero-TTL answers and errors are not cachedS. We make no claim about what either service does. The only public figure we found for a Google cap is Slack’s observation of a 6 hour cap on one DS record, which is Slack’s observation about one record and not a documented clampS.

4.6 What to do§

For records that are stable, a long stale window is cheap insurance against an authoritative outageO. For records you rely on to move traffic, keep the window short or off, and say so in the runbook. RFC 8767′s own risk text, extended attack window and DNSSEC effects, is the reason not to set the window to its maximum by habitS.

4.6.1 Reading the drill§

Read each possible result of that five minute test as follows. The reading is our judgement, not a measured ruleO.

4.6.2 Prove it§

Pause your authoritative in staging, or drop its traffic, for five minutes. Ask your own resolver for one cached record once a second from a probe, and write down three numbers: the last time the old TTL answered, the last time any answer came, and how many failures appeared in between. Compare them with the 5.02 s, 20.06 s and one failure we measured for Unbound.

The question: From the moment a primary dies, how long until users reach the secondary, and which terms of that sum can you shorten?

Two provider documents describe health-checked DNS failover and neither prints the total. Microsoft’s page on Traffic Manager works through a timeline with a 30 second probe interval, 3 tolerated failures, a 10 second timeout and a 30 second TTL, and marks the endpoint degraded after the fourth consecutive failure ^[25]S. Amazon’s pages give the Route 53 health check parameters and recommend 60 or 120 second TTLs for failover records ^{[26, 27]}S. We read both and found no end-to-end cutover time on either page, and we found no source among the 34 we read that composes the piecesS. You have to add the pieces yourself. This chapter adds them, measures the one term a resolver lab can measure, and says which terms stay arithmetic.

5.1 The sum§

A cutover is the time from the failure of the primary to the first successful request on the secondary. We split it into five termsD:

cutover = D + P + K + C + R

D is detection: the time for the health checker to mark the primary down. P is propagation: the time for the changed answer to reach the authoritative servers that answer clients. K is the resolver cache term: the time a resolver keeps serving the old answer after the authoritative changed. C is client cache: anything the application or runtime keeps on top of the resolver. R is connection reuse: a client with an open connection to the dead address never asks DNS again (chapter 6).

Only K can be measured in a resolver lab, and it is the only term we measured. The rest is derived from documentation or left open, and each is labeled so below.

5.1.1 Detection, worked§

Let I be the probe interval, F the number of consecutive failed probes needed to mark the endpoint down, and T the time a failing probe takes to fail (its timeout). Traffic Manager’s documented timeline marks the endpoint degraded after the fourth consecutive failure with 3 tolerated failures, so F is the tolerated failures plus one ^[25]S. Assume probes start every I seconds and the failing ones each wait out the full timeout. The first failing probe starts between 0 and I after the failure. The fourth starts 3 I after the first. It fails T later. That givesD:

D_min = (F - 1) I + T D_max = F I + T

For the documented example, I = 30 s, F = 4, T = 10 s, so D is between 100 and 130 seconds. This is our arithmetic from the documented parameters under the assumptions above. Microsoft does not print that number on the page we readS. Route 53 adds a wrinkle: an endpoint counts as healthy when more than 18 percent of the checkers in many locations report it healthy, and an HTTP check needs a TCP connect within 4 seconds and a response within 2 seconds ^[26]S. We did not model the checker quorum, so for Route 53 treat D_max as a starting estimate and read the provider’s own timing for your configuration.

5.1.2 Cache, then the rest§

If a resolver cached the old answer at time t_fill and the authoritative changes at t_change, the resolver keeps the old answer until it expires. Chapter 2 gives the clamps and caps that can alter the TTL it honors. With the TTL honored as writtenD:

K = TTL - (t_change - t_fill), floored at 0

The next section measures that rule. A resolver that filled just before the change contributes nearly the whole TTL. One that filled a TTL ago contributes nothing. Across many resolvers and many users all phases occur, so a published TTL is a worst case for K, and if cache fill times are spread evenly the typical case is about half of itD.

Table 5.1 applies both formulas to three configurations. Row A is the documented Traffic Manager example. Rows B and C are hypothetical settings we chose to show the spread. They do not describe any product’s defaults.

The sum has not been validated end to end. Our lab had no health checker, no client cache and no connection reuse, so the real cutover is the table’s number plus terms we did not measureD. We treat the table as a lower bound. Figure 5.1 marks the dominant term in each row.

A TTL of 300 seconds in row C means a faster health check buys little: cutting I from 30 s to 10 s there saves about 40 to 60 seconds out of up to 400D. In row A the reverse holds: detection is the larger term and the probe is the larger lever. Check both terms before you tune one.

5.2 What we measured: the cache term§

Experiment E4 tests the K rule. We filled a resolver’s cache with a record, flipped the record at the authoritative at 10, 50 or 90 percent of the TTL after the fill, and polled the resolver every 0.1 seconds until it returned the new address. Expected cutover was the TTL minus the actual time since the fill. We ran Unbound 1.22.0 and BIND 9.20.30 against a scriptable authoritative server we wrote for the lab, with TTLs of 3, 6 and 12 seconds plus a TTL 0 control, 20 cases in all ^[8]M. The harness tolerance was minus 1.2 to plus 1.5 seconds around the expected value.

All 20 cases landed inside the tolerance. The largest gap was 0.95 s late for Unbound and 0.72 s early for BINDM. A TTL 0 record was not reused: the new answer appeared after 0.001 s in Unbound and 0.007 s in BINDM. Both runs of the whole suite agreed on every verdictM. The TTL 12 rows show the rule at three phases:

The cutover shrinks the later the flip lands in the TTL. That is all K is: a countdown that began when some resolver last asked, not when you changed the record. Unbound ran 0.4 to 0.95 s above the arithmetic in every non-zero case and BIND ran equal or up to 0.7 s below. We do not know why. The probe step, whole-second TTL counting and emulated CPU are candidates, and we tested none, so we make no claim about either resolver’s behavior from that offsetM.

5.3 The window before the change: lowering the TTL§

The K rule has a consequence for planned changes. If K is a countdown from the last fill, a TTL you lower today cannot shorten a cache entry that was filled yesterday under the old value. Experiment E5 tests this with an old TTL of 20 seconds and a new TTL of 2 seconds. In three modes we measured the stale window, the time from the record change until the resolver returned the new address ^[8]M:

Lowering the TTL right before the change did almost nothing, since the resolver already held a 20 second entry. Lowering it one old TTL ahead, in our case 22 seconds before a 20 second entry, cut the window to about 2 seconds for Unbound and under 1 second for BINDM. All six cases were within the harness tolerance. The “ahead” run waited out the old entry and then made one query so that the resolver had refilled with the 2 second record before the change. A quiet cache refills only when someone asks, and in production someone usually doesO. The limit is again one old TTL and two resolvers, with one TTL pairM.

The rule for a planned change is the same as the rule for K. Lower the TTL at least one old TTL before the change, change, and raise it again afterO. A zone with a one day TTL needs the lowering a day earlier. A failover record that must be switchable on demand should carry a short TTL permanently, because the change you have to make in an emergency is the one nobody lowered a TTL for.

5.4 When the second address is the fallback§

A record set with two addresses moves part of the work to the client. In E7 the first address was dead and the second alive. Python, Go (both resolver modes) and curl reached the second address within about a quarter of a second when the first refused the connection. A blackholed first address cost one connect timeout or more, and the Java API we ran (OpenJDK 21.0.13, HttpURLConnection) did not use the second address in either case, which is a finding for one JDK build and one API, not for Java in generalM. Chapter 6 has the table. Two addresses helped Python, Go and curl when the failure was a refusal, and helped much less when the failure was silence. We did not compare two addresses against a short TTL, so we do not rank themO.

5.5 Two ways the health check makes it worse§

If every endpoint is unhealthy the check has nothing to offer. Route 53 documents that when all records are unhealthy it treats all of them as healthy rather than return no answer ^[28]S. Traffic Manager documents that when all endpoints are degraded the service responds as if all were online ^[25]S. Both choices keep some answer flowing, and neither can fix a failure of the last good endpoint. Our lab did not implement either behavior, so we did not test them.

Detection is software and it fails too. In the AWS summary for 19 to 20 October 2025, Network Load Balancer health checks flapped and automatic availability-zone failover removed capacity. AWS disabled that failover at 9:36 AM and re-enabled it at 2:09 PM, and the event ran until 2:20 PM on 20 October ^[2]S. A health-checked failover path is a control loop and it can act on a bad signal, so bound how much capacity it may removeO. Chapter 8 tells the DNS automation side of that event.

5.6 A decision table§

The question: After DNS has switched, which of your own clients still go to the old address, and for how long?

Chapters 2 to 5 stopped at the resolver, apart from the preview in chapter 1. A record changed at the authoritative server, the resolver’s copy expired, and the next lookup returned the new address. That is the end of the story only for a client that asks the resolver every time it needs an address. Some do not: in this lab, Java with default settings and a long-lived curl process kept their own copy, and a connection that stays open does not need an address again.

Resolution happens at several layers: the library or runtime cache, the process, the connection, and then the resolver you already know. Each layer can hold the old address for its own time, and the ones that cache by their own rules do not read your TTL. RFC 2181 calls a TTL a maximum time to live, not a mandatory one ^[6]S. A client is free to keep an answer for less, and nothing in a record stops it keeping an answer for more.

We measured one such layer in each of four runtimes: Python, Go, curl and JavaM. We did not measure Node.js, nginx, requests style Python libraries, or any other client. Where this chapter says “clients”, it means those four builds on one lab, and where a conclusion needs more than that, it says so.

6.1 The runtime matrix§

The setup was small on purpose. One Unbound 1.22.0 resolver with a fixed 2 second TTL sat in front of a lab authoritative server. A client made one request every 0.5 seconds, on a new connection each time. About 3.5 seconds into the run we changed the record to a second address, and we logged the first request that reached itM. Python 3.12.16 (glibc 2.36), Go 1.23.12 (with the pure Go and the cgo resolver), curl 7.88.1 and OpenJDK 21.0.13 ran as Linux amd64 images emulated on an arm64 laptop. We ran the whole suite twice, and the two runs agree within a second or so on every case belowM. Times are seconds after the change.

The result splits the clients into two groups. Python (urllib, glibc getaddrinfo), Go in both resolver modes, and a new curl process per request all moved to the new address between 1.76 and 2.45 seconds after the changeM. That is at most the resolver’s 2 second TTL plus one request interval. The spread between 1.76 and 2.45 seconds is where each run landed in the resolver’s refill cycle, not a difference between runtimes (the two runs swapped the order of Python and curl). No client side cache held the old address. For these runtimes the TTL you publish, as clamped by your resolver, is the whole story.

The other group kept the old address on its own clock. OpenJDK 21.0.13 with HttpURLConnection and default settings held the old address for about 30 seconds counted from its first lookup, although the record TTL was 2 secondsM. The change came 3.58 seconds after that lookup, so the first request on the new address arrived 26.66 seconds after the change. We expected 26.42 (30 minus 3.58), and the run came within toleranceM. The Java SE 21 documentation does not promise 30 seconds. It says positive lookups are cached forever when a security manager is installed, and otherwise for an implementation dependent period ^[1]S. So the 30 seconds is what this JDK build did on this setup, not a Java default you can quote.

The cache starts at the client’s lookup, not at your change, so the worst case after a change is the full cache lifetime, which here is 30 seconds if the change lands just after a lookupD. Our run was in the middle of that range, not at its worst.

6.1.1 The Java settings§

Java exposes the cache through the networkaddress.cache.ttl security property, and the documentation says a value of −1 means forever ^[1]S. We set it three ways, through a security properties file, and left the rest aloneM.

• networkaddress.cache.ttl=0: the client followed the resolver and switched 1.68 s after the change.
• networkaddress.cache.ttl=5: the client switched 1.38 s after the change, which is 5 s from its first lookup, as expected.
• networkaddress.cache.ttl=-1: the client did not switch to the new address in a 45 second run. The last of 91 requests that went to the old address was 40.47 s after the change, and there were no errors.

The last row needs its limit. It means “did not switch within 45 seconds”, measured on this build. It does not mean the client never re-resolves, and we did not run it longerM. It is consistent with what the documentation says −1 means.

6.1.2 curl is two programs§

curl 7.88.1 gives two different answers depending on how you run it. A new process per request has an empty cache every time and switched at 1.76 seconds. One long lived process that sent a request every half second through its own DNS cache kept the old address until 55.66 seconds after the changeM. We expected 56.4 seconds from a 60 second cache minus the 3.6 seconds that had already passed, and the run came within tolerance. The command line tool has no option for that cache lifetime, only libcurl does, so we measured only its defaultM.

The same program can behave like two different clients, depending on whether it lives for one request or for a thousand. A health check that shells out to curl and a service that links libcurl may see the world 55 seconds apart.

6.2 Reading the table§

Three points in the table matter more than the numbers.

First, the TTL you publish bounds only the clients on the first line of each group. In the lab, Python, Go and per request curl moved in about 2 seconds, and that figure was set by the resolver’s 2 second TTL, not by anything the clients choseM. For Java and long lived curl, the number that decides your cutover is a client setting you may not know existsO.

Second, the same runtime is two behaviors once you add a setting. Java went from 26.66 seconds to 1.68 seconds to no switch in 45 seconds on one propertyM. That property can be set by an application, by a base image, or by a platform team in a file you never opened. If you cannot name the value in effect for each service, you do not know your cutover timeO.

Third, a table like this is a measurement of a build. Java’s documentation promises an implementation dependent period, so other builds and other vendors may differ ^[1]S. We ran one. The Go and Python rows are equally for the one build each. Print the version next to the number when you test your own.

6.3 The path from change to the last client§

The diagram below puts the cases on one path. The record changes at time zero. The resolver’s copy expires after its own TTL, and from there the paths differ. The fast group asks again on the next request. The Java and long lived curl clients ask only when their own cache has expired, and Java with −1 did not ask within 45 seconds.

An open connection is a further layer we did not measure. Every request in the runtime test used a new connection, so those numbers say nothing about a client that holds one connection for hoursM. The only statement we can make is a vendor’s: Microsoft’s documentation says existing connections are not affected by DNS changes ^[25]S. We read that as a limit of what DNS can do, not as a measured fact about any runtime. Test a keepalive client of your own before assuming it follows the recordO.

6.4 When the first address is dead§

A change in an address is half the problem. The other half is what a client does when a name has two addresses and the first one does not answer. A multi address record set looks like redundancy. Whether it is depends on how the client walks the listO.

We tested two ways of being dead. A refused address answers each connection attempt with an immediate reset. A blackholed address drops every SYN and says nothing. The dead address was listed first and a live one second, with a record TTL of 60 seconds, one request per case, in both runsM. We also tested a dual stack name with a dead AAAA and a live A.

6.4.1 Refused is cheap, blackhole is not§

With a refused first address, Python, Go in both modes and curl reached the second address within about a quarter of a second: 0.01, 0.0, 0.0 and 0.22 secondsM. The curl figure includes process start. A reset costs one round trip, so any client that walks the list can afford to try the next address.

A blackholed address costs a timeout, and this is where the runtimes differ:

• Python with a 3 second connect timeout connected to the live address at 3.02 seconds. It waited out one timeout and then tried the second addressM.
• Go in both modes reached the live address at about 2.0 seconds with a 3 second timeout and at about 15 seconds with no timeout setM.
• curl reached it at 2.52 seconds with a 3 second timeout and 11.01 seconds with noneM.

We did not investigate why Go and curl took 2.0 and 2.5 seconds with a 3 second timeout. With no timeout passed by our harness, Go’s dialer used its own 30 second default and reached the live address after 15.01 seconds. The dial deadline may be shared by the two addresses, which would explain the 15 seconds, but we did not test that. curl took 11.01 seconds for a reason we did not investigateM. Without an explicit connect timeout, a dead first address can cost a client between ten and fifteen seconds in these two runtimes, and the number is not one you chose.

Python without any timeout set had not connected to the live address within the 20 second cap of our harnessM. That limit is the harness, not a Python or kernel timer. The accurate wording is that Python waits, and we cannot say how long.

6.4.2 Java did not try the second address§

OpenJDK 21.0.13 with HttpURLConnection did not reach the live address in any of the three IPv4 cases. With the first address refused, it failed with a ConnectException after 0.01 seconds. With the first address blackholed, it failed with a SocketTimeoutException at 3.02 seconds under a 3 second timeout, and had no success within 20 seconds with the defaultM. The harness expected a fallback in all three, so these are the only IPv4 failures in this test apart from the Python no timeout caseM.

The limit is firm. This is one JDK build and one HTTP API. It is not a statement about all Java, about other HTTP clients on the JVM, or about HttpClient, which we did not run. The address order the client used was the order we listed, with the dead address firstM. The finding is that this client, in this setup, gave up where the others went on to the next address. If your service uses Java for outbound calls, test it, because a second record is not redundancy until a client has used itO.

6.4.3 Dead IPv6, live IPv4: Go and curl moved on, Python waited§

RFC 8305 describes the staggered attempt that addresses the problem for a dual stack name. It sets a 50 millisecond resolution delay and a default 250 millisecond connection attempt delay, with a minimum of 100 ms and a maximum of 2 seconds, and it interleaves address families ^[29]S. With a dead AAAA and a live A, Go in both modes and curl reached the live address in 0.0 to 0.31 seconds across all nine cases, refused or blackholed, with or without a timeoutM. They passed the harness expectation of a live family within 2 seconds, which is the largest delay the RFC allows, so the result is consistent with RFC 8305. We did not isolate the 250 ms default, so we do not claim to have confirmed it.

Python showed no Happy Eyeballs behavior. It connected at 0.01 seconds when the AAAA was refused, at 3.02 seconds when the AAAA was blackholed under a 3 second timeout, and did not succeed within the 20 second cap with no timeoutM. That is a single 3.12.16 build, and the harness did not expect the feature. It is reported as found.

Java connected within 0.02 seconds in all three dual stack cases. We cannot count this as fallback evidence. The address order the client used listed the live address first, so it never met the dead oneM. A test that does not put the dead address first proves nothing about fallback, and ours did not here.

6.5 Composing the client term§

Chapter 5 built cutover as a sum of terms, one of which was the client cache. This chapter supplies that term, and it behaves differently from the others: it is a maximum over your clients, not an averageO. If service A follows the resolver and service B holds a 30 second cache, the time until nobody uses the old address is B’sD.

For the lab, the arithmetic is short. The resolver contributed 2 seconds. Python, Go and per process curl added nothing measurable on top of that, so their last request on the old address came about 1.3 to 2.0 seconds after the changeM. Java at its observed default of 30 seconds counted from its first lookup could hold the old address for up to 30 seconds after a change that lands just after that lookup, and the long lived curl up to 60D. With a 60 second record TTL in front of those caches, the worst case could be the sum of both layers rather than the larger of them, because a client can fill its cache from a resolver copy that is nearly expired or from one that has just been refreshedD. We did not measure that stacked case. The runtime test used a 2 second TTL, so it shows the client cache on its own, not stacked on a long TTL.

The rows that were not run matter as much as those that were. Node.js, nginx, libraries that wrap Python’s HTTP stack, and a Java negative cache setting are all absent, and each is a place where your own services may differ from this tableM.

6.6 Prove it on your own services§

The experiment is cheap, and it is the only way to learn what your build does.

1. Pick a name you control and give it a short TTL, such as 5 seconds, on the authoritative side. Lower it a full old TTL ahead of the test, so the resolver’s copy is not the thing you measureO.
2. Run each service or tool that calls it with a request loop that logs the address it actually connected to and a timestamp. Log the peer address of the socket, not the answer of a separate lookup, because the separate lookup bypasses the cache you want to testO.
3. Change the record to a second address you can see, and note the time.
4. Record when each client last used the old address. The largest value is your client term.
5. Repeat with the first of two addresses refused, then with it blackholed. Note whether a connect timeout is set and what it costs.

Do it once in a staging environment, with the settings your production image carries, and keep the numbers next to your declared cutover time.

6.7 What to do with this§

The measurements argue for one small habit: for every service that calls a name you can change, write down the runtime, the build, the DNS cache setting in effect and the connect timeout, and compare the worst of those to the cutover time you promisedO. The Kubernetes DNS documentation gives a reason to look on pods in particular, since kubelet writes a resolv.conf with three search domains and ndots:5 for each pod ^[30]S. We did not measure what that does to query counts, and chapter 7 covers search lists as a mechanism only.

If the table shows a runtime with a long cache, you have three options. Set the cache property to a value that matches your cutover budget. Shorten the connect timeout so a dead address costs seconds, not tens of seconds. Or stop depending on DNS for the cutover, and put the switch behind an address that does not change. Each costs something and we have not tested any of them in productionO.

The question: Which setups make the same name mean different things to different clients, and which of them can you see from the zone file?

Every chapter so far held the zone fixed and asked how long a copy of it lives. This one asks whether the copy is the one you meant. Three common setups produce answers the author of the zone never sees: split-horizon views, resolver search lists, and a delegation that lives in two places at once. We measured only the last, for two resolvers. The first two are described from documentation and reasoning, and we did not run them. The labels on each claim say which is which.

7.1 Where the author of the zone stops seeing§

A zone file states what one server will say. It does not state which server a client asks, in what order, with what suffix appended, or which copy of the delegation the resolver trusts. The zone owner reads the file. The client sees the result of three other decisions. Figure 7.1 draws the first, and the rest of the chapter takes the other two.

7.1.1 Split-horizon: one name, two answers§

Split-horizon DNS serves different data for the same name depending on who asks, usually an internal address to clients inside a network and a public address to everyone elseD. The zone author sees two files that each look right. The failure sits between them. A resolver cache keys its entries by name and type, and RFC 2181 ranks cached data by where it came from, with authoritative answers above additional-section data ^[6]S. Nothing in that ranking records which view an answer belonged to (reasoning, not in any source we read). If one caching resolver serves both populations, whichever view asked first can fill the entry, and the other population receives it until the TTL runs outD.

This is reasoning from the cache model, not a finding. We did not build a split-horizon lab, so we have no measured leak, no leak duration and no count of resolvers that behave this wayD. What the reasoning does give you is a place to look: the number of distinct views that share one cache. The safe count is one per cache. Two views behind one cache is a configuration to remove or to prove otherwise, and the proof is a query from each side for the same name with a TTL longer than your test.

7.1.2 Search lists: one short name, several queries§

A resolver configuration can carry a search list, a set of suffixes appended to names that look short. The Kubernetes documentation shows an example pod resolv.conf with three search domains and the option ndots:5, and says the kubelet writes the file ^[30]S. The same short name can therefore reach more than one fully qualified nameD, and a name you meant as external may be tried against internal suffixes firstO.

There is no multiplication here. The Kubernetes page we read does not state the query count per lookup, and we ran no capture on a pod, so there is no number hereD. We also did not read the resolv.conf manual page for the exact rule on how ndots decides when a suffix is tried, so treat the sentence above as the consequence to check, not the rule. The check is cheap and is in the callout at the end of this chapter.

7.2 The delegation exists twice§

A delegation is stored by the parent zone as NS records and again by the child as its own NS set. The two copies carry their own TTLs and can differ. A resolver that has both has to pick one. The standards leave room: RFC 9199 section 3.6.2 says resolvers may use the parent TTL, and tells operators to keep the old infrastructure running for the larger of the parent and child TTLs ^[13]S. Moura and co-authors measured the population in 2019 and found most resolvers child-centric and some parent-centric ^[12]S. That is a statement about resolvers on the internet in that year, not about any version you run.

The reason to care is migration. You move a zone to a new provider. You change the NS set at the parent, you lower the TTL there, and you shut the old servers down once the parent TTL has passed. If a resolver follows the child NS TTL instead, it keeps asking the old servers for as long as that TTL says, and the old servers are gone.

7.2.1 Experiment E8: which TTL gets followed§

We tested one thing: when the parent changes a delegation to a new server, how long does the resolver keep using the old server, and does that match the parent TTL or the child TTLM. The lab had three tiers (root, parent, child), each a scriptable authoritative server we wrote, so every statement below is about that lab and not about real top-level servers. The resolver started from root hints. We gave each case a fresh name, moved the delegation at the parent, and polled once a second until the answer came from the new server. The tolerance was −1.2 s to +1.5 s around either expectation, and we ran each case twiceM. The resolvers were Unbound 1.22.0 and BIND 9.20.30 ^[8]M.

All four cases matched one of the two expectations. The two runs agreed to within 0.2 s on every case (for example 30.087 and 30.246 s for the first row). In this lab Unbound followed the child NS TTL and BIND followed the parent NS TTLM. The same change took either 5.1 s or 29.1 s on BIND and either 30.2 s or 6.0 s on Unbound, so the TTL you lowered mattered for one resolver and was ignored by the other.

The result is narrow. It covers two resolvers, two TTL pairs, one lab and one kind of authoritative server, and it is a statement about the NS TTL onlyM. We did not look inside either cache to find why they differ. Reading the RFC 2181 ranking, a resolver that trusts authoritative data over parent-supplied data would lean to the child’s copy, but that is our reading of the rule and not something we measuredD. We make no claim about other versions, other resolvers or real TLD servers.

7.2.2 What follows for a migration§

The lab numbers and the RFC 9199 advice point the same way. If you cannot know which TTL a given resolver follows, the old servers must keep answering for the larger of the twoS. Lowering only the parent TTL to 6 s shortened the switch on BIND and did nothing on Unbound, where the child’s 30 s still ruledM. So the plan is two TTLs and one deadline: lower both, wait out the old maximum, then move, then keep the old servers until the larger value has passed after the changeD.

Suppose the parent keeps the NS set at 48 hours and the child at 1 hour (example values, not measurements)D. A child-centric resolver that cached the old set just before your change can use it for up to 1 hour. A parent-centric one can use it for up to 48 hours. Shutting the old servers after 1 hour breaks the second population for nearly two days. The number to plan with is 48 hours, the larger value, and the cost of choosing the smaller is a slow, partial failure that no health check on the new servers will show, because the clients that fail are still asking the old onesD.

RFC 9199 gives one more rule for in-bailiwick name servers: their A and AAAA records should carry a TTL no longer than the NS TTL ^[13]S. A long address TTL under a short NS TTL leaves clients holding the old server address after the delegation has moved. We did not test it.

7.3 A signed delegation adds a third copy§

DNSSEC puts a DS record at the parent that must match the child’s key. The operator controls the child, not the parent’s TTL. Slack’s engineering post about its 2021 DNSSEC rollout describes what that costs. When they rolled back, the DS record was already cached: the parent.com asks for 24 hours, and Slack observed Google Public DNS capping it at 6 hours. Validating resolvers returned SERVFAIL, so the rollback made things worse ^[20]S. The 6 hours is Slack’s observation about one record. It is not a documented clamp of that service, and the vendor pages we read state none.

We planned a DS cache arm for E8 (a signed zone, a validating resolver, a wrong re-sign) and did not run it. It needed signing tooling and a validating resolver setup that this round did not build, so this chapter has no measured DS duration, for any resolver. Slack’s lesson stands as Slack stated it: do not assume cache behavior. If you sign a zone, the rollback has a TTL you do not control, and the plan has to price that before the first change.

A delegation or a DS record is a cache entry you do not own. Plan for the longest TTL anyone along the path may keep.

7.4 Reading your own delegations§

You can look at both copies of a delegation without any lab. Ask a parent zone’s name server for your zone’s NS set, ask a child server for the same set, and compare names and TTLs. The standard dig tool does this with a non-recursive query to each serverO. Put three lines in your runbook before any migration:

• Whether the two NS sets are identical, name for name. Any difference is a population of resolvers that will see one set or the other.
• Both TTLs, and the larger of the two. That value is the earliest time the old servers can be shut downS.
• The TTL on the address records of any in-bailiwick name server, which RFC 9199 advises should not exceed the NS TTL ^[13]S.

If you control the child but not the parent, as with most domains under a registrar, the parent TTL is a number you read and cannot change. Plan with it as a fixed inputD. The DS record has the same property, which is why a signing change deserves a longer lead time than any other edit to the zoneO.

7.5 What was and was not measured§

The question: What stops a bug in your DNS automation from emptying a record, and what do resolvers do with the result?

In the model of chapter 1: the writer upstream of every clock.

Every chapter so far treated the zone as given. In production the zone is the output of software: a controller that reconciles records, an infrastructure-as-code run, a health check that removes an address. This chapter has no experiment of ours behind it. Every factual claim is sourced (S), derived from a source by arithmetic (D), or our opinion (O), and the tags say which. We did not build or run any DNS automation for this paper.

8.1 The case: automation racing itself§

AWS’s summary of the 19 and 20 October 2025 event gives the root cause as a latent race condition in the DynamoDB DNS management system, which left the regional endpoint with no IP addresses ^[2]S. Its account is this. A delayed Enactor applied an old plan after a newer one had been applied, and the cleanup of old plans deleted that old plan. The system was left inconsistent, no Enactor could apply later plans, and operators had to repair it by hand. AWS disabled the Planner and Enactor automation worldwide until it was fixedS.

Read as a database problem, this is a lost update: two writers hold different versions of the desired state, nothing orders them at write time, and the stale one winsO. That is our reading, not AWS’s wording. Nothing in the summary says the design lacked an ordering check, and we do not claim to know the internals beyond what the summary states.

8.1.1 The tail after the fix§

AWS gives the DNS part of the timeline in PDT: DNS failures from 11:48 PM on 19 October, DNS information restored at 2:25 AM, and recovery at 2:40 AM once cached DNS records had expired ^[2]S. From those stamps, DNS-caused errors lasted about 2 hours 52 minutes, and about 15 minutes of that followed the restoration of the dataD. The summary does not split that 15 minutes into resolver caching and client behaviour, and we do not split it eitherS. It is one unit. The wider event ran to 2:20 PM on 20 October, and AWS says Network Load Balancer health checks flapped and automatic availability zone failover removed capacity until it was disabled at 9:36 AM and re-enabled at 2:09 PMS. A health check acted on a state that automation had already damaged.

8.2 Fail-open is a design choice§

Two providers document the opposite of an empty answer. Route 53 says that when all records in a set are unhealthy, it treats them all as healthy instead of returning no answer ^[28]S. Azure Traffic Manager says that when all endpoints are degraded it responds as if all were online ^[25]S. Both are fail-open: when the checker has nothing good to point at, the service sends traffic to the endpoints it believes are badS.

That is a different failure from the AWS one. A health check that marks everything down is the checker doubting the targets, and these services have a rule for that case. A writer that deletes the record set is the control plane changing the data, and the checker’s rule does not apply. A fail-open check does not protect you from a bad write, and a bad write is not something a probe of the endpoints can see; a probe of the DNS answers canO.

8.3 Slack: the rollback is also a write§

Slack’s account of 30 September 2021 is the other shape. Route 53 signed an NSEC record for a wildcard that omitted type information, some resolvers cached “no records of any type”, and A lookups failed for under 1 percent of users over about 24 hours ^[20]S. The rollback then made it worse. The DS record had been cached, the.com TTL asks for 24 hours, and Slack observed Google Public DNS capping it at 6 hours, so validating resolvers returned SERVFAIL after the rollbackS. The 6 hours is Slack’s observation about that one record, not a documented resolver setting. Slack’s stated lesson is not to assume cache behaviourS. We did not measure DS caching (it was not run, see Appendix A), so this stays Slack’s account.

The point for automation is that undo is a write with the same cache tail as the original change. A pipeline whose only recovery step is “apply the previous state” has not been shown to recover anything until someone has timed that step against real cachesO.

8.4 Guardrails (opinion)§

None of the following is a finding (the whole table is opinion, tag O), and none was said by AWS or Slack. They follow from the shape of the two incidents, and we would want them in any controller, IaC pipeline or health-check hook that can write a record setO.

A floor is cheap to write and a fence is not, because the fence needs shared state between the writers, and that is a design cost you pay onceO. The canary only helps if the check reads the answer a resolver would return, and chapter 9 lists what to watch for thatO. Every guard should fail closed on the write path (refuse and page a person) and open only on the data path, where Route 53 and Traffic Manager put itO.

A fail-open health check protects you from your probes. It does not protect you from your writer.

The question: What do we set, what do we do before a change, and what do we alert on?

In the model of chapter 1: the one clock you control, set on purpose, and the others you watch.

This chapter turns the measurements into rules. Every number marked as ours comes from the lab described in the appendix: Unbound 1.22.0 and BIND 9.20.30 behind a scriptable test authoritative, run twice ^[8]M. Those numbers are lower bounds for a real cutover. The lab had no health checker, one host clock and emulated CPUs, so it measured cache expiry and nothing about detection time. Treat each rule below as a starting point to check against your own resolvers.

9.1 TTL policy§

We start from the published advice and change it only where we measured. Table 9.1 is the policy. The ranges come from the sources, and the choice of a value inside a range is our opinionO.

Two limits on the table. First, a TTL you publish is not the whole delay: client runtimes add their own caches (chapter 6). Second, the lab never tested a TTL above 20 s, so the table’s long values rest on the cited sources and not on our numbersM.

9.2 The change plan§

Figure 9.1 is the order of work. Its first step is the one people skip.

9.2.1 Before: lower the TTL ahead§

In our lab, with a 20 second TTL, lowering it to 2 seconds right before the change barely helped: the stale window stayed at about 18 to 19 seconds. Lowering it one full old TTL ahead cut the window to about 2 seconds in Unbound and under 1 second in BIND (E5)M ^[8]. In the ahead case the lab waited out the old entry and made one query so the resolver held the 2 s record before the change. A resolver that cached the record before you lowered the TTL keeps it for the old TTL, so the lowered value only helps caches that refresh after it.

While you wait, read two more numbers. The SOA negative TTL tells you how long a name that does not exist yet will keep saying so (E2)M. The parent NS TTL matters if the change is a delegation: the two resolvers disagreed on which NS TTL they follow. In a three-tier lab, Unbound 1.22.0 followed the child NS TTL and BIND 9.20.30 the parent NS TTL (E8, lab-mode)M. RFC 9199 tells operators to keep the old infrastructure for the larger of the twoS ^[13].

9.2.2 At the change§

Change the record and leave the old backend running. After the authoritative answer changes, a resolver in our lab kept serving the old address for about the remaining TTL, within 0.95 seconds of that arithmetic in Unbound and within 0.72 seconds in BIND, across TTLs of 3, 6 and 12 seconds (E4)M. Nothing you do at the authoritative shortens that remainder.

9.2.3 After: wait for the slowest cache§

The window after the change is the new TTL plus the longest client cache, so measure the clientD. In our lab a new Python, Go or curl process moved within about 2.5 seconds of a 2 second TTL, while OpenJDK 21.0.13 with default settings held the old address for about 30 seconds from its first lookup and a long-lived curl 7.88.1 process for about 56 seconds. With networkaddress.cache.ttl set to −1, Java did not switch within the 45 second run (E6)M. A worked example: a record at 3600 seconds is lowered to 60 an hour ahead. After the change, a client like the Java default adds about 30 seconds, so we plan for about 90 seconds and still check each client, because that sum is arithmetic and not an end to end measurementD.

A second address is not a safe substitute for waiting. When the first of two addresses was dead, Go, curl and Python reached the live one, but Python waited for its connect timeout on a blackholed address, and the Java HttpURLConnection client we tested failed instead of trying the second address (E7)M. Test your own clients before you rely on it.

When the answers agree from three resolvers and the slowest client has moved, restore the long TTL and retire the old backend. Doing it in this order, with the TTL raised last, is our ruleO.

9.3 Monitoring§

Monitoring is the part we did not measure. These are opinions, with the lab result that motivates eachO.

9.4 Before you change anything§

If you keep one habit from this paper, keep the order of operations: find out what every cache between your zone and your users will do, then change the record. The rest is detail. This page covers what the evidence does not settle.

10.1 Where the lab stops§

We measured two resolvers, Unbound 1.22.0 and BIND 9.20.30, and four client runtimes, behind a test authoritative we wrote ourselves ^[8]M. Every figure is a lower bound for a real cutover, because the lab had no health checker, no anycast, no middleboxes and no real latency. The number your users wait for during a failover includes the time your health check takes to notice, and we did not measure that. The model in chapter 5 gives you the shape of the sumD. Only your own check settings, your own resolvers and your own clients can fill it in.

The documented behaviour of PowerDNS Recursor, CoreDNS and dnsmasq came from their manuals, and we did not run them. The same goes for public resolvers: the vendor pages we read state no TTL clamp, negative cache duration or serve-stale policy (the Google performance page says only that zero-TTL answers and errors are not cached), and we did not test them. We would not guess either way. Appendix A lists every experiment that did not run and why.

10.2 What we would not trust yet§

Two results arrived with a warning attached. The two resolvers followed different NS TTLs in our delegation test (E8)M, and they disagreed about a negative TTL when our test authoritative sent a reply a compliant one would not (E2)M. Both are lab-mode findings. They say what these two versions did with these replies. They are not a claim about either product in general, and a different version could behave differently. The useful conclusion is narrower: do not assume your resolver behaves like the one in the example you copied.

The same caution applies to the Java figures. One JDK build and one HTTP API failed to try a second address in our test (E7)M. That is a reason to test your own clients, not a rule about Java.

10.3 The harness is the contribution§

The scripts, the raw records and the results file sit in the experiments directory of the paper source, and the whole suite ran twice with the same verdict on all 91 cases (maximum difference between runs 0.86 s)M. We would rather you rerun it against your resolver versions and tell us where it disagrees than take any number here on trust. A failing row in that suite is a finding, and we report it as one.

10.4 Next§

Three experiments would change this paper most: a health checker in the loop to measure detection time, split-horizon views with a shared cache, and a signed zone to measure how long validating resolvers keep a bad DS record. They are listed in Appendix A as not run.

If you do one thing after reading this, do the first item in the box at the end of chapter 9: read the negative TTL of your zone and write it down.

This appendix says what was measured and what was not, so that you can tell a finding from a gap.

A.1 What ran§

A failing row is a finding, not a harness error. The six rows outside tolerance are the two BIND cases of E2 with an unlowered SOA TTL, and four client cases in E7 (three Java, one Python).

A.2 Experiment status§

The ids follow the results file, which differs from the plan we wrote before running. The plan’s health-checker failover grid and its split-horizon experiment were not run, and the plan’s ids do not match the ones below, except E1 to E3.

A.3 What was not measured§

• Resolvers. PowerDNS Recursor, CoreDNS, dnsmasq and Knot Resolver were not run. Their behaviour in this paper comes from their documentation, with the page versions and access date given in the text.
• Public resolvers. No test was made against any public DNS service. The vendor pages read state no TTL clamp, negative cache duration or serve-stale policy; the Google performance page says only that zero-TTL answers and errors are not cached.
• Health-check detection. No health checker was in the loop. Every cutover number is the cache term only.
• Split-horizon, resolv.conf and search lists. Views with a shared cache, nameserver ordering, SERVFAIL and REFUSED handling and ndots query counts were not run. Chapter 7 describes the mechanisms only.
• DS caching. No validating resolver was tested, so no DS claim is measured.
• Keepalive connections. Every E6 request used a new connection. The statement that existing connections ignore DNS changes is the vendor’s, not a lab result.
• Real-world conditions. No anycast, no resolver farms, no ISP middleboxes, no loss or jitter. A lab figure is a lower bound for a real cutover.

A.4 Threats to validity§

One host clock and an emulated CPU add overhead to every figure. The poll step limits resolution (0.2 s in E1 and E2). The tolerance window is 2.7 s wide, so a pass on a 3 s or 6 s TTL means consistent with the rule, not equal to it. Each case ran once per run, twice in all, on one machine, so two agreeing runs give no variance estimate and do not show variation between hosts. E7 ran one request per case. The E5 ahead case includes an explicit refill query and is a confirmation of the arithmetic. The authoritative is our own code, and the reply shapes it sends decide some outcomes, as the two BIND cases of E2 show. Each resolver was tested at one version, and each client at one build.

Each list is a working copy. Items marked as opinion are our own rulesO, and the others point to the measurement behind them.

B.1 Change plan§

B.2 Resolver configuration review§

B.3 Incident, first 15 minutes§

These are the tables of the results file for the two runs, with the columns trimmed to those the text uses. The full records, including the per-case authoritative query counts and the run-by-run comparison of all 91 cases, are in the experiments directory of the paper source ^[8].

C.0.1 E1. TTL honored or clamped (seconds)§

unb is Unbound 1.22.0, bind is BIND 9.20.30. def is the default config. Tolerance: measured minus expected within −1.2 s to +1.5 s. Only TTLs of 5 and 20 s were run.

C.0.2 E2. Negative caching duration (seconds)§

Tolerance as E1. Expected is min(SOA TTL, SOA MINIMUM), then the configured cap. The two failing BIND rows are the lab-mode finding of chapter 3: the test authoritative sent an unlowered SOA TTL on the wire.

C.0.3 E3. Serve-stale during an authoritative outage (seconds)§

Tolerance: default, no success after 6.5 s. Serve-stale, last success between 17 s and 22.5 s. A first failure of 5.02 s for BIND with serve-stale is recorded as run and is not explained here.

C.0.4 E4. Cutover after a record flip, cache expiry term only (seconds)§

Tolerance as E1. Expected is the TTL minus the time since the cache fill. No health checker was in the loop.

C.0.5 E5. Stale window after a change, TTL lowered late or ahead (seconds)§

Old TTL 20 s, new TTL 2 s. Tolerance as E1.

C.0.6 E6. Client runtimes: old address still in use (seconds after the change)§

Record TTL 2 s, one request every 0.5 s, a new connection each time. Resolver: Unbound 1.22.0. Runtimes: Python 3.12.16, Go 1.23.12, curl 7.88.1, OpenJDK 21.0.13.

C.0.7 E7. Two addresses, the first dead (seconds)§

Dead address listed first for v4, dead AAAA with live A for v6. Rows with no pass value had no expectation and are reported as found. curl time includes process start. Timeout none means no connect timeout was set, and the harness cap was 20 s.

C.0.8 E8. Which NS TTL the resolver follows (seconds)§

Polled once a second with fresh names. Both rows per resolver matched one of the two expectations. Lab-mode, three-tier lab of custom servers.

Authoritative server

The server that holds a zone’s records and answers for them, as opposed to a resolver that caches answers for clients. In this paper’s lab it is a custom scriptable server, not a production authoritative.

Clamping

A resolver raising a TTL to a configured floor or cutting it to a configured cap before it caches the record. Measured in two resolvers in chapter 2.

Cutover

The time from “the old address stopped being right” to “users reach the new one”. In this paper it is the sum of detection, authoritative change, cache expiry and client behavior.

Detection time

How long a health checker takes to mark an endpoint down: probe interval times tolerated failures, plus the probe timeout.

Delegation

The NS records in a parent zone that say which servers answer for a child zone. Parent and child can carry different TTLs for the NS set.

Negative caching

Caching the fact that a name or type does not exist. RFC 2308 ties its duration to the SOA record ^[9].

NODATA

A reply that the name exists but has no record of the asked type.

NXDOMAIN

A reply that the name does not exist.

Resolver

A server that looks names up for clients and caches what it learns. Also called a recursive resolver.

Serve-stale

Answering from an expired cache entry when the authoritative servers cannot be reached. RFC 8767 describes it and its risks ^[11].

SERVFAIL

A reply that the resolver could not complete the lookup.

SOA MINIMUM

A field in the zone’s SOA record. RFC 2308 uses the lower of the SOA record’s own TTL and this field as the negative-caching time ^[9].

Stub resolver

The small resolver in a client’s operating system or runtime that forwards queries to a full resolver and may cache the answers.

TTL

Time to live. RFC 2181 describes it as a maximum time to live, not a mandatory one ^[6]. The zone administrator sets it, and RFC 1034 notes it may be lowered before an anticipated change ^[4].