Skip to content
Login
Research studio · GLINR Working Papers

Working papers on running software

Long-form papers on the failure modes we meet running our own stack: restores, DNS changes and schema migrations. Each one states how every claim is known, cites primary sources and says what it did not test.

How we writeAtom feed
Topic

Showing 3 of 3 papers

Latest revision first

  • GLINR-WP-2026-101Working paper

    Restore Is the Product

    Backups you can prove: measuring RPO and RTO, mapping threats to surviving copies, and testing restores before you need them

    A backup job reports that it ran. It does not report that your data can come back. In tests on one laptop, four backup jobs that saved nothing useful all exited 0, flipped bytes in a restic repository passed the default check, and a one-database restore that took 0.21 s still needed four rounds of fixes before the application could log in. This paper treats a backup as a system you measure, starting from the restore: four gates a copy must pass, public incidents mapped to the gate each one failed, the 3-2-1-1-0 rule recast as a table of failure domains, and a method to declare and then measure RPO and RTO on a clean host. It closes with a runbook and an evidence checklist. The scripts and raw results are published.

    Reliability, Data, Operations · Practitioner · Revision 0.1.0, November 2026 · 79 min read

    • backups
    • restore testing
    • RPO
    • RTO
    • failure domains
    • immutable backups
    • ransomware
    • disaster recovery
    • restore drills
    Read the paperPDF 1.0 MB
    Cite
    APA
    GLINR Studios. (2026). Restore Is the Product: Backups you can prove: measuring RPO and RTO, mapping threats to surviving copies, and testing restores before you need them (Working paper No. GLINR-WP-2026-101, Version 0.1.0). https://glinr.com/research/2026-101-backups-you-can-restore
    BibTeX
    @techreport{glinr2026wp101,
  author      = {{GLINR Studios}},
  title       = {Restore Is the Product: Backups you can prove: measuring RPO and RTO, mapping threats to surviving copies, and testing restores before you need them},
  institution = {GLINR Studios},
  type        = {Working paper},
  number      = {GLINR-WP-2026-101},
  year        = {2026},
  month       = nov,
  version     = {0.1.0},
  url         = {https://glinr.com/research/2026-101-backups-you-can-restore}
}

    BibTeX fileRIS file

  • GLINR-WP-2026-301Working paper

    DNS in Production

    TTLs, failover and failure modes, measured in two resolvers and four client runtimes

    A DNS change takes effect when several independent clocks expire, and the TTL you set is only one of them. This paper measures the others in a Docker lab with two resolvers, Unbound 1.22.0 and BIND 9.20.30, and four client runtimes: Python, Go, curl and Java. Both resolvers honored configured TTL floors and caps. The cache part of a cutover equaled the remaining TTL to within about 1 s, and lowering a TTL only at the moment of change barely helped. Serve-stale kept answering for the configured window in Unbound, and intermittently in BIND. Client runtimes differed widely in how long they kept the old address and in whether they tried a second address when the first was dead. The paper reads the 2025 AWS DynamoDB outage against these clocks, and ends with a TTL policy, a pre-change runbook and a monitoring checklist. The authoritative server is a custom lab server, so every figure is indicative and a lower bound for a real cutover. The harness and raw results are published.

    Networking, Reliability, Operations · Practitioner · Revision 0.1.0, November 2026 · 106 min read

    • DNS TTL
    • DNS failover
    • negative caching
    • serve-stale
    • resolver TTL clamping
    • DNS cutover time
    • client DNS caching
    • delegation
    • RFC 2308
    • RFC 8767
    Read the paperPDF 1.2 MB
    Cite
    APA
    GLINR Studios. (2026). DNS in Production: TTLs, failover and failure modes, measured in two resolvers and four client runtimes (Working paper No. GLINR-WP-2026-301, Version 0.1.0). https://glinr.com/research/2026-301-dns-in-production
    BibTeX
    @techreport{glinr2026wp301,
  author      = {{GLINR Studios}},
  title       = {DNS in Production: TTLs, failover and failure modes, measured in two resolvers and four client runtimes},
  institution = {GLINR Studios},
  type        = {Working paper},
  number      = {GLINR-WP-2026-301},
  year        = {2026},
  month       = nov,
  version     = {0.1.0},
  url         = {https://glinr.com/research/2026-301-dns-in-production}
}

    BibTeX fileRIS file

  • GLINR-WP-2026-501Working paper

    Zero-Downtime Migrations, Tested

    Lock queues, expand and contract and rolling restarts, measured on Postgres 16 and 17 and nginx

    A one line ALTER TABLE runs in milliseconds, yet it can stall every query on the table for as long as an unrelated transaction stays open. This paper measures that on PostgreSQL 16.15 and 17.11 with a pgbench workload at a fixed rate, then tests lock_timeout with retry, changes that rewrite or scan a table, concurrent indexes, NOT VALID constraints, a rename done in one step versus expand and contract, and a rolling restart behind nginx 1.27.2. A queued ADD COLUMN stalled all reads for about 5 s, and lock_timeout cut each stall to about the timeout. An ordered procedure is derived from the steps, not measured as a whole. The database ran on tmpfs with scaled data, so every time is a shape, not a production time. Replicas, managed services, Kubernetes and migration tools were not tested. The harness and raw results are published at release.

    Deployment, Data, Operations · Practitioner · Revision 0.1.0, November 2026 · 97 min read

    • zero downtime migration
    • Postgres lock queue
    • lock_timeout retry
    • expand and contract
    • CREATE INDEX CONCURRENTLY
    • NOT VALID constraint
    • rolling restart
    • connection draining
    Read the paperPDF 1.1 MB
    Cite
    APA
    GLINR Studios. (2026). Zero-Downtime Migrations, Tested: Lock queues, expand and contract and rolling restarts, measured on Postgres 16 and 17 and nginx (Working paper No. GLINR-WP-2026-501, Version 0.1.0). https://glinr.com/research/2026-501-zero-downtime-migrations
    BibTeX
    @techreport{glinr2026wp501,
  author      = {{GLINR Studios}},
  title       = {Zero-Downtime Migrations, Tested: Lock queues, expand and contract and rolling restarts, measured on Postgres 16 and 17 and nginx},
  institution = {GLINR Studios},
  type        = {Working paper},
  number      = {GLINR-WP-2026-501},
  year        = {2026},
  month       = nov,
  version     = {0.1.0},
  url         = {https://glinr.com/research/2026-501-zero-downtime-migrations}
}

    BibTeX fileRIS file

New papers by email

One note when a paper is published or revised. Nothing else.

Occasional notes on what we ship. Unsubscribe any time.