<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>High Availability on miti99</title><link>https://miti99.com/en/tags/high-availability/</link><description>Recent content in High Availability on miti99</description><generator>Hugo</generator><language>en</language><lastBuildDate>Mon, 16 Nov 2026 17:39:19 +0700</lastBuildDate><atom:link href="https://miti99.com/en/tags/high-availability/index.xml" rel="self" type="application/rss+xml"/><item><title>Newsletter #122</title><link>https://miti99.com/en/post/2026/07/10/</link><pubDate>Fri, 10 Jul 2026 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2026/07/10/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #122.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="the-worst-take-in-tech-new-grad-hiring"&gt;&lt;a href="https://www.scottkennedy.us/new-grads" rel="noopener noreferrer"&gt;The worst take in tech: new grad hiring&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Scott Kennedy pushes back hard against the spreading narrative that companies should stop hiring new grads because AI can automate entry-level work. According to that argument, junior engineers historically learned the craft by doing simple tasks; now that AI handles those tasks, juniors won&amp;rsquo;t develop engineering judgment while budgets get spent on tokens instead. Kennedy calls this dead wrong: over the past year, he has seen the strongest new-grad talent pool in more than a decade of recruiting, and Replit doubled its new-grad hiring target to capitalize on it. Driven, curious, high-slope graduates can outperform many experienced hires within a couple of years when given strong mentorship and hard problems; writing them off because of AI is short-sighted.&lt;/p&gt;
&lt;p&gt;He also points out a distinct advantage of &amp;ldquo;AI-native&amp;rdquo; graduates: having grown up alongside modern AI tools, they have no legacy habits to unlearn and use AI coding workflows with natural fluency—something he sees repeatedly in interviews. Even so, the formula for succeeding as a junior engineer hasn&amp;rsquo;t changed: curiosity to dig beneath the surface and debug deeply, resilience to learn from failed attempts, humility to ask for help when stuck, and seeking out high-challenge environments. AI changes how the work gets done, but it doesn&amp;rsquo;t diminish the value of learning speed, grit, and judgment.&lt;/p&gt;
&lt;h2 id="parsing-gigabytes-of-json-per-second"&gt;&lt;a href="https://arxiv.org/pdf/1902.08318" rel="noopener noreferrer"&gt;Parsing Gigabytes of JSON per Second&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This paper introduces &lt;code&gt;simdjson&lt;/code&gt;, the first standard-compliant JSON parser capable of parsing gigabytes of JSON per second on a single core of a commodity CPU. The motivation is straightforward: JSON is ubiquitous across the web and distributed systems, and when applications ingest massive volumes of JSON documents, parsing easily becomes a primary CPU bottleneck despite decades of parser research.&lt;/p&gt;
&lt;p&gt;What sets &lt;code&gt;simdjson&lt;/code&gt; apart from prior validating parsers is its pervasive use of SIMD vector instructions, allowing the processor to inspect and classify dozens of bytes in parallel within a single instruction. As a result, &lt;code&gt;simdjson&lt;/code&gt; executes roughly one-quarter or fewer CPU instructions than RapidJSON—the fastest traditional parser benchmarked—while still performing full UTF-8 and structural validation. The authors released the library under a permissive open-source license, making the results easy to reproduce and enabling widespread adoption across the industry.&lt;/p&gt;
&lt;h2 id="six-sql-patterns-i-use-to-catch-transaction-fraud"&gt;&lt;a href="https://analytics.fixelsmith.com/posts/sql-fraud-patterns/" rel="noopener noreferrer"&gt;Six SQL patterns I use to catch transaction fraud&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This post presents six practical SQL patterns as a first line of defense for catching anomalous transaction behavior across credit cards, e-commerce, and healthcare billing. The velocity pattern groups transactions into rolling time windows to spot rapid bursts of activity from a single cardholder; combining multiple window sizes catches both automated card-testing scripts firing in seconds and slower fraud sprees spread over hours. The impossible-travel pattern uses &lt;code&gt;LAG&lt;/code&gt; window functions paired with geographic distance formulas to flag a card used in two distant cities within an unrealistic timeframe. The suspicious-amount pattern hunts for small round test charges or transactions clustered just below reporting thresholds, while the merchant-anomaly pattern compares each merchant&amp;rsquo;s activity against its own historical baseline to spot compromised terminals.&lt;/p&gt;
&lt;p&gt;The off-hours pattern compares transaction timestamps against a cardholder&amp;rsquo;s normal spending hours, making it reliable only once an account has built up sufficient history. Finally, a feature-engineering pattern pre-computes derived window columns—such as seconds since previous transaction or trailing 24-hour spend—so combining multiple fraud signals becomes a simple &lt;code&gt;WHERE&lt;/code&gt; filter. No single rule is accurate enough in isolation, whereas a transaction tripping three or four patterns at once is almost certainly fraudulent. In production, queries still need careful &lt;code&gt;NULL&lt;/code&gt; handling, human review for false positives, and date-range partitioning before running window functions to keep warehouse costs in check.&lt;/p&gt;
&lt;h2 id="we-replaced-redis-with-mysql-for-inventory-reservationsand-it-scaled-2026"&gt;&lt;a href="https://shopify.engineering/scaling-inventory-reservations" rel="noopener noreferrer"&gt;We replaced Redis with MySQL for inventory reservations—and it scaled (2026)&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Shopify migrated its flash-sale inventory reservation system from Redis to MySQL so that temporary checkout reservations and the canonical inventory ledger live inside the exact same ACID transaction. Updating two separate stores without atomicity risks overselling or leaked reservations when failures happen mid-flight. Instead of storing a single quantity counter row per SKU—which would serialize every concurrent buyer behind a single row lock—the new schema represents each individual unit of inventory as its own row, capped at a pool of 1,000 rows per SKU-location pair so tables stay compact while queries remain fast even for massive stock counts. MySQL 8&amp;rsquo;s &lt;code&gt;SELECT ... FOR UPDATE SKIP LOCKED&lt;/code&gt; lets concurrent checkouts skip rows currently locked by other buyers instead of waiting in line, drastically cutting lock contention during flash sales. A composite primary key minimizes per-row lock overhead, &lt;code&gt;READ COMMITTED&lt;/code&gt; isolation prevents gap locks from blocking concurrent inventory restocks, and deterministic lock ordering prevents deadlocks.&lt;/p&gt;
&lt;p&gt;During rollout, however, the real bottleneck turned out to be connection pool exhaustion: database CPU sat low while requests queued up waiting for connections. Tagging queries with business workflow metadata and measuring at the proxy layer revealed unrelated checkout operations holding database connections open far too long. After cleaning up connection lifetimes and tuning configs, primary database reads dropped by 50%, total transactions fell by 33%, and writer CPU stayed below 50% even during peak flash sales. To migrate safely, Shopify dual-wrote to both Redis and MySQL while Redis remained primary, shadowed and compared results against live traffic, and shifted primary reads incrementally by cohort with instant rollback ready at every step.&lt;/p&gt;
&lt;h2 id="cell-based-architecture-for-resilient-payment-systems"&gt;&lt;a href="https://americanexpress.io/cell-based-architecture-for-resilient-payment-systems/" rel="noopener noreferrer"&gt;Cell-Based Architecture for Resilient Payment Systems&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;American Express structures its core payment processing platform into self-contained cells, where each cell bundles its own services, databases, and supporting infrastructure. A cell is defined as a fault-isolation boundary rather than a specific deployment technology: an entire cell can fail, undergo maintenance, or be drained of traffic without impacting neighboring cells. Slowly changing reference data like currency exchange rates is replicated into every cell ahead of time to avoid synchronous cross-cell lookups, while stateful transactions are routed deterministically to the cell holding the authoritative state. A Global Transaction Router governs all ingress traffic and strictly forbids services inside one cell from calling directly into another; while this sometimes requires duplicating services across cells, it preserves hard fault isolation and keeps latency low.&lt;/p&gt;
&lt;p&gt;When a cell fails mid-transaction, in-flight requests aren&amp;rsquo;t resumed halfway through on another cell—they are replayed from the beginning on a healthy cell using the original request payload. That retry is safe because a globally unique transaction ID enforces end-to-end idempotency, allowing downstream systems to detect and deduplicate repeated work. Traffic shifts across cells gradually by percentage rather than flipping all at once. The platform also minimizes non-essential dependencies on the critical path by making audit logging and config refreshes asynchronous, gracefully degrading auxiliary features so payment authorization never stalls. True resilience comes from strict blast-radius boundaries and architectural discipline, not just monitoring and retries.&lt;/p&gt;
&lt;h2 id="when-failover-isnt-safe-building-high-availability-postgresql-on-kubernetes"&gt;&lt;a href="https://www.datadoghq.com/blog/engineering/postgresql-ha-kubernetes/" rel="noopener noreferrer"&gt;When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;During a simulated availability-zone failure game day, Datadog discovered that their high-availability PostgreSQL clusters on Kubernetes could not fail over safely. When network latency spiked during the drill, the primary kept accepting writes and replicating asynchronously while standby replicas fell further and further behind. When the primary was terminated, Patroni rightly refused to promote any standby because their replication lag exceeded &lt;code&gt;maximum_lag_on_failover&lt;/code&gt;—preventing silent data loss, but leaving the cluster wedged and unable to accept writes until human operators intervened. The first lesson: nominal high availability means nothing if no replica is actually safe to promote when disaster strikes.&lt;/p&gt;
&lt;p&gt;Datadog moved to a hybrid replication topology: standby instances eligible for failover use synchronous replication, while read-only replicas stay asynchronous to avoid unnecessary latency penalties. In Patroni, they enabled &lt;code&gt;synchronous_mode&lt;/code&gt;, &lt;code&gt;synchronous_mode_strict&lt;/code&gt;, and &lt;code&gt;synchronous_commit = remote_apply&lt;/code&gt;, guaranteeing a transaction is acknowledged only after its WAL records have been replayed and applied on the synchronous standby. Synthetic benchmarks at this highest durability setting showed average write latency increasing by 53% and throughput dropping by 34%, but rolling the change out gradually across datacenters with soak periods confirmed that real-world application impact remained well within acceptable bounds. If all synchronous standbys become unavailable, the primary halts writes rather than silently risking data loss, turning an invisible durability hazard into an explicit error that callers can retry or queue. Balancing durability and performance requires actively monitoring &lt;code&gt;SyncRep&lt;/code&gt; wait events alongside standby health.&lt;/p&gt;
&lt;h2 id="from-christmas-outage-to-1-app-store-ranking-an-aura-frames-postgres-scaling-retrospective"&gt;&lt;a href="https://andyatkinson.com/postgresql-rds-scaling-aws-christmas-day-peak" rel="noopener noreferrer"&gt;From Christmas Outage to #1 App Store Ranking: An Aura Frames Postgres Scaling Retrospective&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;On Christmas Day 2024—the biggest traffic day of the year for digital photo frames—Aura Frames suffered a three-hour PostgreSQL outage when unbounded WAL growth filled their dedicated WAL volume. The culprit was an RDS PostgreSQL 14.1 default that attached replication slots to read replicas: a replication slot forces the primary to retain WAL segments until every replica has consumed them, so when a replica fell behind under holiday load, WAL piled up until the primary ran out of disk space and crashed. The team immediately capped slot retention using &lt;code&gt;max_slot_wal_keep_size&lt;/code&gt; (noting that &lt;code&gt;wal_keep_size&lt;/code&gt; sets a minimum floor, not a ceiling), choosing to invalidate a lagging replica rather than let the primary go down.&lt;/p&gt;
&lt;p&gt;To prepare for Christmas 2025, instead of vertically scaling a single massive primary again, Aura partitioned at the table level (functional sharding): the ten highest-write tables were split out onto seven new primaries, creating eight databases in total. This proved a pragmatic middle ground before needing horizontal row sharding on their largest table. The Ruby on Rails monolith stayed intact, routing queries cleanly via Active Record Multiple Databases, while physical replication moved data over with just 5–10 minutes of cutover downtime. Total provisioned CPU and memory increased roughly 4.7x, and on Christmas Day 2025 the system peaked at 226,000 transactions per second—sustaining over 100,000 TPS for ten straight hours at an average query latency of 25 microseconds. Continuous testing, canary rollouts, and load testing de-risked the data-access changes; temporarily over-provisioning for a predictable annual peak made complete sense, and AWS Blue/Green deployments let them scale the fleet back down afterward to restore cost efficiency.&lt;/p&gt;
&lt;h2 id="gitignore-isnt-the-only-way-to-ignore-files-in-git"&gt;&lt;a href="https://nelson.cloud/.gitignore-isnt-the-only-way-to-ignore-files-in-git/" rel="noopener noreferrer"&gt;.gitignore Isn’t the Only Way To Ignore Files in Git&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Git supports three distinct scopes for ignoring files, each suited to a different job. &lt;code&gt;.gitignore&lt;/code&gt; is committed into the repository, making it the right place for project-wide rules shared by every contributor, like build outputs and dependency folders. &lt;code&gt;.git/info/exclude&lt;/code&gt; lives inside your local &lt;code&gt;.git&lt;/code&gt; directory and is never tracked or pushed, applying only to your local clone—ideal for personal scratch files, local scripts, or project-specific tool configs that don&amp;rsquo;t belong in the team&amp;rsquo;s shared &lt;code&gt;.gitignore&lt;/code&gt;. Finally, &lt;code&gt;~/.config/git/ignore&lt;/code&gt; acts as your user-wide global ignore file across every repository on your machine, perfect for OS- or editor-generated noise like macOS &lt;code&gt;.DS_Store&lt;/code&gt; files or editor swap files.&lt;/p&gt;
&lt;p&gt;You can customize the location of the global ignore file via &lt;code&gt;git config --global core.excludesFile ~/.gitignore_global&lt;/code&gt; and reset it to the default with &lt;code&gt;git config --global --unset core.excludesFile&lt;/code&gt;. And whenever you aren&amp;rsquo;t sure which rule across those three layers is causing a file to be ignored, &lt;code&gt;git check-ignore -v &amp;lt;path&amp;gt;&lt;/code&gt; is the fastest way to debug it: the command prints the exact ignore file, line number, and pattern that matched—or outputs nothing if the file isn&amp;rsquo;t ignored.&lt;/p&gt;
&lt;h2 id="the-only-scalable-delete-in-postgres-is-drop-table"&gt;&lt;a href="https://planetscale.com/blog/the-only-scalable-delete" rel="noopener noreferrer"&gt;The only scalable delete in Postgres is DROP TABLE&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;In PostgreSQL, running a large &lt;code&gt;DELETE&lt;/code&gt; doesn&amp;rsquo;t immediately free disk space—it creates a mountain of follow-up work for the database. Under MVCC, deleted rows remain on disk as dead tuples so concurrent transactions can still see a consistent snapshot; autovacuum eventually marks that space reusable internally, but rarely shrinks the underlying data files on disk (except when completely empty pages sit at the very end of a relation), and indexes and sequential scans still pay the cost of skipping over dead tuples in the meantime. Mass deletes also generate heavy WAL volume that must be replicated downstream, inflating write latency, vacuum debt, and replica lag—while cascading foreign-key deletes can unexpectedly amplify a small delete into gigabytes of churn. &lt;code&gt;VACUUM FULL&lt;/code&gt; reclaims OS disk space, but only by taking an expensive exclusive table lock and rewriting the entire table.&lt;/p&gt;
&lt;p&gt;By contrast, &lt;code&gt;DROP TABLE&lt;/code&gt; and &lt;code&gt;TRUNCATE&lt;/code&gt; acquire an &lt;code&gt;AccessExclusiveLock&lt;/code&gt; but execute in nearly constant time regardless of row count: they unlink the underlying physical files directly without generating per-row dead tuples or vacuum debt. Consequently, schemas that need regular bulk purges should be designed so large deletes become table drops. For time-series or retention-bound data, time-based table partitioning turns recurring deletes into instantly dropping the oldest partition. For a one-off cleanup where most rows are being deleted, copying the surviving rows into a temporary table, running &lt;code&gt;TRUNCATE&lt;/code&gt; on the original, and inserting them back avoids per-row delete WAL (though structural and re-insert WAL still applies); if you cannot afford a long lock, you can dual-write and backfill into a fresh table before swapping names in a brief transaction, taking care to handle foreign keys, sequences, and concurrent writes. When the rows to delete are a small fraction of the table, batched deletes in small transactions remain the practical choice so autovacuum can keep up between batches.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.&lt;/em&gt;&lt;/p&gt;</description></item><item><title>Newsletter #75</title><link>https://miti99.com/en/post/2025/12/28/</link><pubDate>Sun, 28 Dec 2025 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2025/12/28/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #75.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="front-end-vs-back-end-vs-system-design--what"&gt;&lt;a href="https://designgurus.substack.com/p/front-end-vs-back-end-vs-system-design" rel="noopener noreferrer"&gt;Front-End vs. Back-End vs. System Design – What&amp;rsquo;s the Difference in Interviews?&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Front-end, back-end, and system design interviews evaluate very different skill sets, so each requires its own preparation strategy. Front-end interviews revolve around the client side: HTML, CSS, JavaScript, and usually a framework like React. They place less emphasis on complex algorithms and lean more toward domain knowledge like responsive design, CSS specificity, browser rendering pipelines, asynchronous JavaScript, and load-time optimization. Back-end interviews are more traditional: LeetCode-style data structures and algorithms with complexity analysis, paired with practical knowledge of APIs, databases (SQL and NoSQL, indexing, schema design), caching, and message queues. System design interviews zoom out to the big picture, testing your ability to architect a real-world system from scratch that is scalable, reliable, and maintainable.&lt;/p&gt;
&lt;p&gt;The author uses a house-building analogy: front-end is like interior design (how the rooms look and feel), back-end is like the structure and foundation (whether the house stands firm and utilities work properly), and system design is like the urban architect planning the entire neighborhood, deciding where houses, roads, and utilities go. To prepare for front-end interviews, build a few small projects and brush up on browser fundamentals and HTTP. For back-end, practicing algorithms and mastering database fundamentals remain the core focus.&lt;/p&gt;
&lt;h2 id="why-starting-with-microservices-can-be-your-biggest-architectural-mistake"&gt;&lt;a href="https://designgurus.substack.com/p/why-starting-with-microservices-can" rel="noopener noreferrer"&gt;Why Starting with Microservices Can Be Your Biggest Architectural Mistake&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Microservices are often treated as the industry default, but starting a brand-new project with microservices is a common and costly mistake: what works for a 5,000-engineer organization is usually poison for a five-person team. The core advice is &amp;ldquo;Monolith First.&amp;rdquo; A monolith&amp;rsquo;s superpower is simplicity: all code lives in one place, deploys as a single unit, and function calls happen instantaneously in the same memory space. Microservices introduce three nightmares. First, networks are unreliable, turning simple function calls into distributed systems problems with retries and timeouts. Second, each service owns its own database, so you can no longer &lt;code&gt;JOIN&lt;/code&gt; tables and must stitch data together in application code. Third, 10 services mean 10 build pipelines, 10 servers to monitor, and 10 places to dig through logs.&lt;/p&gt;
&lt;p&gt;At the start of a project, you also don&amp;rsquo;t yet know where the domain boundaries should be. Refactoring a monolith is just moving files between folders, whereas refactoring microservices requires changing API contracts, migrating databases, and coordinating multi-service deployments. A pragmatic middle ground is the Modular Monolith: a single codebase cleanly organized into modules like &lt;code&gt;/src/users&lt;/code&gt; and &lt;code&gt;/src/orders&lt;/code&gt; that still run inside a single runtime. This gives you the organizational clarity of microservices without the operational tax. Only move to microservices when the team outgrows the monolith (around 100 engineers), when individual features have vastly different scaling needs, or when you genuinely need polyglot technology stacks. Don&amp;rsquo;t optimize prematurely; wait until the pain is real.&lt;/p&gt;
&lt;h2 id="the-linux-kernel-is-just-a-program"&gt;&lt;a href="https://serversfor.dev/linux-inside-out/the-linux-kernel-is-just-a-program/" rel="noopener noreferrer"&gt;The Linux kernel is just a program&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This article demystifies the Linux kernel—often treated as an impenetrable &amp;ldquo;black box&amp;rdquo;—through hands-on experiments. At its core, the kernel provides an abstraction layer for using hardware conveniently and safely: a unified API for interacting with hardware, managing how programs share CPU, memory, and resources, and enforcing access control. The closest analogy is that the kernel is simply a &amp;ldquo;runtime&amp;rdquo; for your computer. On most distributions, the kernel is just a compressed file of a few megabytes named &lt;code&gt;vmlinuz-…&lt;/code&gt; sitting in &lt;code&gt;/boot&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;The author copies that file out and boots it directly with QEMU; the kernel starts up in about two seconds and then panics because it cannot find a root filesystem to hand off execution to an &lt;code&gt;init&lt;/code&gt; program—which is the exact expected behavior. Next, the author writes a tiny &lt;code&gt;init&lt;/code&gt; program in Go, packs it into a minimal &lt;code&gt;initramfs&lt;/code&gt;, and boots again. This time, the kernel runs the Go binary as PID 1. From this experiment, several core concepts click into place: a Linux distribution is just the kernel plus userland programs and configuration files; a process is a running program; a PID is a process identifier; &lt;code&gt;init&lt;/code&gt; is the very first process, responsible for starting other programs; and &lt;code&gt;init&lt;/code&gt; is the first program running in user space, where the kernel hands over control after initialization. This practical approach builds a clear mental model of how Linux actually works.&lt;/p&gt;
&lt;h2 id="why-your-load-balancer-is-the-bottleneck-and-how-to-fix-it"&gt;&lt;a href="https://designgurus.substack.com/p/dont-let-your-load-balancer-crash" rel="noopener noreferrer"&gt;Why Your Load Balancer is the Bottleneck (and How to Fix It)&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Distributed systems have an ironic paradox: you add a load balancer to distribute traffic evenly across application servers, but when your product takes off, the load balancer itself becomes the bottleneck. Your application servers are healthy and your database is barely breaking a sweat, yet users complain that the site is slow or unreachable. According to the author, this is what separates junior developers from senior architects: plenty of people know how to put a load balancer in front of a service, but far fewer know what to do when the load balancer itself is the problem. The post explains why this happens and introduces five scaling strategies used by major tech companies.&lt;/p&gt;
&lt;p&gt;Strategies covered include DNS Round Robin to split traffic right at the DNS layer, choosing between Layer 4 and Layer 7 routing to reduce processing overhead, and Direct Server Return (DSR), which lets backend servers send responses straight back to the client without passing back through the load balancer. Each approach comes with trade-offs: DNS Round Robin is simple but slow to propagate changes, Layer 4 is faster than Layer 7 but offers fewer routing features, and DSR saves massive egress bandwidth at the cost of complex network configuration. Mastering these techniques lets your load-balancing tier scale horizontally alongside your application servers instead of becoming a single point of congestion.&lt;/p&gt;
&lt;h2 id="the-high-availability-blueprint-designing-systems-that-never-sleep"&gt;&lt;a href="https://designgurus.substack.com/p/the-high-availability-blueprint-designing" rel="noopener noreferrer"&gt;The High Availability Blueprint: Designing Systems That Never Sleep&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;High Availability (HA) is the art of building systems that &amp;ldquo;never sleep&amp;rdquo; on top of hardware that is guaranteed to fail. HA isn&amp;rsquo;t just about preventing failures—it&amp;rsquo;s about masking them so users never notice. Availability is measured in &amp;ldquo;nines&amp;rdquo;: 99% equates to about 3.65 days of downtime per year, 99.9% is about 8.76 hours, and 99.999% (&amp;ldquo;five nines&amp;rdquo;) is just over 5 minutes; every additional nine is exponentially harder and more expensive to achieve. The core philosophy is &amp;ldquo;assume everything will fail and plan for it.&amp;rdquo; The biggest enemy is a Single Point of Failure (SPOF)—any component whose failure brings down the entire system—and the antidote is redundancy: duplicating critical components.&lt;/p&gt;
&lt;p&gt;Key techniques include load balancers paired with health checks to automatically remove unhealthy servers from rotation; Leader-Follower database replication where a Follower is promoted to Leader upon failure; and failover strategies in two flavors. Active-Passive is simple but wasteful since standby servers sit idle most of the time. Active-Active utilizes all hardware continuously but is more complex, and in a two-node setup each node must be provisioned to handle 100% of the traffic if its peer goes down. Additionally, rate limiting (returning HTTP 429 when thresholds are exceeded) prevents cascading failures during traffic spikes. Achieving five nines requires geographic distribution across multiple regions to survive regional disasters. The guiding mantra: &amp;ldquo;Two is one, and one is none&amp;rdquo;—always have a backup plan.&lt;/p&gt;
&lt;h2 id="bonus"&gt;Bonus&lt;/h2&gt;&lt;h3 id="images"&gt;Images&lt;/h3&gt;&lt;p&gt;&lt;img src="https://substackcdn.com/image/fetch/$s_!ETm5!,w_1100,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2112a03d-cc48-4db5-9740-94adc1a7efbf_2360x2920.png" alt="Common Network Protocols Every Engineer Should Know"&gt;
&lt;img src="https://substackcdn.com/image/fetch/$s_!5bFA!,w_1100,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F722913d5-a10f-44c2-98ed-edc27a92a137_1444x1882.jpeg" alt="8 Popular Network Protocols"&gt;
&lt;img src="https://substackcdn.com/image/fetch/$s_!pen8!,w_1100,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F22fdf2d2-5ce5-4c2b-90f9-3bcfcecb5fc1_1370x1536.jpeg" alt="9 best practices for developing microservices"&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.&lt;/em&gt;&lt;/p&gt;</description></item></channel></rss>