<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Infrastructure on miti99</title><link>https://miti99.com/en/tags/infrastructure/</link><description>Recent content in Infrastructure on miti99</description><generator>Hugo</generator><language>en</language><lastBuildDate>Mon, 16 Nov 2026 17:39:19 +0700</lastBuildDate><atom:link href="https://miti99.com/en/tags/infrastructure/index.xml" rel="self" type="application/rss+xml"/><item><title>Newsletter #122</title><link>https://miti99.com/en/post/2026/07/10/</link><pubDate>Fri, 10 Jul 2026 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2026/07/10/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #122.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="the-worst-take-in-tech-new-grad-hiring"&gt;&lt;a href="https://www.scottkennedy.us/new-grads" rel="noopener noreferrer"&gt;The worst take in tech: new grad hiring&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Scott Kennedy pushes back hard against the spreading narrative that companies should stop hiring new grads because AI can automate entry-level work. According to that argument, junior engineers historically learned the craft by doing simple tasks; now that AI handles those tasks, juniors won&amp;rsquo;t develop engineering judgment while budgets get spent on tokens instead. Kennedy calls this dead wrong: over the past year, he has seen the strongest new-grad talent pool in more than a decade of recruiting, and Replit doubled its new-grad hiring target to capitalize on it. Driven, curious, high-slope graduates can outperform many experienced hires within a couple of years when given strong mentorship and hard problems; writing them off because of AI is short-sighted.&lt;/p&gt;
&lt;p&gt;He also points out a distinct advantage of &amp;ldquo;AI-native&amp;rdquo; graduates: having grown up alongside modern AI tools, they have no legacy habits to unlearn and use AI coding workflows with natural fluency—something he sees repeatedly in interviews. Even so, the formula for succeeding as a junior engineer hasn&amp;rsquo;t changed: curiosity to dig beneath the surface and debug deeply, resilience to learn from failed attempts, humility to ask for help when stuck, and seeking out high-challenge environments. AI changes how the work gets done, but it doesn&amp;rsquo;t diminish the value of learning speed, grit, and judgment.&lt;/p&gt;
&lt;h2 id="parsing-gigabytes-of-json-per-second"&gt;&lt;a href="https://arxiv.org/pdf/1902.08318" rel="noopener noreferrer"&gt;Parsing Gigabytes of JSON per Second&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This paper introduces &lt;code&gt;simdjson&lt;/code&gt;, the first standard-compliant JSON parser capable of parsing gigabytes of JSON per second on a single core of a commodity CPU. The motivation is straightforward: JSON is ubiquitous across the web and distributed systems, and when applications ingest massive volumes of JSON documents, parsing easily becomes a primary CPU bottleneck despite decades of parser research.&lt;/p&gt;
&lt;p&gt;What sets &lt;code&gt;simdjson&lt;/code&gt; apart from prior validating parsers is its pervasive use of SIMD vector instructions, allowing the processor to inspect and classify dozens of bytes in parallel within a single instruction. As a result, &lt;code&gt;simdjson&lt;/code&gt; executes roughly one-quarter or fewer CPU instructions than RapidJSON—the fastest traditional parser benchmarked—while still performing full UTF-8 and structural validation. The authors released the library under a permissive open-source license, making the results easy to reproduce and enabling widespread adoption across the industry.&lt;/p&gt;
&lt;h2 id="six-sql-patterns-i-use-to-catch-transaction-fraud"&gt;&lt;a href="https://analytics.fixelsmith.com/posts/sql-fraud-patterns/" rel="noopener noreferrer"&gt;Six SQL patterns I use to catch transaction fraud&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This post presents six practical SQL patterns as a first line of defense for catching anomalous transaction behavior across credit cards, e-commerce, and healthcare billing. The velocity pattern groups transactions into rolling time windows to spot rapid bursts of activity from a single cardholder; combining multiple window sizes catches both automated card-testing scripts firing in seconds and slower fraud sprees spread over hours. The impossible-travel pattern uses &lt;code&gt;LAG&lt;/code&gt; window functions paired with geographic distance formulas to flag a card used in two distant cities within an unrealistic timeframe. The suspicious-amount pattern hunts for small round test charges or transactions clustered just below reporting thresholds, while the merchant-anomaly pattern compares each merchant&amp;rsquo;s activity against its own historical baseline to spot compromised terminals.&lt;/p&gt;
&lt;p&gt;The off-hours pattern compares transaction timestamps against a cardholder&amp;rsquo;s normal spending hours, making it reliable only once an account has built up sufficient history. Finally, a feature-engineering pattern pre-computes derived window columns—such as seconds since previous transaction or trailing 24-hour spend—so combining multiple fraud signals becomes a simple &lt;code&gt;WHERE&lt;/code&gt; filter. No single rule is accurate enough in isolation, whereas a transaction tripping three or four patterns at once is almost certainly fraudulent. In production, queries still need careful &lt;code&gt;NULL&lt;/code&gt; handling, human review for false positives, and date-range partitioning before running window functions to keep warehouse costs in check.&lt;/p&gt;
&lt;h2 id="we-replaced-redis-with-mysql-for-inventory-reservationsand-it-scaled-2026"&gt;&lt;a href="https://shopify.engineering/scaling-inventory-reservations" rel="noopener noreferrer"&gt;We replaced Redis with MySQL for inventory reservations—and it scaled (2026)&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Shopify migrated its flash-sale inventory reservation system from Redis to MySQL so that temporary checkout reservations and the canonical inventory ledger live inside the exact same ACID transaction. Updating two separate stores without atomicity risks overselling or leaked reservations when failures happen mid-flight. Instead of storing a single quantity counter row per SKU—which would serialize every concurrent buyer behind a single row lock—the new schema represents each individual unit of inventory as its own row, capped at a pool of 1,000 rows per SKU-location pair so tables stay compact while queries remain fast even for massive stock counts. MySQL 8&amp;rsquo;s &lt;code&gt;SELECT ... FOR UPDATE SKIP LOCKED&lt;/code&gt; lets concurrent checkouts skip rows currently locked by other buyers instead of waiting in line, drastically cutting lock contention during flash sales. A composite primary key minimizes per-row lock overhead, &lt;code&gt;READ COMMITTED&lt;/code&gt; isolation prevents gap locks from blocking concurrent inventory restocks, and deterministic lock ordering prevents deadlocks.&lt;/p&gt;
&lt;p&gt;During rollout, however, the real bottleneck turned out to be connection pool exhaustion: database CPU sat low while requests queued up waiting for connections. Tagging queries with business workflow metadata and measuring at the proxy layer revealed unrelated checkout operations holding database connections open far too long. After cleaning up connection lifetimes and tuning configs, primary database reads dropped by 50%, total transactions fell by 33%, and writer CPU stayed below 50% even during peak flash sales. To migrate safely, Shopify dual-wrote to both Redis and MySQL while Redis remained primary, shadowed and compared results against live traffic, and shifted primary reads incrementally by cohort with instant rollback ready at every step.&lt;/p&gt;
&lt;h2 id="cell-based-architecture-for-resilient-payment-systems"&gt;&lt;a href="https://americanexpress.io/cell-based-architecture-for-resilient-payment-systems/" rel="noopener noreferrer"&gt;Cell-Based Architecture for Resilient Payment Systems&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;American Express structures its core payment processing platform into self-contained cells, where each cell bundles its own services, databases, and supporting infrastructure. A cell is defined as a fault-isolation boundary rather than a specific deployment technology: an entire cell can fail, undergo maintenance, or be drained of traffic without impacting neighboring cells. Slowly changing reference data like currency exchange rates is replicated into every cell ahead of time to avoid synchronous cross-cell lookups, while stateful transactions are routed deterministically to the cell holding the authoritative state. A Global Transaction Router governs all ingress traffic and strictly forbids services inside one cell from calling directly into another; while this sometimes requires duplicating services across cells, it preserves hard fault isolation and keeps latency low.&lt;/p&gt;
&lt;p&gt;When a cell fails mid-transaction, in-flight requests aren&amp;rsquo;t resumed halfway through on another cell—they are replayed from the beginning on a healthy cell using the original request payload. That retry is safe because a globally unique transaction ID enforces end-to-end idempotency, allowing downstream systems to detect and deduplicate repeated work. Traffic shifts across cells gradually by percentage rather than flipping all at once. The platform also minimizes non-essential dependencies on the critical path by making audit logging and config refreshes asynchronous, gracefully degrading auxiliary features so payment authorization never stalls. True resilience comes from strict blast-radius boundaries and architectural discipline, not just monitoring and retries.&lt;/p&gt;
&lt;h2 id="when-failover-isnt-safe-building-high-availability-postgresql-on-kubernetes"&gt;&lt;a href="https://www.datadoghq.com/blog/engineering/postgresql-ha-kubernetes/" rel="noopener noreferrer"&gt;When failover isn’t safe: Building high-availability PostgreSQL on Kubernetes&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;During a simulated availability-zone failure game day, Datadog discovered that their high-availability PostgreSQL clusters on Kubernetes could not fail over safely. When network latency spiked during the drill, the primary kept accepting writes and replicating asynchronously while standby replicas fell further and further behind. When the primary was terminated, Patroni rightly refused to promote any standby because their replication lag exceeded &lt;code&gt;maximum_lag_on_failover&lt;/code&gt;—preventing silent data loss, but leaving the cluster wedged and unable to accept writes until human operators intervened. The first lesson: nominal high availability means nothing if no replica is actually safe to promote when disaster strikes.&lt;/p&gt;
&lt;p&gt;Datadog moved to a hybrid replication topology: standby instances eligible for failover use synchronous replication, while read-only replicas stay asynchronous to avoid unnecessary latency penalties. In Patroni, they enabled &lt;code&gt;synchronous_mode&lt;/code&gt;, &lt;code&gt;synchronous_mode_strict&lt;/code&gt;, and &lt;code&gt;synchronous_commit = remote_apply&lt;/code&gt;, guaranteeing a transaction is acknowledged only after its WAL records have been replayed and applied on the synchronous standby. Synthetic benchmarks at this highest durability setting showed average write latency increasing by 53% and throughput dropping by 34%, but rolling the change out gradually across datacenters with soak periods confirmed that real-world application impact remained well within acceptable bounds. If all synchronous standbys become unavailable, the primary halts writes rather than silently risking data loss, turning an invisible durability hazard into an explicit error that callers can retry or queue. Balancing durability and performance requires actively monitoring &lt;code&gt;SyncRep&lt;/code&gt; wait events alongside standby health.&lt;/p&gt;
&lt;h2 id="from-christmas-outage-to-1-app-store-ranking-an-aura-frames-postgres-scaling-retrospective"&gt;&lt;a href="https://andyatkinson.com/postgresql-rds-scaling-aws-christmas-day-peak" rel="noopener noreferrer"&gt;From Christmas Outage to #1 App Store Ranking: An Aura Frames Postgres Scaling Retrospective&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;On Christmas Day 2024—the biggest traffic day of the year for digital photo frames—Aura Frames suffered a three-hour PostgreSQL outage when unbounded WAL growth filled their dedicated WAL volume. The culprit was an RDS PostgreSQL 14.1 default that attached replication slots to read replicas: a replication slot forces the primary to retain WAL segments until every replica has consumed them, so when a replica fell behind under holiday load, WAL piled up until the primary ran out of disk space and crashed. The team immediately capped slot retention using &lt;code&gt;max_slot_wal_keep_size&lt;/code&gt; (noting that &lt;code&gt;wal_keep_size&lt;/code&gt; sets a minimum floor, not a ceiling), choosing to invalidate a lagging replica rather than let the primary go down.&lt;/p&gt;
&lt;p&gt;To prepare for Christmas 2025, instead of vertically scaling a single massive primary again, Aura partitioned at the table level (functional sharding): the ten highest-write tables were split out onto seven new primaries, creating eight databases in total. This proved a pragmatic middle ground before needing horizontal row sharding on their largest table. The Ruby on Rails monolith stayed intact, routing queries cleanly via Active Record Multiple Databases, while physical replication moved data over with just 5–10 minutes of cutover downtime. Total provisioned CPU and memory increased roughly 4.7x, and on Christmas Day 2025 the system peaked at 226,000 transactions per second—sustaining over 100,000 TPS for ten straight hours at an average query latency of 25 microseconds. Continuous testing, canary rollouts, and load testing de-risked the data-access changes; temporarily over-provisioning for a predictable annual peak made complete sense, and AWS Blue/Green deployments let them scale the fleet back down afterward to restore cost efficiency.&lt;/p&gt;
&lt;h2 id="gitignore-isnt-the-only-way-to-ignore-files-in-git"&gt;&lt;a href="https://nelson.cloud/.gitignore-isnt-the-only-way-to-ignore-files-in-git/" rel="noopener noreferrer"&gt;.gitignore Isn’t the Only Way To Ignore Files in Git&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Git supports three distinct scopes for ignoring files, each suited to a different job. &lt;code&gt;.gitignore&lt;/code&gt; is committed into the repository, making it the right place for project-wide rules shared by every contributor, like build outputs and dependency folders. &lt;code&gt;.git/info/exclude&lt;/code&gt; lives inside your local &lt;code&gt;.git&lt;/code&gt; directory and is never tracked or pushed, applying only to your local clone—ideal for personal scratch files, local scripts, or project-specific tool configs that don&amp;rsquo;t belong in the team&amp;rsquo;s shared &lt;code&gt;.gitignore&lt;/code&gt;. Finally, &lt;code&gt;~/.config/git/ignore&lt;/code&gt; acts as your user-wide global ignore file across every repository on your machine, perfect for OS- or editor-generated noise like macOS &lt;code&gt;.DS_Store&lt;/code&gt; files or editor swap files.&lt;/p&gt;
&lt;p&gt;You can customize the location of the global ignore file via &lt;code&gt;git config --global core.excludesFile ~/.gitignore_global&lt;/code&gt; and reset it to the default with &lt;code&gt;git config --global --unset core.excludesFile&lt;/code&gt;. And whenever you aren&amp;rsquo;t sure which rule across those three layers is causing a file to be ignored, &lt;code&gt;git check-ignore -v &amp;lt;path&amp;gt;&lt;/code&gt; is the fastest way to debug it: the command prints the exact ignore file, line number, and pattern that matched—or outputs nothing if the file isn&amp;rsquo;t ignored.&lt;/p&gt;
&lt;h2 id="the-only-scalable-delete-in-postgres-is-drop-table"&gt;&lt;a href="https://planetscale.com/blog/the-only-scalable-delete" rel="noopener noreferrer"&gt;The only scalable delete in Postgres is DROP TABLE&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;In PostgreSQL, running a large &lt;code&gt;DELETE&lt;/code&gt; doesn&amp;rsquo;t immediately free disk space—it creates a mountain of follow-up work for the database. Under MVCC, deleted rows remain on disk as dead tuples so concurrent transactions can still see a consistent snapshot; autovacuum eventually marks that space reusable internally, but rarely shrinks the underlying data files on disk (except when completely empty pages sit at the very end of a relation), and indexes and sequential scans still pay the cost of skipping over dead tuples in the meantime. Mass deletes also generate heavy WAL volume that must be replicated downstream, inflating write latency, vacuum debt, and replica lag—while cascading foreign-key deletes can unexpectedly amplify a small delete into gigabytes of churn. &lt;code&gt;VACUUM FULL&lt;/code&gt; reclaims OS disk space, but only by taking an expensive exclusive table lock and rewriting the entire table.&lt;/p&gt;
&lt;p&gt;By contrast, &lt;code&gt;DROP TABLE&lt;/code&gt; and &lt;code&gt;TRUNCATE&lt;/code&gt; acquire an &lt;code&gt;AccessExclusiveLock&lt;/code&gt; but execute in nearly constant time regardless of row count: they unlink the underlying physical files directly without generating per-row dead tuples or vacuum debt. Consequently, schemas that need regular bulk purges should be designed so large deletes become table drops. For time-series or retention-bound data, time-based table partitioning turns recurring deletes into instantly dropping the oldest partition. For a one-off cleanup where most rows are being deleted, copying the surviving rows into a temporary table, running &lt;code&gt;TRUNCATE&lt;/code&gt; on the original, and inserting them back avoids per-row delete WAL (though structural and re-insert WAL still applies); if you cannot afford a long lock, you can dual-write and backfill into a fresh table before swapping names in a brief transaction, taking care to handle foreign keys, sequences, and concurrent writes. When the rows to delete are a small fraction of the table, batched deletes in small transactions remain the practical choice so autovacuum can keep up between batches.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.&lt;/em&gt;&lt;/p&gt;</description></item><item><title>Newsletter #113</title><link>https://miti99.com/en/post/2026/06/24/</link><pubDate>Wed, 24 Jun 2026 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2026/06/24/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #113.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="migrating-from-go-to-rust"&gt;&lt;a href="https://corrode.dev/learn/migration-guides/go-to-rust/" rel="noopener noreferrer"&gt;Migrating from Go to Rust&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Matthias Endler writes a comprehensive guide for backend teams considering migrating services from Go to Rust. The central message isn&amp;rsquo;t &amp;ldquo;Rust is always faster than Go,&amp;rdquo; but that Rust pushes many classes of risk into the type system so the compiler catches them before production: &lt;code&gt;nil&lt;/code&gt; values, error handling, data races, resource lifetimes, and ownership. Go optimizes for team velocity—fast compile times, a strong standard library, and easy-to-use goroutines. Rust trades that for a stricter compiler, no GC, &lt;code&gt;Result&lt;/code&gt;, &lt;code&gt;Option&lt;/code&gt;, traits, and &lt;code&gt;Send&lt;/code&gt;/&lt;code&gt;Sync&lt;/code&gt; checks. Familiar Go habits have to be remapped: &lt;code&gt;if err != nil&lt;/code&gt; becomes &lt;code&gt;Result&lt;/code&gt; with &lt;code&gt;?&lt;/code&gt;, &lt;code&gt;nil&lt;/code&gt; becomes &lt;code&gt;Option&lt;/code&gt;, interfaces become traits, and goroutines become tasks when genuinely needed.&lt;/p&gt;
&lt;p&gt;The most practical section covers migration strategy: don&amp;rsquo;t rewrite everything at once. Start with a hot path, worker, or clearly bounded service, keep the existing API contract, and shift traffic gradually endpoint by endpoint through a gateway using the strangler pattern. While &lt;code&gt;cgo&lt;/code&gt; works, it is often more complex than splitting the Rust portion into a separate service. The author is also candid about the costs: the borrow checker makes the first month rough, compile times are slower, async function coloring is annoying, and parts of the ecosystem are smaller. There is no need to abandon Go entirely: Go remains a great fit for Kubernetes tooling, CLIs, and &amp;ldquo;boring&amp;rdquo; glue services, while Rust is worth the investment for foundational services where reliability, P99 latency, and bugs like nil dereferences or data races carry higher operational costs than the learning and build overhead.&lt;/p&gt;
&lt;h2 id="ai-engineering-for-developers"&gt;&lt;a href="https://www.lucavallin.com/blog/ai-engineering-for-developers" rel="noopener noreferrer"&gt;AI Engineering for Developers&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Luca Cavallin writes a deep-dive guide to AI engineering for developers already comfortable with backends, HTTP, queues, Kubernetes, and production operations. His main argument: when bringing LLMs into a product, the model is not the whole product; what actually creates value is the system around it—prompts, RAG, evals, agents, inference, observability, security, and cost controls. That makes the discipline much closer to backend engineering than model training. Because AI outputs are non-deterministic and can shift across models, prompts, or data, a working demo isn&amp;rsquo;t enough to ship: you need eval datasets, regression checks on every prompt or model change, canary rollouts, and monitoring from day one.&lt;/p&gt;
&lt;p&gt;The article progresses from fundamentals to practical execution: selecting models by task, cost, and latency; exhausting prompt engineering before reaching for RAG; treating RAG primarily as a search problem where hybrid search, rerankers, chunking strategy, and source citations matter more than stuffing more context; fine-tuning only after prompting and RAG hit their ceiling; and optimizing inference through caching, batching, and model routing. Agents are framed as tool-calling loops that require strict schemas, narrow permissions, end-to-end tracing, and human approval gates for risky actions. The production section emphasizes guardrails, dedicated CI/CD pipelines for prompt and model changes, per-feature cost attribution, and treating every agent as a service with its own identity, least-privilege access, and audit logs.&lt;/p&gt;
&lt;h2 id="the-invisible-engineering-behind-lambdas-network"&gt;&lt;a href="https://www.allthingsdistributed.com/2026/04/the-invisible-engineering-behind-lambdas-network.html" rel="noopener noreferrer"&gt;The invisible engineering behind Lambda’s network&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This post on All Things Distributed recounts nearly a decade of networking optimizations behind AWS Lambda—the kind of &amp;ldquo;invisible&amp;rdquo; engineering customers only notice because functions start faster and more reliably. The original bottleneck was cold starts when Lambda connected to a customer VPC: beyond spinning up a Firecracker microVM and initializing the runtime, the system had to establish a private network path into the customer&amp;rsquo;s VPC, with Geneve tunnel creation sitting right on the hot path. Rather than maintaining custom kernel patches, the Lambda team turned to eBPF: pre-creating tunnels with a dummy VNI and overwriting the packet header once the real VNI arrived, cutting tunnel setup latency from ~150ms to 200 microseconds.&lt;/p&gt;
&lt;p&gt;Even more instructive is how they scaled to 4,000 networks per worker. Instead of creating tap devices, veth pairs, namespaces, and routing rules on every function invocation, they pre-allocated everything at worker startup, turning a load-variable cost into a one-time fixed cost. They replaced stateful NAT with stateless eBPF packet rewriting, cutting NAT setup latency 100x; collapsed over 125,000 iptables rules in the root namespace down to 144 static rules by moving per-slot rules into their respective namespaces; and reduced RTNL lock contention by reordering network creation and attaching eBPF programs in batches. At scale, iptables rules, conntrack, or a single kernel lock can all become bottlenecks. The result is a unified topology serving both traditional workloads and SnapStart, boosting snapshot network throughput 20x and getting repackaged for reuse in Aurora DSQL.&lt;/p&gt;
&lt;h2 id="selective-test-execution-at-stripe-fast-ci-for-a-50m-line-ruby-monorepo"&gt;&lt;a href="https://stripe.dev/blog/selective-test-execution-at-stripe-fast-ci-for-a-50m-line-ruby-monorepo" rel="noopener noreferrer"&gt;Selective Test Execution at Stripe: Fast CI for a 50M-line Ruby monorepo&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Aditya Anchuri describes how Stripe keeps CI fast for a ~50-million-line Ruby monorepo containing nearly 100,000 test files and 1.2 million test targets. Running the entire suite sequentially would take roughly four months, while Stripe runs around 50,000 builds per week. Their Selective Test Execution (STE) system solves this by running an average of just ~5% of the test suite per build (with a median under 0.5%), consuming less than 10% of the compute required by a run-everything strategy.&lt;/p&gt;
&lt;p&gt;What makes Stripe&amp;rsquo;s approach clever is that they don&amp;rsquo;t attempt static dependency analysis for Ruby, where metaprogramming, dynamic dispatch, configuration files, and code generation make static analysis either miss dependencies or become overly conservative. Instead, they observe runtime behavior: an internal C++ library injected via &lt;code&gt;LD_PRELOAD&lt;/code&gt; intercepts every file open, associates read files with the currently executing test, and propagates tracking across child processes. The &lt;code&gt;open&lt;/code&gt; path writes only minimal logs, keeping aggregation and indexing off the hot path. Those logs are compiled into a roaring bitmap index mapping changed files to required tests, storing roughly three billion data points while remaining fast to query. File changes are detected using &lt;code&gt;hashdeep&lt;/code&gt; so generated files are covered alongside git-tracked files. STE also relies on reproducible baseline data and practical CI safety nets: always re-running currently failing tests, always running tests that glob the filesystem, handling linters via changed-file lists, and storing metadata in MongoDB with Monotonic Revision IDs to quickly select baselines in commit order.&lt;/p&gt;
&lt;h2 id="being-oncall-taught-me-everything"&gt;&lt;a href="https://yaoyue.org/blog/2026-oncall/" rel="noopener noreferrer"&gt;Being oncall taught me everything&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Yao Yue looks back on 7.5 years of being on call for Twitter&amp;rsquo;s distributed caching layer—the highest-throughput service and the one involved in the most severe site-wide outages. The post doesn&amp;rsquo;t romanticize on-call life, but argues that living alongside production is what shaped the author&amp;rsquo;s mindset as an infrastructure engineer. On-call teaches you that at scale, predictability matters more than being fast in the average case, tail latency matters more than the median, simple and clear architecture is a lifeline during a crisis, and operational excellence (deep observability, consistent configuration, automation readiness, sensible defaults) comes from up-front design rather than frantic patching two weeks before launch.&lt;/p&gt;
&lt;p&gt;Even more memorable are the human lessons. Periodically restarting Memcached to prevent site-wide cascading failures sometimes triggered incidents of its own, teaching the author to pick low-risk windows, give advance notice, and—most importantly—own mistakes immediately, since that is the fastest way to earn back teammates&amp;rsquo; trust. Many outages were resolved only through doggedly narrowing down possibilities and leaning on colleagues across every layer of the stack: site ops, kernel engineers, upstream service owners, managers, and fellow on-call engineers. The conclusion is blunt: engineers don&amp;rsquo;t truly understand software until they see it run and break in production, and people who write software shouldn&amp;rsquo;t consider themselves above deploying, monitoring, or debugging what they build.&lt;/p&gt;
&lt;h2 id="claude-as-your-performance-analysis-partner"&gt;&lt;a href="https://developers.redhat.com/articles/2026/05/29/claude-your-performance-analysis-partner" rel="noopener noreferrer"&gt;Claude as your performance analysis partner&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Archana Ravindar experiments with using Claude as a partner for Go performance analysis across CPU profiles and execution traces, focusing on the Green Tea garbage collector on POWER10 using the &lt;code&gt;sweet&lt;/code&gt; benchmark suite. Claude&amp;rsquo;s value doesn&amp;rsquo;t lie in immediately generating a working optimization patch, but in rapidly digesting massive, dense artifacts like &lt;code&gt;pprof&lt;/code&gt; outputs, assembly dumps, and trace timelines. For example, Claude pinpointed a hot path in &lt;code&gt;runtime.tryDeferToSpanScan&lt;/code&gt;, spotted an atomic &lt;code&gt;Load8&lt;/code&gt; followed by &lt;code&gt;Or8&lt;/code&gt; pattern, and suggested fusing them into a single &lt;code&gt;Or32&lt;/code&gt; instruction. That experiment actually ran slower, and Claude then helped explain why: false sharing and contention across GC workers—a clear illustration that fewer atomic instructions doesn&amp;rsquo;t automatically mean faster code.&lt;/p&gt;
&lt;p&gt;Claude also proved useful for low-level analysis: cross-referencing assembly to see that the compiler wasn&amp;rsquo;t caching &lt;code&gt;q.class.sizeclass&lt;/code&gt;, ranking GC TODO comments by profile heat, and suggesting patterns to look for in traces such as idle processors during GC, mark assist gaps, overly frequent GC cycles, or prolonged &lt;code&gt;gcMarkDone&lt;/code&gt; phases. The author draws a clear distinction: profiles tell you where cost lives, while traces explain why the system is stalling, especially around GC and scheduling. Still, every hypothesis must be benchmarked, because optimizations tied to hardware architecture, cache lines, instruction sets, or register pressure are easy to get wrong without full context; the best use of AI here is narrowing the search space and building an investigation checklist, not replacing benchmarks.&lt;/p&gt;
&lt;h2 id="things-you-didn"&gt;&lt;a href="https://jon.chrt.dev/2026/04/15/things-you-didnt-know-about-indexes.html" rel="noopener noreferrer"&gt;Things you didn&amp;rsquo;t know about indexes&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Jon Charter explains Postgres indexes with an approachable analogy: an index is like the index at the back of a book, letting the database locate rows through a sorted structure instead of scanning the entire table. That speed comes with a fundamental trade-off: faster reads mean slower writes. Data modifications (&lt;code&gt;INSERT&lt;/code&gt;, &lt;code&gt;UPDATE&lt;/code&gt;, &lt;code&gt;DELETE&lt;/code&gt;) usually have to update indexes as well and incur cleanup overhead; indexes also consume disk space, compete for cache memory, and give the query planner more paths to evaluate. Consequently, &amp;ldquo;index everything&amp;rdquo; is rarely a good strategy.&lt;/p&gt;
&lt;p&gt;The most useful section covers common pitfalls that prevent an index from being used. Composite indexes depend on column order, so the order must match real query patterns: &lt;code&gt;(type_1, type_2)&lt;/code&gt; helps queries filtering on &lt;code&gt;type_1&lt;/code&gt; or both columns, but is generally ineffective when filtering only on &lt;code&gt;type_2&lt;/code&gt; (though PostgreSQL 18&amp;rsquo;s skip scan helps in some cases). Wrapping a column in a function like &lt;code&gt;lower(name)&lt;/code&gt; causes a plain index on &lt;code&gt;name&lt;/code&gt; to be ignored because the database needs a structure sorted by the exact expression &lt;code&gt;lower(name)&lt;/code&gt;; implicit type casts cause the same issue. The right way to verify index usage is running &lt;code&gt;EXPLAIN&lt;/code&gt; or &lt;code&gt;EXPLAIN ANALYZE&lt;/code&gt; to confirm what the planner actually chooses rather than guessing. Finally, the post introduces functional indexes, partial indexes for small slices of data like soft-deleted rows or &lt;code&gt;is_legendary = true&lt;/code&gt;, and covering indexes with &lt;code&gt;INCLUDE&lt;/code&gt; to enable &lt;code&gt;Index Only Scan&lt;/code&gt;.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.&lt;/em&gt;&lt;/p&gt;</description></item><item><title>Newsletter #111</title><link>https://miti99.com/en/post/2026/06/11/</link><pubDate>Thu, 11 Jun 2026 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2026/06/11/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #111.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="learnings-from-100k-lines-of-rust-with-ai"&gt;&lt;a href="https://zfhuang99.github.io/rust/claude%20code/codex/contracts/spec-driven%20development/2025/12/01/rust-with-ai.html" rel="noopener noreferrer"&gt;Learnings from 100K Lines of Rust with AI&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Cheng Huang recounts using AI agents like Claude Code, Codex CLI, and Copilot to modernize Azure&amp;rsquo;s Replicated State Library (RSL) — a multi-Paxos consensus engine — in Rust. The resulting codebase reached roughly 130k lines of code and over 1,300 tests. To get AI to correctly implement such a complex distributed system, the author relied heavily on &amp;ldquo;code contracts&amp;rdquo;: having the AI write preconditions, postconditions, and invariants for critical functions, enforcing them as runtime &lt;code&gt;assert&lt;/code&gt;s during testing, and using them to drive both targeted unit tests and property-based tests. That approach caught a subtle Paxos safety violation before the code ever reached production.&lt;/p&gt;
&lt;p&gt;The second technique was lightweight spec-driven development: instead of maintaining rigid, heavyweight requirements and design docs, the author used &lt;code&gt;spec-kit&lt;/code&gt;&amp;rsquo;s &lt;code&gt;/specify&lt;/code&gt; command to generate user stories with acceptance criteria, followed by &lt;code&gt;/clarify&lt;/code&gt; so the AI could critique and refine them; each user story then became a right-sized unit of work for an agent. The third technique was AI-driven performance optimization: over roughly three weeks, the AI added latency instrumentation, ran benchmarks, analyzed bottlenecks, and proposed improvements like reducing allocations, zero-copy parsing, and cutting lock contention — pushing throughput from ~23k to 300k operations per second while Rust preserved memory safety. The author closes with the hope that agents will soon execute entire user stories end to end, automate the contract workflow, and optimize performance across even larger codebases, while humans retain architectural oversight.&lt;/p&gt;
&lt;h2 id="always-be-blaming"&gt;&lt;a href="https://matklad.github.io/2026/05/18/always-be-blaming.html" rel="noopener noreferrer"&gt;Always Be Blaming&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;matklad proposes reading code across multiple &amp;ldquo;dimensions.&amp;rdquo; 2D reading means forming your own mental sketch of how a solution should look first, then comparing it against the actual code to spot surprises; 3D reading means tracing how the code evolved over time, because most production code is Markovian — its shape today depends on its shape yesterday, not just on the problem it solves. The fourth dimension, and the highest goal, is reconstructing the original author&amp;rsquo;s reasoning and train of thought — applying &amp;ldquo;theory of mind&amp;rdquo; to reading code.&lt;/p&gt;
&lt;p&gt;To do that, matklad treats &lt;code&gt;git&lt;/code&gt; history as a first-class investigative tool. On GitHub, press &lt;code&gt;y&lt;/code&gt; to pin the URL to a specific commit SHA, open the blame view, and repeatedly click &amp;ldquo;blame prior to change,&amp;rdquo; opening relevant commits in new tabs and browsing surrounding files at that exact point in time. Locally, he uses custom keybindings: &lt;code&gt;,b l&lt;/code&gt; to check out the commit that last touched the current line, &lt;code&gt;,b p&lt;/code&gt; to step back to the parent commit, &lt;code&gt;,b u&lt;/code&gt; to jump back to the previous inspection point, and &lt;code&gt;,b w&lt;/code&gt; to copy the GitHub link. This &amp;ldquo;in-place blame&amp;rdquo; workflow keeps the LSP, test runner, and search tools like &lt;code&gt;rg&lt;/code&gt; working normally as you travel through time. He also laments that code review comments — an invaluable source of context — remain locked inside third-party hosting databases rather than living inside the Git repository itself.&lt;/p&gt;
&lt;h2 id="scaling-postgresql-to-power-800-million-chatgpt-users"&gt;&lt;a href="https://openai.com/index/scaling-postgresql/" rel="noopener noreferrer"&gt;Scaling PostgreSQL to power 800 million ChatGPT users&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Bohan Zhang shares how OpenAI scaled PostgreSQL to serve ChatGPT and its API across roughly 800 million users. In just one year, PostgreSQL load grew more than 10x, yet the system still relies on a single Azure primary for all writes paired with nearly 50 read replicas spread across multiple regions. The biggest pressure point is write amplification: PostgreSQL&amp;rsquo;s MVCC copies an entire row on every update, amplifying both writes and reads while bloating tables and indexes. To protect the primary, OpenAI offloaded write-heavy, easily partitionable workloads to Azure Cosmos DB, banned adding new tables to the existing PostgreSQL cluster, and held off on sharding because it would require rewriting hundreds of application endpoints.&lt;/p&gt;
&lt;p&gt;Most overload incidents followed the exact same cascade: an upstream trigger — a wave of cache misses, an expensive join, or a write spike from a feature launch — pushed latency up, requests timed out, and client retries amplified the load into a death spiral. To break that chain, OpenAI optimized expensive queries, deployed PgBouncer for connection pooling, added cache locking to prevent cache-miss stampedes, isolated workloads by priority tier, enforced multi-layer rate limiting, capped schema migrations at a 5-second timeout, and throttled backfills. They are also testing cascading replication with Azure to add more read replicas without making the primary ship WAL to all of them directly. As a result, PostgreSQL handles millions of queries per second while keeping client-side p99 latency in the tens of milliseconds and maintaining five-nines availability.&lt;/p&gt;
&lt;h2 id="scaling-managed-agents-decoupling-the-brain-from-the-hands"&gt;&lt;a href="https://www.anthropic.com/engineering/managed-agents" rel="noopener noreferrer"&gt;Scaling Managed Agents: Decoupling the brain from the hands&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Anthropic explains the architecture behind Claude Managed Agents, their hosted service for running long-lived AI agents. The starting observation is that agent harnesses often bake in workarounds for whatever current models struggle with, and those assumptions quickly go stale as models improve. Rather than bundling the model, harness, sandbox, and session state inside a single container, Anthropic decoupled them into three components behind stable interfaces — much like an operating system virtualizes hardware: the session is an append-only event log, the harness is the control loop coordinating Claude and tool calls, and the sandbox is the code execution environment. The pieces communicate through clean primitives like &lt;code&gt;execute()&lt;/code&gt;, &lt;code&gt;emitEvent()&lt;/code&gt;, &lt;code&gt;getSession()&lt;/code&gt;, and &lt;code&gt;getEvents()&lt;/code&gt;.&lt;/p&gt;
&lt;p&gt;This decoupling turns containers from &amp;ldquo;pets&amp;rdquo; that must be kept alive into disposable &amp;ldquo;cattle&amp;rdquo;: a crashed sandbox is just a failed tool call that can be spun up fresh, while the stateless harness can restart at any time and resume from the session event log. A single harness can also coordinate multiple sandboxes, including reaching into a customer&amp;rsquo;s VPC without requiring network peering. On security, credentials never live inside the sandbox where untrusted code runs; tokens are either bound to resources at provisioning time or held in a vault behind an authenticating proxy, making them much harder for prompt injections to steal. Latency improved dramatically too, because inference starts immediately without waiting for a container to boot — cutting time-to-first-token by roughly 60% at p50 and over 90% at p95.&lt;/p&gt;
&lt;h3 id="bonus"&gt;Bonus&lt;/h3&gt;&lt;p&gt;&lt;strong&gt;Images:&lt;/strong&gt;
&lt;img src="https://substackcdn.com/image/fetch/$s_!fap_!,w_1100,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2Fc111a2c6-71bd-45a7-8938-e8bb4f772cbc_1378x1390.png" alt="How does Claude Code keep long sessions from running out of context?"&gt;
&lt;img src="https://substackcdn.com/image/fetch/$s_!VfjL!,w_1100,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F38f19060-e838-440a-971d-bdc1f2d3a167_1280x1546.jpeg" alt="How does a request actually travel through Claude Code?"&gt;
&lt;img src="https://substackcdn.com/image/fetch/$s_!cqW0!,w_1100,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F2e0de719-d0ac-497e-8f9c-b9c841422db8_2484x3002.png" alt="Forward Proxy, Reverse Proxy, and API Gateway Explained"&gt;
&lt;img src="https://substackcdn.com/image/fetch/$s_!eLEi!,w_1100,c_limit,f_auto,q_auto:good,fl_progressive:steep/https%3A%2F%2Fsubstack-post-media.s3.amazonaws.com%2Fpublic%2Fimages%2F52ccd4c2-dc53-4b9a-95b9-d87d6c353f88_2484x3002.png" alt="RAGs vs Agents"&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026.&lt;/em&gt;&lt;/p&gt;</description></item><item><title>Newsletter #110</title><link>https://miti99.com/en/post/2026/06/07/</link><pubDate>Sun, 07 Jun 2026 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2026/06/07/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #110.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="go-podcast--086-just-use-postgres-man"&gt;&lt;a href="https://share.transistor.fm/s/5159e8a9" rel="noopener noreferrer"&gt;go podcast() | 086: Just use postgres, man&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;In this podcast episode, Dominic St-Pierre and Morten Vistisen revolve around a familiar question: should you store telemetry data in ClickHouse or just stick with PostgreSQL? Morten wants to try ClickHouse, while Dominic takes the pragmatic stance: below roughly 10 million rows, PostgreSQL performs nearly as well, and scaling rarely happens overnight, leaving plenty of time to migrate when you actually need to. Morten also recounts having to rewrite a component in his DeployQuad project because the LLM-generated code from the initial prototype didn&amp;rsquo;t behave as expected, underscoring the importance of truly understanding the code you ship.&lt;/p&gt;
&lt;p&gt;Dominic introduces StaticBackend v1.7.0, adding multi-account user support and expanded server-side functions powered by Goja (a JavaScript runtime written in Go), while lamenting how tedious releases are when updating multiple NPM packages compared to Go&amp;rsquo;s simple tag-and-push workflow. Toward the end, both hosts express skepticism about over-relying on AI code generation: it disrupts deep flow states while programming, causes skills to atrophy, and may not even deliver a net productivity gain. They remain convinced that core programming skills are as valuable as ever.&lt;/p&gt;
&lt;h2 id="that-one-time-i-used-go-panics-for-flow-control"&gt;&lt;a href="https://noncrab.net/posts/panic-as-flow-control/" rel="noopener noreferrer"&gt;That one time I used Go panics for flow control&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;The author recounts an overload incident on an in-memory Go data service: when query volume exceeded capacity, requests that normally took 1–2 seconds stretched to a full minute, and upstream retries only piled on more load. The root problem was that Go&amp;rsquo;s standard library sorting functions neither accept a &lt;code&gt;context&lt;/code&gt; nor return an error, so the service kept burning CPU cycles sorting results for queries whose callers had already timed out and walked away. That library design is normally reasonable, since sorting usually happens in batch jobs or over slices small enough that execution time is negligible.&lt;/p&gt;
&lt;p&gt;Rather than rewriting the entire sorting layer, the team used a &lt;code&gt;panic&lt;/code&gt;/&lt;code&gt;recover&lt;/code&gt; pair as an escape hatch: the comparison function checks whether the &lt;code&gt;context&lt;/code&gt; has been canceled and, if so, calls &lt;code&gt;panic&lt;/code&gt; with a dedicated sentinel value. A &lt;code&gt;defer&lt;/code&gt; wrapper around the sort catches that specific sentinel &lt;code&gt;panic&lt;/code&gt; and converts it into a normal error while re-panicking on anything else so real bugs aren&amp;rsquo;t swallowed. That let the system abort useless sorts immediately. The author readily admits this is &amp;ldquo;an ugly solution to an ugly problem,&amp;rdquo; strictly for narrow, tightly controlled situations.&lt;/p&gt;
&lt;h2 id="the-last-six-months-in-llms-in-five-minutes"&gt;&lt;a href="https://simonwillison.net/2026/May/19/5-minute-llms/" rel="noopener noreferrer"&gt;The last six months in LLMs in five minutes&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This post is the annotated version of Simon Willison&amp;rsquo;s lightning talk at PyCon US 2026, recapping developments in large language models from November 2025 through May 2026. The standout trend is that coding agents crossed a decisive quality threshold: thanks to OpenAI and Anthropic applying reinforcement learning with verifiable rewards throughout 2025, agents went from &amp;ldquo;often works&amp;rdquo; to &amp;ldquo;almost always works&amp;rdquo; and became everyday workhorses. In November alone, the title of &amp;ldquo;best model&amp;rdquo; changed hands five times between Anthropic, OpenAI, and Google, while Willison continued using his signature benchmark of asking models to generate an SVG of a pelican riding a bicycle.&lt;/p&gt;
&lt;p&gt;The second major trend is local open-weight models blowing past expectations, including Google&amp;rsquo;s Gemma 4 family, China&amp;rsquo;s GLM-5.1, and Qwen3.6-35B-A3B, which weighs in at roughly 21GB and runs comfortably on a standard laptop. The talk also highlights OpenClaw, a personal AI assistant project that went so viral that &amp;ldquo;Claws&amp;rdquo; became the generic term for the entire category of personal agents and Mac Minis sold out as people bought them to run Claws at home.&lt;/p&gt;
&lt;h2 id="prompts-are-technical-debt-too"&gt;&lt;a href="https://www.seangoedecke.com/prompts-are-technical-debt-too/" rel="noopener noreferrer"&gt;Prompts are technical debt too&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Sean Goedecke argues that prompts — including instruction files like &lt;code&gt;AGENTS.md&lt;/code&gt; and &lt;code&gt;CLAUDE.md&lt;/code&gt; — are a form of technical debt, and in some ways worse than code debt. When code doesn&amp;rsquo;t change, its behavior stays largely the same, and when it breaks it usually fails loudly; prompts, by contrast, silently degrade every time the underlying model is upgraded. A prompt painstakingly tuned to work around an older model&amp;rsquo;s quirks can become useless or actively counterproductive on a newer model without anyone noticing. According to Goedecke, an agent harness with elaborate prompts wrapped around an older model will almost always lose to a minimal harness running a newer model.&lt;/p&gt;
&lt;p&gt;From there, he offers concrete advice: stick to mainstream AI coding tools like Claude Code, Cursor, or Copilot with their default configurations, since their engineering teams continuously re-tune the system prompts for you; avoid installing extra MCP servers or skills unless strictly necessary; if you must write an &lt;code&gt;AGENTS.md&lt;/code&gt;, stick to core factual details about the project rather than trying to micromanage model behavior, and never check in model-generated instructions without reviewing them carefully. Finally, regularly prune custom prompts to keep maintenance overhead low.&lt;/p&gt;
&lt;h2 id="we-see-something-that-works-and-then-we-understand-it"&gt;&lt;a href="https://lemire.me/blog/2025/12/04/we-see-something-that-works-and-then-we-understand-it/" rel="noopener noreferrer"&gt;We see something that works, and then we understand it&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Daniel Lemire pushes back against the &amp;ldquo;linear model of innovation&amp;rdquo; — the belief that theoretical understanding must come first and practical progress follows afterward. Borrowing Kevin Kelly&amp;rsquo;s term &amp;ldquo;thinkism,&amp;rdquo; he critiques the illusion that thinking hard enough in the abstract can solve any problem. That mindset dominates schools and bureaucracies, where rules are fixed in advance: students learn a concept first and then solve textbook problems designed to require only that exact concept. History usually works the other way around: pendulum clocks were keeping time in 1656, well before Newton and Hooke formulated the laws of force and motion that explained why they worked.&lt;/p&gt;
&lt;p&gt;In Lemire&amp;rsquo;s view, real R&amp;amp;D always happens with incomplete understanding; people entering industry often learn more in a single year on the job than across their entire university degree because real-world engineering demands continuous experimentation and observation. He draws two lessons: breakthroughs require prioritizing hands-on experimentation over abstract theorizing, and we shouldn&amp;rsquo;t expect AI to solve every complex problem purely from existing knowledge, because the physical world is always richer and messier than what we already know.&lt;/p&gt;
&lt;h2 id="how-vercel-cut-build-wait-times-from-90-seconds-to-5"&gt;&lt;a href="https://blog.bytebytego.com/p/how-vercel-cut-build-wait-times-from" rel="noopener noreferrer"&gt;How Vercel Cut Build Wait Times From 90 Seconds To 5&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This article examines how Vercel cut build environment provisioning wait times from 90 seconds down to 5 seconds — an 18x speedup — using an internal platform called Hive. The core challenge is that Vercel runs untrusted build code from thousands of customers on shared hardware, meaning every build must be treated as potentially hostile. Standard containers aren&amp;rsquo;t isolated enough because they share the host Linux kernel, where a single kernel vulnerability could expose another customer&amp;rsquo;s workload. Instead of Kubernetes, Vercel built Hive on AWS Firecracker microVMs: each build runs inside a &amp;ldquo;cell&amp;rdquo; — a tiny virtual machine with its own dedicated kernel that boots in roughly 125 milliseconds. The architecture spans regional clusters where physical &amp;ldquo;boxes&amp;rdquo; host multiple cells, coordinated by a control plane.&lt;/p&gt;
&lt;p&gt;The speedup comes from three compounding optimizations: container image caching and disk snapshots cut cold starts from ~50 seconds down to 5 seconds; a warm pool of pre-booted cells waiting for work means most builds experience near-zero queue wait; and Firecracker&amp;rsquo;s millisecond boot times make maintaining and replenishing that warm pool economical. The trade-off is that keeping warm cells idle consumes resources and every cell must be destroyed after a single build for security, but owning the underlying infrastructure has unlocked new product capabilities for Vercel.&lt;/p&gt;
&lt;h2 id="the-anatomy-of-an-ai-native-org"&gt;&lt;a href="https://ajeygore.in/content/the-anatomy-of-an-ai-native-org" rel="noopener noreferrer"&gt;The Anatomy of an AI-Native Org&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Ajey Gore argues that AI is reshaping organizational structure by eliminating the &amp;ldquo;translation layer.&amp;rdquo; For thirty years, software companies have operated across three tiers: leadership decides the &amp;ldquo;Why,&amp;rdquo; product defines the &amp;ldquo;What,&amp;rdquo; and engineering translates those decisions into code — the &amp;ldquo;How.&amp;rdquo; Much of the day-to-day work in that model consists of translating between well-defined inputs and outputs. According to Gore, AI doesn&amp;rsquo;t target a specific job title so much as that entire category of translation work — turning requirements into code or specs into UI components — making it orders of magnitude cheaper, while the strategic work at both ends remains just as hard as ever.&lt;/p&gt;
&lt;p&gt;The shape of an &amp;ldquo;AI-Native&amp;rdquo; organization shifts accordingly: the &amp;ldquo;Why&amp;rdquo; layer stays small; the &amp;ldquo;What&amp;rdquo; layer expands because judgment and taste become critical when directing AI; the human &amp;ldquo;How&amp;rdquo; layer shrinks sharply in headcount but rises in difficulty, focusing on architecture, security, and verification mechanisms that make AI output trustworthy; and AI agents handle routine translation under human supervision. Managers can no longer just coordinate — they must contribute directly — while engineers should move toward problem definition rather than competing with AI on raw code translation.&lt;/p&gt;
&lt;h2 id="how-to-stay-resilient-in-a-difficult-job"&gt;&lt;a href="https://andiroberts.com/executive-coaching/how-to-stay-resilient-in-a-difficult-job" rel="noopener noreferrer"&gt;How to Stay Resilient in a Difficult Job&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This article outlines 7 practical, science-backed strategies for maintaining mental resilience when stuck in a difficult job — whether dealing with a poor manager, irregular shifts, or chronic stress. The author stresses that resilience here doesn&amp;rsquo;t mean forcing toxic positivity or pretending the job is fun; it means conserving energy, keeping an objective perspective, and maintaining a sense of agency so you can do good work, protect your health, and stay positioned for future moves.&lt;/p&gt;
&lt;p&gt;The strategies include: grounding yourself mentally at the start of the day and stabilizing your physiology with sunlight, hydration, and movement before work; focusing strictly on what lies within your control and upholding your own professional standards even when surrounding systems are messy; setting emotional boundaries so you don&amp;rsquo;t carry work stress home; finding at least one trusted colleague to confide in; and consistently taking small steps toward landing a better role.&lt;/p&gt;
&lt;h2 id="what"&gt;&lt;a href="https://brooker.co.za/blog/2026/05/18/whats-easy-whats-hard.html" rel="noopener noreferrer"&gt;What&amp;rsquo;s Easy Now? What&amp;rsquo;s Hard Now?&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Marc Brooker proposes the &amp;ldquo;feedback loop hypothesis&amp;rdquo;: over the long run, AI coding agents will find tasks with clear, measurable feedback loops &amp;ldquo;easy,&amp;rdquo; while tasks lacking effective automated feedback will remain &amp;ldquo;hard&amp;rdquo; — and this divide will be the primary determinant of agent capability. Opening with an analogy from electrical engineering, he notes that strong negative feedback can turn simple, nonlinear, even flawed components into remarkably accurate amplifiers.&lt;/p&gt;
&lt;p&gt;From that premise, he flips the conventional wisdom that web development is easy and systems programming is hard. A database storage engine like RocksDB has a relatively compact specification — an API plus safety and liveness properties — that can be verified automatically. Web development, by contrast, lacks an equivalent automated oracle and relies on slow, inconsistent human judgment. Current practice already reflects this: agents excel at performance optimization when given a solid benchmark suite, struggle with subjective architectural trade-offs, and get guided toward correct code in languages like Rust thanks to strict compiler errors. Consequently, formal specification languages, compile-time checks, and rigorous testing frameworks will only grow in importance.&lt;/p&gt;
&lt;h2 id="making-user-sequence-data-more-cost-efficient-faster-and-easier-to-use"&gt;&lt;a href="https://medium.com/pinterest-engineering/making-user-sequence-data-more-cost-efficient-faster-and-easier-to-use-2a56a928cae1" rel="noopener noreferrer"&gt;Making User-Sequence Data More Cost-Efficient, Faster, and Easier to Use&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Pinterest&amp;rsquo;s engineering team shares how they redesigned their user-sequence data platform — the ordered history of a user&amp;rsquo;s recent actions enriched with signals like embeddings, device context, and country. This data powers ML ranking, retrieval, and recommendation models across the Home feed, Related Pins, and search results, serving training pipelines, offline analytics, and real-time inference alike. Because it must be simultaneously fresh, complete, consistent between training and serving, and schema-stable, it is often one of the most expensive and brittle parts of a data stack.&lt;/p&gt;
&lt;p&gt;The core principle of the redesign is &amp;ldquo;One definition, multiple runtimes&amp;rdquo;: each signal is defined once and shared across real-time indexing, batch backfills for historical data, and online serving via a unified execution engine supporting both streaming and batch modes. That eliminates training-serving skew by construction. Combined with a Lambda architecture to balance freshness and completeness, plus columnar storage and time-based partitioning, Pinterest significantly cut storage costs and network bandwidth while turning the addition of new event types or signals into a mostly configuration-driven change.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026.&lt;/em&gt;&lt;/p&gt;</description></item><item><title>Newsletter #9</title><link>https://miti99.com/en/post/2025/04/01/</link><pubDate>Tue, 01 Apr 2025 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2025/04/01/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #9.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="to-avoid-being-replaced-by-llms-do-what-they-can"&gt;&lt;a href="https://www.seangoedecke.com/what-llms-cant-do/" rel="noopener noreferrer"&gt;To avoid being replaced by LLMs, do what they can&amp;rsquo;t&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Sean Goedecke offers career advice for software engineers facing a future where large language models (LLMs) keep getting better at writing code, broken down across three time horizons. In the short term, lean heavily into AI tools, understand how language models work under the hood so you can get staffed on AI projects, and push for promotion—since junior roles are the most likely to be automated first. The author expects this phase to last at least five years, because large enterprises simply don&amp;rsquo;t move that fast.&lt;/p&gt;
&lt;p&gt;In the medium term, he advises diving into sprawling legacy codebases inside large organizations. LLMs currently shine at competitive programming, where problems are self-contained, well-specified, easy to verify, and only a few hundred lines long; building features inside a decade-old enterprise system is the exact opposite: ambiguous requirements, messy verification, and millions of lines of code. Conquering that will require solving massive-context reasoning and building better evals against proprietary data locked inside company firewalls. Over the long run, the one thing LLMs cannot replace is accountability: engineers are trusted because they have skin in the game when things go wrong, whereas an LLM has nothing to lose. Companies will always need at least one human engineer in the loop to supervise the models and turn their plans into commitments the business can rely on.&lt;/p&gt;
&lt;h2 id="how-instagram-scaled-its-infrastructure-to-support-a-billion-users"&gt;&lt;a href="https://blog.bytebytego.com/p/how-instagram-scaled-its-infrastructure" rel="noopener noreferrer"&gt;How Instagram Scaled Its Infrastructure To Support a Billion Users&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This ByteByteGo article traces how Instagram scaled its infrastructure from a 13-person photo-sharing app running on AWS to a platform serving over a billion users. In the early days, the team had to manually spin up extra servers before every weekend, the database was constantly overwhelmed, and neither monitoring nor load balancing was automated. After being acquired by Facebook, Instagram migrated into Facebook&amp;rsquo;s data centers and scaled along three axes: scaling horizontally (adding servers and data centers), squeezing more performance out of each machine (batching queries, aggressive caching, and rewriting CPU-heavy Python functions in C++), and scaling the engineering organization through continuous deployment exceeding 40 production pushes a day.&lt;/p&gt;
&lt;p&gt;Architecturally, Django handles web requests, RabbitMQ and Celery process asynchronous background jobs like push notifications, PostgreSQL stores structured relational data requiring strong consistency, Cassandra handles distributed workloads like feeds and activity logs, Memcached serves as the caching tier, and Facebook&amp;rsquo;s Haystack stores photos and videos. One notable technique is the &lt;strong&gt;Memcache Lease&lt;/strong&gt; mechanism to prevent thundering herds: when a cached item expires, only the first app server granted a &amp;ldquo;lease&amp;rdquo; token is allowed to query the database, while other servers wait briefly or serve slightly stale data. Their deployment pipeline combines code review, automated tests, canary rollouts on a small subset of machines, and per-function CPU profiling in production.&lt;/p&gt;
&lt;h2 id="the-state-of-scala--clojure-surveys"&gt;&lt;a href="https://www.jvm-weekly.com/p/the-state-of-scala-and-clojure-surveys" rel="noopener noreferrer"&gt;The State of Scala &amp;amp; Clojure Surveys&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Artur Skowronski (JVM Weekly) examines two recent community surveys covering Scala and Clojure. VirtusLab&amp;rsquo;s late-2024 Scala Survey of 232 respondents found that while 93.1% like or love Scala, the average project age is now seven years—meaning most Scala work happens on legacy systems. The most popular ecosystems are Typelevel (41.8%), Akka (35.3%), and ZIO (23.3%), with 53% of projects mixing multiple stacks, and only 22.4% of commercial codebases have migrated fully to Scala 3. The top three pain points are slow compile times, ecosystem fragmentation complicating the Scala 3 upgrade, and a shortage of resources to maintain older libraries. Hiring is another sore spot: 56.5% don&amp;rsquo;t know where to find Scala developers and 23.2% of companies are considering migrating to Kotlin, Rust, or Go—even though 88.4% of respondents would still pick Scala for a greenfield project.&lt;/p&gt;
&lt;p&gt;On the Clojure side, the 2024 State of Clojure survey shows 73% of respondents using the language professionally, primarily for web development and enterprise backends; 58% have upgraded to Clojure 1.12 and 54% run on Java 21 LTS. A highlight this year is the growing ecosystem of Clojure dialects: Babashka is widely loved for fast scripting without JVM startup overhead, and ClojureDart brings Clojure to the Dart/Flutter ecosystem for web and mobile, alongside experimental compilers like Squint, Jank, and Cherry. For newcomers, the steepest learning curve remains grokking macros and homoiconicity.&lt;/p&gt;
&lt;h2 id="linkedin-integrates-protocol-buffers-with-restli-for-improved-microservices-performance"&gt;&lt;a href="https://www.linkedin.com/blog/engineering/infrastructure/linkedin-integrates-protocol-buffers-with-rest-li-for-improved-m" rel="noopener noreferrer"&gt;LinkedIn Integrates Protocol Buffers With Rest.li for Improved Microservices Performance&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Karthik Ramgopal and Aman Gupta share how LinkedIn replaced JSON with Protocol Buffers (Protobuf) as the default serialization format for Rest.li, their open-source REST framework powering over 50,000 production API endpoints. While JSON is human-readable and universally supported, its text-based representation produces bulky payloads that consume extra bandwidth and make serialization and deserialization a major CPU bottleneck. After benchmarking alternatives including FlatBuffers, Cap&amp;rsquo;n Proto, MessagePack, CBOR, and Kryo, the team chose Protobuf for the best overall balance of compact payload size, serialization speed, and broad language support across LinkedIn&amp;rsquo;s stack.&lt;/p&gt;
&lt;p&gt;The catch was that Rest.li&amp;rsquo;s PDL schema language lacks the explicit numeric field tags that Protobuf requires. The team solved this with a bidirectional &lt;strong&gt;symbol table&lt;/strong&gt; mapping field names to integer IDs: services generate their symbol table at startup and clients fetch and cache it dynamically, while for web and mobile clients the table is generated at build time in an append-only manner. Rollout was controlled incrementally via client-side configuration. Across production, average throughput improved by 6.25% for responses and 1.77% for requests, latency dropped by up to 60% for services with large payloads, and no service saw a meaningful regression. LinkedIn&amp;rsquo;s next step is gradually migrating from Rest.li to gRPC.&lt;/p&gt;
&lt;h2 id="story-redis-and-its-creator-antirez"&gt;&lt;a href="https://blog.brachiosoft.com/en/posts/redis/" rel="noopener noreferrer"&gt;Story: Redis and its creator antirez&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This Brachiosoft profile traces the journey of Salvatore Sanfilippo (antirez), the creator of Redis. Growing up in Sicily, antirez got his start on a TI-99/4A home computer at age six, learning BASIC alongside his father. After setting programming aside during his teenage years, he returned to it around age 18–19, taught himself C, and discovered a security flaw in the &lt;code&gt;ping&lt;/code&gt; utility. His post on the Bugtraq mailing list landed him a job offer at Milan security firm SECLAB, where he invented the Idle Scan port-scanning technique and wrote the &lt;code&gt;hping&lt;/code&gt; tool before moving back to Sicily six months later.&lt;/p&gt;
&lt;p&gt;Redis grew out of LLOOGG, a real-time web analytics tool antirez launched in 2007. Because MySQL couldn&amp;rsquo;t keep up with the disk I/O of real-time per-visit list updates, he prototyped an in-memory list database in Tcl called LMDB, rewrote it in C, and released Redis in 2009. The initial reception on Hacker News was lukewarm, but Redis quickly won pivotal early adopters: GitHub used it to build the Resque background job queue, Instagram built its early storage layer almost entirely on Redis, and Twitter repeatedly flew antirez out to their headquarters to help architect their timeline cache. VMware later hired him to work on Redis full-time, followed by stints at Pivotal and Redis Labs. In 2020, after more than a decade at the helm, antirez stepped down as maintainer, worn out by the constant tension between keeping the codebase crafted like a work of art and meeting the practical demands of a massive global user base.&lt;/p&gt;
&lt;h2 id="tracing-structured-logging-but-better-in-every-way"&gt;&lt;a href="https://andydote.co.uk/2023/09/19/tracing-is-better/" rel="noopener noreferrer"&gt;Tracing: structured logging, but better in every way&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Andy Dote explains why he believes distributed tracing beats structured logging in every way, and shows how to migrate incrementally from logs to traces. Log levels (debug, info, warning…), he argues, are largely meaningless; free-text messages force slow full-text searches; log output gets interleaved with stdout; individual log lines lack causal relationships; and timing measurements have to be wired up manually and inconsistently. Traces solve all of these: every span automatically records a start time, end time, and duration, maintains parent-child hierarchy, propagates across service boundaries for distributed tracing, and makes it trivial to plot latency distributions or answer negative queries like &amp;ldquo;which requests did &lt;em&gt;not&lt;/em&gt; find the user in the cache?&amp;rdquo;&lt;/p&gt;
&lt;p&gt;The hands-on walkthrough takes a small Go service using OpenTelemetry through three steps: adding a tracer and opening a span in each function, wrapping errors so status codes and error events are recorded on the active span, and replacing log lines with filterable span attributes—turning &lt;code&gt;&amp;quot;found user in cache&amp;quot;&lt;/code&gt; into &lt;code&gt;user_in_cache: true&lt;/code&gt;. The resulting code is only a few lines longer while capturing far richer telemetry. For loops, the author recommends extracting the loop body into a helper function with its own child span or recording aggregate counts after the loop to avoid overwriting span attributes, adopting Observability-Driven Development, and viewing traces in tools like Honeycomb or Lightstep.&lt;/p&gt;
&lt;h2 id="software-engineering-job-openings-hit-five-year-low"&gt;&lt;a href="https://newsletter.pragmaticengineer.com/p/software-engineering-job-openings" rel="noopener noreferrer"&gt;Software engineering job openings hit five-year low?&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Gergely Orosz analyzes Indeed job-posting data showing that US software engineering openings currently sit at just 65% of their January 2020 baseline—down 3.5x from their mid-2022 peak and 8% lower than a year ago. Similar drops appear across Canada, the UK, France, and Germany, with Australia the lone market still above 2020 levels. Even more striking, while total job postings across all sectors on Indeed are up 10% compared to 2020, software development postings have fallen 34%—a much steeper drop than marketing (-19%), sales (-8%), or banking and finance (-7%).&lt;/p&gt;
&lt;p&gt;The author points to several overlapping drivers: rising interest rates ending the zero-interest-rate policy (ZIRP) era and drying up venture capital; companies still digesting the overhiring binge of 2021–2022; enterprises taking a &amp;ldquo;wait and see&amp;rdquo; approach to headcount while evaluating GenAI productivity gains (such as Salesforce freezing engineering hiring after reporting a 30% AI productivity boost); and lean teams proving how much a small group can ship—like Linear with 25 engineers or Bluesky with 13. He notes that Indeed skews toward traditional enterprises and may undercount startups and Big Tech. Looking ahead, he sketches a few scenarios: smaller, higher-leverage teams spawning a wave of new startups; platforms that turn plain-English specs into working apps; or a surge of AI-generated apps built by non-programmers eventually creating &lt;em&gt;more&lt;/em&gt; demand for professional engineers to scale and maintain them.&lt;/p&gt;
&lt;h2 id="we-switched-from-java-to-go-and-don"&gt;&lt;a href="https://glasskube.dev/blog/from-java-to-go/" rel="noopener noreferrer"&gt;We switched from Java to Go and don&amp;rsquo;t regret it&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Philip Miglinci from Glasskube shares why his team switched from Java/Kotlin to Go. Despite years of experience with Java and Spring, while building Kubernetes tooling they noticed that other Kubernetes operators—mostly written in Go—used next to no resources, whereas their Kotlin-based operator and its supporting stack consumed over 2 GB of RAM even when completely idle. That prompted them to rewrite their open-source package manager in Go, and later choose Go for Distr, their open-source software distribution platform and first full Go web application.&lt;/p&gt;
&lt;p&gt;The post compares the two ecosystems across several dimensions. Java relies on JIT compilation and supports incremental builds and hot reloading, but Gradle and Maven are memory-hungry; Go compiles AOT into a single static binary, with a lightweight Spring Boot app taking roughly 8.2 seconds to start compared to under 100 milliseconds for a Go server. Java favors batteries-included frameworks like Spring or Quarkus, whereas Go prefers small, composable libraries with less &amp;ldquo;black magic,&amp;rdquo; eschewing Spring-style annotation-driven dependency injection in favor of explicit constructor or function arguments (keeping &lt;code&gt;context.Context&lt;/code&gt; strictly for deadlines, cancellation signals, and request-scoped values). Debugging and IDE support are roughly on par—though Java stack traces are easier to follow—and GoReleaser shines for release automation. The author concludes that Go is a natural fit for cloud-native and Kubernetes tooling, even if Java remains a strong choice for other domains.&lt;/p&gt;
&lt;h2 id="secure-java-applications-a-deep-look-into-3-different-issues"&gt;&lt;a href="https://developers.redhat.com/articles/2024/11/18/secure-java-applications-deep-look-3-different-issues" rel="noopener noreferrer"&gt;Secure Java applications: A deep look into 3 different issues&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Martin Balao Alonso (Red Hat) examines three classes of vulnerabilities that can compromise data confidentiality, integrity, or availability in Java applications. The first is &lt;strong&gt;integer overflow&lt;/strong&gt;: &lt;code&gt;byte&lt;/code&gt;, &lt;code&gt;short&lt;/code&gt;, &lt;code&gt;int&lt;/code&gt;, and &lt;code&gt;long&lt;/code&gt; use signed two&amp;rsquo;s-complement representation. While &lt;code&gt;javac&lt;/code&gt; catches certain out-of-range literal assignments without an explicit cast, arithmetic operations silently wrap around according to the language specification. Using &lt;code&gt;Math::xxxExact&lt;/code&gt; methods or &lt;code&gt;BigInteger&lt;/code&gt; helps detect overflow immediately or compute with arbitrary-precision integers.&lt;/p&gt;
&lt;p&gt;The second issue is &lt;strong&gt;untrusted input driving resource allocation&lt;/strong&gt;: if a server pre-allocates a buffer based directly on a length field declared in an incoming message header, an attacker can trigger an &lt;code&gt;OutOfMemoryError&lt;/code&gt; at almost zero cost; defenses include allocating memory incrementally as bytes arrive and capping concurrent connections per client. The third is &lt;strong&gt;timing attacks&lt;/strong&gt; involving the JVM interpreter and JIT compiler: security-sensitive code must run in constant time regardless of secret values, yet even a simple byte-by-byte array comparison takes different execution paths depending on where a mismatch occurs—and an attacker can even manipulate the C2 JIT compiler&amp;rsquo;s branch profiling statistics to trigger a deoptimization trap that leaks secret information through timing differences. The author recommends constant-time bitwise operations like XOR for secret comparisons and carefully inspecting the native machine code emitted by the JIT.&lt;/p&gt;
&lt;h2 id="java-devs-rejoice-this-open-source-secret-weapon-will-cut-your-coding-time-in-half"&gt;&lt;a href="https://www.indiehackers.com/post/java-devs-rejoice-this-open-source-secret-weapon-will-cut-your-coding-time-in-half-a7869a56de" rel="noopener noreferrer"&gt;Java Devs Rejoice! This Open-Source Secret Weapon Will Cut Your Coding Time in Half&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Brett Hoffman introduces &lt;strong&gt;esProc SPL&lt;/strong&gt;, an open-source data processing language written entirely in Java. The post opens with a familiar dilemma: writing structured data manipulation logic in plain Java is verbose (even grouping and aggregating by two fields takes dozens of lines of boilerplate), while SQL is concise but tightly coupled to a relational database, making it hard to scale independently or run outside the DB. Java 8 Streams, Kotlin, and Scala tried to bridge the gap but still fall short of SQL&amp;rsquo;s expressiveness—and because Java is compiled, every logic change requires a full rebuild and redeploy.&lt;/p&gt;
&lt;p&gt;According to the author, SPL solves both problems: its syntax is as concise as SQL—and often shorter for complex order-dependent calculations, like computing the longest streak of consecutive price increases for a stock in a single line—while running independently of any database; it comes with a dedicated IDE supporting step-by-step debugging and visual data inspection; it handles both in-memory and out-of-core datasets larger than RAM, enabling parallel execution simply by adding an &lt;code&gt;@m&lt;/code&gt; flag. SPL embeds directly into Java applications via a standard JDBC driver, is lightweight enough to run on Android or act as a compute engine inside microservices, joins data seamlessly across heterogeneous sources (databases, REST APIs, CSVs, JSON), and—because it is interpreted—supports hot-swapping business logic on the fly without restarting the service.&lt;/p&gt;
&lt;h2 id="bonus-interesting-videos-from-inside-java"&gt;Bonus: Interesting videos from &lt;a href="https://inside.java/" rel="noopener noreferrer"&gt;Inside Java&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;&lt;a href="https://inside.java/2025/02/15/devoxxbelgium-gc-progress/" rel="noopener noreferrer"&gt;Garbage Collection in Java - The progress since JDK 8&lt;/a&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.&lt;/em&gt;&lt;/p&gt;</description></item><item><title>Newsletter #8</title><link>https://miti99.com/en/post/2025/03/24/</link><pubDate>Mon, 24 Mar 2025 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2025/03/24/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #8.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="let"&gt;&lt;a href="https://info.michael-simons.eu/2025/02/05/lets-deadlock-all-the-things/" rel="noopener noreferrer"&gt;Let&amp;rsquo;s deadlock all the things&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Michael J. Simons recounts a &amp;ldquo;rookie mistake&amp;rdquo; he made while migrating code from legacy &lt;code&gt;HttpURLConnection&lt;/code&gt; to Java&amp;rsquo;s modern &lt;code&gt;HttpClient&lt;/code&gt;. Because &lt;code&gt;HttpClient&lt;/code&gt; does not directly expose an &lt;code&gt;OutputStream&lt;/code&gt; for streaming data into a request body (for instance, when using Jackson to serialize a large JSON payload), he connected a &lt;code&gt;PipedInputStream&lt;/code&gt; and a &lt;code&gt;PipedOutputStream&lt;/code&gt; and ended up performing both the write and the read on the same thread—even though the JavaDoc explicitly warns that doing so can deadlock. Initial unit tests passed because the test payloads were tiny, but as soon as a payload exceeded the default 1,024-byte pipe buffer in production, the thread blocked forever; repeated enough times, it starves the entire thread pool.&lt;/p&gt;
&lt;p&gt;What could have prevented the bug, the author reflects, was a little less overconfidence, a second pair of eyes in code review, and thorough boundary testing. AI assistants might help generate larger test payloads, he notes, but rarely catch subtle API misuses—and even IntelliJ didn&amp;rsquo;t flag a warning. The fix was to create the two streams independently, connect them outside both the reader and writer lambdas, run the write operation on a dedicated virtual thread, and close the output stream inside that thread. It&amp;rsquo;s a great reminder to read the JavaDoc carefully and understand the threading model of the APIs you use.&lt;/p&gt;
&lt;h2 id="engineers-who-won"&gt;&lt;a href="https://www.seangoedecke.com/taking-a-position/" rel="noopener noreferrer"&gt;Engineers who won&amp;rsquo;t commit&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Sean Goedecke argues that sitting on the fence in technical discussions is only acceptable when you are a junior engineer. Once you are the person in the room with the most context, skill, or influence, you have a responsibility to take a position—even when you are only 55–60% confident. Staying silent forces less-informed teammates to guess, creates a vacuum where the loudest, least-competent voice can steer the group into a bad idea, and ultimately punts hard technical decisions onto engineering managers. The author bluntly calls this refusal to commit cowardice, rooted in a fear of being wrong that he himself had to overcome earlier in his career.&lt;/p&gt;
&lt;p&gt;Managers, he notes, are generally quite forgiving when you make a well-reasoned call that turns out to be wrong, as long as you aren&amp;rsquo;t wrong all the time. Refusing to give time estimates is the same failure mode: managers need a rough ballpark to plan around, and answering &amp;ldquo;it depends&amp;rdquo; just forces them to make up a number themselves. The one exception is genuinely toxic workplaces where engineers are punished for missed estimates—there, keeping your head down is understandable. He also adds that when brainstorming with peers who share your level of context, staying open-minded and exploratory is completely fine.&lt;/p&gt;
&lt;h2 id="patterns-for-building-realtime-features"&gt;&lt;a href="https://zknill.io/posts/patterns-for-building-realtime/" rel="noopener noreferrer"&gt;Patterns for building realtime features&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Zak Knill surveys four common architectural patterns for building real-time features where a server pushes one user&amp;rsquo;s changes out to other connected clients. The &lt;strong&gt;poke/pull&lt;/strong&gt; pattern is the easiest to bolt onto an existing system: the server sends a lightweight &amp;ldquo;poke&amp;rdquo; notification prompting clients to refetch the latest state via standard HTTP APIs, though this can trigger a thundering herd of simultaneous requests that must be mitigated with caching or client-side jitter. &lt;strong&gt;Push state&lt;/strong&gt; sends the entire updated state payload directly, preventing client drift, but scales poorly as state objects grow and makes it hard for clients to tell exactly what changed. &lt;strong&gt;Push ops&lt;/strong&gt; sends only the delta operation (such as flipping a todo item&amp;rsquo;s &lt;code&gt;completed&lt;/code&gt; field), keeping payloads small, but requires a way to fetch initial state and demands that the server track each client&amp;rsquo;s current state version. &lt;strong&gt;Event sourcing&lt;/strong&gt; broadcasts the domain event itself, requiring every client to reimplement the business logic needed to interpret and fold that event into local state.&lt;/p&gt;
&lt;p&gt;At the transport layer, WebSockets, Server-Sent Events (SSE), and long polling all operate as 1-to-1 HTTP connections between a single client and a single server instance. Once you scale horizontally across multiple server replicas, the instance that processes a write is often not the instance holding open connections to the other clients that need the update. Rather than having every server replica poll the database, the author recommends using a Pub/Sub backbone to fan updates out across instances and manage connection infrastructure.&lt;/p&gt;
&lt;h2 id="optimizing-the-databases-at-quora"&gt;&lt;a href="https://quoraengineering.quora.com/Optimizing-the-databases-at-Quora" rel="noopener noreferrer"&gt;Optimizing the databases at Quora&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;The Quora engineering team shares how they optimized their database tier as the platform scaled to handle massive data volumes while keeping user-facing latency low. Their first strategy was partitioning data (sharding) by &lt;code&gt;user_id&lt;/code&gt; to distribute load across multiple database servers. Alongside sharding, the team profiled common query patterns, added targeted indexes, and eliminated redundant queries. Quora also built a multi-layered caching architecture—spanning application-level caches for hot objects, database-level query caching, and a distributed Memcached tier—while separating read and write paths so each side could scale independently.&lt;/p&gt;
&lt;p&gt;Beyond those core techniques, Quora invested heavily in comprehensive monitoring and alerting to track database health and catch anomalies early. Key takeaways include treating performance optimization as an ongoing discipline rather than a one-off project, building automation tooling for routine database administration, and constantly balancing raw performance gains against system simplicity.&lt;/p&gt;
&lt;h2 id="failure-mitigation-for-microservices-an-intro-to-aperture"&gt;&lt;a href="https://careersatdoordash.com/blog/failure-mitigation-for-microservices-an-intro-to-aperture/" rel="noopener noreferrer"&gt;Failure Mitigation for Microservices: An Intro to Aperture&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Cong Ma and Matt Ranney from DoorDash examine common failure modes in microservice architectures: cascading failures rippling across service dependencies, retry storms where aggressive retries pile extra load onto an already degraded service, death spirals where failed nodes shift traffic onto surviving nodes until they collapse too, and metastable failures where a positive feedback loop keeps a system wedged even after the original trigger disappears. Traditional defenses like local load shedding, circuit breakers, and reactive autoscaling only see a single service&amp;rsquo;s local state, require brittle threshold tuning, and cannot coordinate with one another; the authors also advocate predictive autoscaling over reactive scaling that kicks in only after load has already spiked.&lt;/p&gt;
&lt;p&gt;From there, the post evaluates Aperture, an open-source reliability management platform from FluxNinja designed around global visibility and closed-loop control. Aperture collects telemetry from service nodes into Prometheus, uses a standalone controller to monitor deviations from SLOs based on declarative YAML policies, and actuates proportional responses—such as load shedding or distributed rate limiting—at individual nodes. After integrating Aperture into a core service in their staging environment, DoorDash found it worked well as an easy-to-configure load shedder compared to their existing setup, though more advanced capabilities like coordinated multi-service failure mitigation remained to be tested.&lt;/p&gt;
&lt;h2 id="zero-configuration-service-mesh-with-on-demand-cluster-discovery"&gt;&lt;a href="https://netflixtechblog.com/zero-configuration-service-mesh-with-on-demand-cluster-discovery-ac6483b52a51" rel="noopener noreferrer"&gt;Zero Configuration Service Mesh with On-Demand Cluster Discovery&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;The Netflix engineering team recounts their journey migrating to a service mesh. Netflix began moving to the cloud in 2008, with the streaming platform running entirely on AWS by 2010. Back when cloud-native tooling barely existed, they built Eureka for service discovery and Ribbon for inter-process communication (IPC). That architecture served them well for a decade, but as internal traffic diversified across REST, GraphQL, and gRPC in multiple programming languages—and teams needed advanced capabilities like adaptive concurrency limits and circuit breaking—maintaining polyglot client libraries became unsustainable. Moving to an Envoy-based service mesh let them centralize IPC logic in an out-of-process proxy.&lt;/p&gt;
&lt;p&gt;The biggest roadblock was that Envoy expects every upstream cluster a service might call to be declared ahead of time, whereas a typical Netflix service talks to dozens of upstream clusters whose owners often don&amp;rsquo;t know the full dependency list—and pushing every cluster in the company to every Envoy proxy was impossible with millions of endpoints. Netflix partnered with Kinvolk and the Envoy community to build &lt;strong&gt;On-Demand Cluster Discovery&lt;/strong&gt;: when an outgoing request targets an unknown cluster, Envoy pauses the request, queries the control plane via CDS (Cluster Discovery Service), fetches its endpoints via EDS backed by Eureka, and resumes the request. This made migration completely zero-configuration for service owners, at the cost of a few extra milliseconds on the very first request to a new cluster—meaning only ultra-latency-sensitive services need to pre-declare clusters or pre-warm connections.&lt;/p&gt;
&lt;h2 id="the-quest-to-understand-metric-movements"&gt;&lt;a href="https://medium.com/pinterest-engineering/the-quest-to-understand-metric-movements-8ab12ae97cda" rel="noopener noreferrer"&gt;The Quest to Understand Metric Movements&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;The Pinterest engineering team introduces the three pillars behind their automated Root Cause Analysis (RCA) system for investigating sudden spikes or drops in core business metrics—movements that can stem from anything from an OS upgrade to a broken data pipeline. The first approach, &lt;strong&gt;Slice and Dice&lt;/strong&gt;, breaks a metric down across dimensions like country, device type, and Pin format into a hierarchical segment tree (inspired by LinkedIn&amp;rsquo;s ThirdEye) and scores each segment&amp;rsquo;s anomaly contribution; this proved especially effective for video playback metrics. The second approach, &lt;strong&gt;General Similarity&lt;/strong&gt;, searches for other metrics that moved in a similar pattern over the same time window using Pearson correlation, Spearman rank correlation, Euclidean distance, and dynamic time warping—with Pearson and Spearman proving most useful because they provide p-values and capture inverse relationships cleanly (with the standard caveat that correlation does not imply causation).&lt;/p&gt;
&lt;p&gt;The third approach, &lt;strong&gt;Experiment Effects&lt;/strong&gt;, inverts traditional A/B test analysis: given a moving metric, it ranks active experiments that had the strongest statistically significant impact on that metric using Welch&amp;rsquo;s t-test and noise filters, and is now integrated into Pinterest&amp;rsquo;s experimentation platform covering nearly 2,000 metrics. Analysts can chain all three techniques iteratively to narrow down root causes. Looking ahead, Pinterest plans to incorporate user feedback loops, explore causal discovery to map cause-and-effect graphs across metrics, and embed RCA directly into their data discovery and visualization tools.&lt;/p&gt;
&lt;h2 id="meta"&gt;&lt;a href="https://cacm.acm.org/research/metas-hyperscale-infrastructure-overview-and-insights/" rel="noopener noreferrer"&gt;Meta&amp;rsquo;s Hyperscale Infrastructure: Overview and Insights&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Chunqiang Tang provides a sweeping overview of Meta&amp;rsquo;s hyperscale infrastructure and the lessons learned along the way, arguing that while few engineers directly build infrastructure at this scale, many mainstream industry technologies originate there. Meta&amp;rsquo;s engineering culture revolves around four pillars: moving fast with continuous deployment, technological openness via a unified monorepo and major open-source projects like PyTorch, Llama, and Presto, conducting systems research directly on production infrastructure rather than in an isolated research lab, and standardizing on shared infrastructure building blocks (for instance, converging product teams onto ZippyDB as a common key-value store). To maximize developer productivity, Meta pushes continuous deployment to extreme scale—executing over 100,000 configuration changes per day with 97% of services deploying completely automatically, while roughly 50% more engineers write serverless functions than traditional long-running services.&lt;/p&gt;
&lt;p&gt;To drive down hardware costs, Meta treats its global fleet of data centers as a single planet-scale computer, automatically placing and migrating workloads across regions, co-designing hardware and software to compensate for commodity hardware failure rates, and tiering storage across hot, warm, and cold media. On the AI front, Meta co-designs the entire stack—from PyTorch, custom AI accelerators, and datacenter fabrics up to foundation models like Llama—and projects that before the end of the decade, more than half of its datacenter power capacity will be dedicated to AI workloads.&lt;/p&gt;
&lt;h2 id="incremental-platforms-monolithic-modular-architecture"&gt;&lt;a href="https://newsletter.optimistengineer.com/p/incremental-platforms-monolithic" rel="noopener noreferrer"&gt;Incremental Platforms: Monolithic Modular Architecture&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Kicking off a series on incremental architecture for SaaS products, Marcos F. Lobo introduces the &lt;strong&gt;Modular Monolith&lt;/strong&gt;: packaging and deploying the entire application as a single artifact while structuring the internal codebase into well-defined modules with clear responsibilities that communicate strictly through explicit interfaces and contracts. This approach retains the operational simplicity of a monolith while capturing the modularity benefits of decoupled design. To pull it off, the author advises defining strict module boundaries and using patterns like Facade to hide internal implementation details, enforcing high cohesion and low coupling within each module, aligning package structure with module boundaries so modules can be extracted later if needed, pairing use-case testing with CI/CD, and wiring dependencies via dependency injection or in-process messaging and events.&lt;/p&gt;
&lt;p&gt;The benefits include simple single-artifact deployment and operations, easier refactoring and local development, fast in-memory calls between modules instead of network RPCs, straightforward transactional consistency, and natural alignment with Domain-Driven Design when modules map to bounded contexts. According to the author, a modular monolith is an ideal starting point for early-stage products, small teams, and systems that prioritize strong consistency, while serving as a clean stepping stone toward microservices if and when business scale demands it.&lt;/p&gt;
&lt;h2 id="bonus-interesting-videos-from-bytebytego"&gt;Bonus: Interesting videos from &lt;a href="https://bytebytego.com/" rel="noopener noreferrer"&gt;ByteByteGo&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;&lt;a href="https://www.youtube.com/watch?v=3Kqal7QaCCM" rel="noopener noreferrer"&gt;How the Garbage Collector Works in Java, Python, and Go!&lt;/a&gt;&lt;/p&gt;
&lt;h2 id="bonus-2-interesting-images-from-bytebytego"&gt;Bonus 2: Interesting images from &lt;a href="https://bytebytego.com/" rel="noopener noreferrer"&gt;ByteByteGo&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;&lt;img src="https://substack-post-media.s3.amazonaws.com/public/images/c3beec9e-cd04-4748-ae7e-3299b42883f6_2360x2824.png" alt="How to Build Idempotent APIs"&gt;
&lt;img src="https://substack-post-media.s3.amazonaws.com/public/images/2f632296-4214-4ec8-a1d2-280e9b7f2696_1280x1532.gif" alt="12 Algorithms for System Design Interviews"&gt;
&lt;img src="https://substack-post-media.s3.amazonaws.com/public/images/65a92a81-f2c5-4aed-9faa-35a66124cffe_1283x1536.gif" alt="How Kubernetes Works?"&gt;
&lt;img src="https://substack-post-media.s3.amazonaws.com/public/images/a6903386-4b5f-450e-b64d-839c6fdf8238_1280x1601.gif" alt="PostgreSQL 101: The Everything Database"&gt;
&lt;img src="https://substack-post-media.s3.amazonaws.com/public/images/06c5ad46-faaf-479b-bde3-b5de8033b4e9_1280x1664.gif" alt="Top 12 Tips for API Security"&gt;&lt;/p&gt;
&lt;h2 id="bonus-3-interesting-images-from-designgurus"&gt;Bonus 3: Interesting images from &lt;a href="https://designgurus.io/" rel="noopener noreferrer"&gt;DesignGurus&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;&lt;img src="https://www.designgurus.io/_next/image?url=https%3A%2F%2Fstorage.googleapis.com%2Fdownload%2Fstorage%2Fv1%2Fb%2Fdesigngurus-prod.appspot.com%2Fo%2Fee5726e8e29469477b999c100%3Fgeneration%3D1726724098184989%26alt%3Dmedia&amp;amp;w=3840&amp;amp;q=75&amp;amp;dpl=dpl_3Rx6M949Pc1cQKT3QTdj87FcKNdg" alt="System Design Master Template"&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.&lt;/em&gt;&lt;/p&gt;</description></item></channel></rss>