<?xml version="1.0" encoding="utf-8" standalone="yes"?><rss version="2.0" xmlns:atom="http://www.w3.org/2005/Atom"><channel><title>Data Engineering on miti99</title><link>https://miti99.com/en/tags/data-engineering/</link><description>Recent content in Data Engineering on miti99</description><generator>Hugo</generator><language>en</language><lastBuildDate>Mon, 16 Nov 2026 17:39:19 +0700</lastBuildDate><atom:link href="https://miti99.com/en/tags/data-engineering/index.xml" rel="self" type="application/rss+xml"/><item><title>Newsletter #83</title><link>https://miti99.com/en/post/2026/02/24/</link><pubDate>Tue, 24 Feb 2026 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2026/02/24/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #83.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="how-uber-scaled-data-replication-to-move-petabytes-every-day"&gt;&lt;a href="https://www.uber.com/en-IN/blog/scaled-data-replication/" rel="noopener noreferrer"&gt;How Uber Scaled Data Replication to Move Petabytes Every Day&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Uber shares how its engineering team scaled HiveSync—its cross-region data replication service built on Apache Hadoop DistCp—to keep pace with a data lake exceeding 350 PB. In Q3 2022 alone, daily replicated data volume jumped from 250 TB to 1 PB, the number of datasets managed by HiveSync grew from 30,000 to 144,000, and daily replication jobs surged from 10,000 to 374,000. This growth was driven by an active-passive architecture that concentrated ~90% of writes in the primary region, onboarding all Hive datasets into HiveSync, and an upcoming cloud migration. Under that load, replication latency SLAs (P100 under 4 hours, P99.9 under 20 minutes) were slipping.&lt;/p&gt;
&lt;p&gt;To fix this, Uber made four targeted improvements to DistCp. First, they offloaded two resource-intensive phases—Copy Listing and Input Splitting—from the central HiveSync server to the YARN Application Master, cutting job submission latency by 90%. Second, they parallelized Copy Listing across a thread pool making concurrent &lt;code&gt;namenode&lt;/code&gt; RPCs, reducing p99 latency by 60% and max latency by 75%. Third, they parallelized file concatenation inside the Copy Committer, slashing merge time by 97.29%. Finally, they enabled &amp;ldquo;Uber jobs&amp;rdquo; to run single-mapper replication tasks (which accounted for 52% of all jobs) directly inside the Application Master JVM, eliminating roughly 268,000 YARN container launches per day. Together, these optimizations increased incremental replication throughput 5x within a year and enabled Uber to migrate over 306 PB of data to the cloud.&lt;/p&gt;
&lt;h2 id="how-to-write-a-good-spec-for-ai-agents"&gt;&lt;a href="https://addyosmani.com/blog/good-spec/" rel="noopener noreferrer"&gt;How to write a good spec for AI agents&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Addy Osmani lays out a practical framework for writing specifications for AI coding agents, addressing a classic tension: a spec needs enough detail to steer the agent accurately without bloating the prompt and overwhelming the context window. Principle one is to start with a high-level vision—provide a concise objective and let the agent expand it into a detailed plan that you review and refine. Principle two is to structure the spec like a Product Requirements Document (PRD) covering six essential areas: executable commands, testing conventions, project structure, code style examples, Git workflow, and explicit boundaries. An analysis by GitHub of over 2,500 agent configuration files found that vague, high-level instructions were the single most common failure mode.&lt;/p&gt;
&lt;p&gt;Principle three is to break work into modular prompts or sub-agent tasks rather than one massive monolith, since the &amp;ldquo;curse of instructions&amp;rdquo; causes model adherence to drop sharply when too many rules compete at once. Principle four is building in quality gates: a three-tier boundary system (&amp;ldquo;Always do,&amp;rdquo; &amp;ldquo;Ask first,&amp;rdquo; and &amp;ldquo;Never do&amp;rdquo;) paired with self-verification hooks. Finally, treat the spec as a living document that gets tested and updated as the codebase evolves. The article also warns against common pitfalls like vague prompts (&amp;ldquo;build something cool&amp;rdquo;), dumping raw context without summarization, skipping human code review, and moving faster than your ability to verify.&lt;/p&gt;
&lt;h2 id="unconventional-postgresql-optimizations"&gt;&lt;a href="https://hakibenita.com/postgresql-unconventional-optimizations" rel="noopener noreferrer"&gt;Unconventional PostgreSQL Optimizations&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Haki Benita walks through three unconventional PostgreSQL optimization techniques. First is enabling &lt;code&gt;constraint_exclusion = on&lt;/code&gt; so the query planner uses table &lt;code&gt;CHECK&lt;/code&gt; constraints to skip full table scans altogether. When a user accidentally queries a value that violates a check constraint—for example, &lt;code&gt;WHERE plan = 'Pro'&lt;/code&gt; when the constraint only allows lowercase &lt;code&gt;'free'&lt;/code&gt; or &lt;code&gt;'pro'&lt;/code&gt;—PostgreSQL&amp;rsquo;s default &lt;code&gt;partition&lt;/code&gt; setting still scans the entire table (taking ~7.4 ms in the benchmark), whereas turning &lt;code&gt;constraint_exclusion&lt;/code&gt; on lets the planner prove the query cannot match any rows and return instantaneously. Because checking constraints adds planning overhead, PostgreSQL defaults to only checking them on inherited partitions and &lt;code&gt;UNION ALL&lt;/code&gt; queries, making &lt;code&gt;on&lt;/code&gt; best suited for analytics or reporting databases where users write ad-hoc queries.&lt;/p&gt;
&lt;p&gt;The second technique is indexing a lower-cardinality expression: instead of building a B-Tree index on a raw &lt;code&gt;timestamptz&lt;/code&gt; column (214 MB), indexing the truncated date via &lt;code&gt;date_trunc('day', ...)&lt;/code&gt; shrinks the index to 66 MB—over 3x smaller—and speeds up queries through page-level deduplication. Because expression indexes only kick in when queries write the exact same expression, he suggests pairing them with PostgreSQL 18&amp;rsquo;s virtual generated columns for ergonomics. The third technique enforces uniqueness on large text values like URLs using a hash index: PostgreSQL doesn&amp;rsquo;t support &lt;code&gt;UNIQUE&lt;/code&gt; hash indexes directly, but you can create an &lt;code&gt;EXCLUDE USING hash (url WITH =)&lt;/code&gt; exclusion constraint, shrinking the index from 154 MB (B-Tree) down to 32 MB—nearly 5x smaller. The trade-offs are that exclusion constraints cannot be referenced by foreign keys and do not support &lt;code&gt;ON CONFLICT (url) DO UPDATE&lt;/code&gt;.&lt;/p&gt;
&lt;h2 id="software-engineering-when-machine-writes-the-code"&gt;&lt;a href="https://www.shayon.dev/post/2026/19/software-engineering-when-the-machine-writes-code/" rel="noopener noreferrer"&gt;Software engineering when machine writes the code&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Shayon Mukherjee reflects on what happens to software engineering when machines write the bulk of the code. He isn&amp;rsquo;t anti-AI, but worries that in chasing raw velocity we are trading away deep system comprehension. By Jevons paradox, making a resource more efficient increases total consumption; similarly, AI won&amp;rsquo;t lead us to write less software—it will lead us to build sprawling, complex systems faster than humans can mentally absorb them. He contrasts writing code by hand—where you build a rich mental model through hundreds of micro-decisions—with reviewing AI-generated PRs, where you try to reconstruct understanding after the fact. Accepting working code without truly understanding why it works creates dangerous blind spots during 3 a.m. incidents and long-term maintenance.&lt;/p&gt;
&lt;p&gt;He is especially concerned for junior engineers: if they skip the formative struggle of implementing things from scratch, making mistakes, and debugging for hours, they may never develop the pattern recognition and architectural intuition that only come from friction. His suggested remedy is using AI as a Socratic tutor to explain concepts and trade-offs while still writing the implementation yourself. He also proposes a zoned approach to codebases: core domain logic demands deep human ownership with minimal AI generation, standard application code can lean on AI with thorough review, and pure boilerplate can be delegated almost entirely to AI.&lt;/p&gt;
&lt;h2 id="a-guide-to-effective-prompt-engineering"&gt;&lt;a href="https://blog.bytebytego.com/p/a-guide-to-effective-prompt-engineering" rel="noopener noreferrer"&gt;A Guide to Effective Prompt Engineering&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;ByteByteGo presents a practical guide to prompt engineering—crafting instructions that reliably elicit the right output from large language models. Anyone can type a prompt, but getting consistent, high-quality results is a distinct engineering skill. A well-structured prompt typically combines four elements: task description, background context, illustrative examples, and specific output constraints. Modern LLM APIs also split instructions into a system prompt (defining role, tone, and guardrails) and a user prompt (the actual task). Clarity is paramount: state exactly what you want, define the output format explicitly, and never assume the model can read your mind; supplying sufficient context also dramatically reduces hallucinations. The post notes how models use in-context learning from examples in the prompt, and points out the &amp;ldquo;lost in the middle&amp;rdquo; effect where models pay closer attention to instructions at the very beginning and end of a long prompt.&lt;/p&gt;
&lt;p&gt;Five core prompting techniques are covered: zero-shot prompting (direct instructions with no examples, suited for simple tasks); few-shot prompting (providing two to five input–output examples to demonstrate the exact pattern and format); chain-of-thought prompting (asking the model to reason step by step before answering complex logic or math problems); role prompting (assigning a persona to anchor perspective and depth); and prompt chaining (breaking a complex workflow into a sequence of focused prompts). On best practices, iterate systematically, version-control your prompts, and test against diverse inputs and edge cases while avoiding vague phrasing, bloated walls of text, and unspecified output formats.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.&lt;/em&gt;&lt;/p&gt;</description></item><item><title>Newsletter #27</title><link>https://miti99.com/en/post/2025/05/14/</link><pubDate>Wed, 14 May 2025 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2025/05/14/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #27.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="heading"&gt;&lt;a href="https://blog.cloudflare.com/introducing-the-cloudflare-vite-plugin/" rel="noopener noreferrer"&gt;&amp;ldquo;Just use Vite&amp;rdquo;&amp;hellip; with the Workers runtime&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Cloudflare announced version 1.0 of the Cloudflare Vite plugin alongside official support for React Router v7. Previously, Vite&amp;rsquo;s development server always executed server-side code inside Node.js—even when the application targeted Cloudflare Workers in production—causing subtle, hard-to-catch discrepancies between local dev and production. Built on the experimental Environment API introduced in Vite 6, the new plugin runs your Worker code directly inside the native Workers runtime (&lt;code&gt;workerd&lt;/code&gt;) during local development, keeping dev behavior as close to production as possible.&lt;/p&gt;
&lt;p&gt;For single-page applications (SPAs) built with React, Vue, or Svelte, you can scaffold a new project with the &lt;code&gt;create-cloudflare&lt;/code&gt; CLI or migrate an existing app simply by adding &lt;code&gt;@cloudflare/vite-plugin&lt;/code&gt; to your Vite config and creating a &lt;code&gt;wrangler.jsonc&lt;/code&gt; file. When you edit server-side Worker code, Vite hot-reloads the Worker environment without losing client-side UI state in the browser. Running &lt;code&gt;vite build&lt;/code&gt; outputs both client and server bundles, &lt;code&gt;vite preview&lt;/code&gt; previews the production build inside the Workers runtime, and &lt;code&gt;wrangler deploy&lt;/code&gt; deploys straight to Cloudflare without an extra bundling step. With React Router v7 (the successor to Remix), a single entry file defines the Worker across both dev and build without needing a separate adapter, and the plugin supports the full Cloudflare Developer Platform including KV, D1, Durable Objects, and Workers AI.&lt;/p&gt;
&lt;h2 id="the-best-programmers-i-know"&gt;&lt;a href="https://endler.dev/2025/best-programmers/" rel="noopener noreferrer"&gt;The Best Programmers I Know&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Matthias Endler distills the common traits he has observed in the best programmers he has ever worked with—a list he wishes he had when starting out. First and foremost, they read the canonical reference documentation directly instead of guessing, copying from Stack Overflow, or asking an LLM. Second, they know their tools inside and out: why the tool was created, who maintains it, where its limits lie, and what the surrounding ecosystem looks like. Great programmers also actually read error messages carefully, break intimidating problems down into small, solvable steps, and aren&amp;rsquo;t afraid to get their hands dirty in unfamiliar codebases—which is how they become the go-to person on their team.&lt;/p&gt;
&lt;p&gt;Beyond raw technical chops, they generously help others, write clearly because clear writing reflects clear thinking, and never stop learning—some of the sharpest programmers Endler knows are in their 60s and still run circles around developers half their age. They don&amp;rsquo;t care about titles or hierarchy, build lasting reputation by shipping widely used work, stay patient with both computers and people, never blame &amp;ldquo;random&amp;rdquo; hardware glitches or compiler bugs before checking their own assumptions, aren&amp;rsquo;t afraid to say &amp;ldquo;I don&amp;rsquo;t know,&amp;rdquo; refuse to guess when requirements are ambiguous, and relentlessly favor simple code. Endler notes that this isn&amp;rsquo;t a checklist or a competition, just a reminder that there are no shortcuts around deliberate practice.&lt;/p&gt;
&lt;h2 id="rdel-87-how-do-ai-coding-tools-actually-change-developer-work"&gt;&lt;a href="https://rdel.substack.com/p/rdel-87-how-do-ai-coding-tools-actually" rel="noopener noreferrer"&gt;RDEL #87: How do AI coding tools actually change developer work?&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;In this installment of &lt;em&gt;Research-Driven Engineering Leadership&lt;/em&gt;, Lizzie Matusov summarizes a three-week randomized controlled trial conducted by Microsoft and the Institute for Work Life across 228 engineers at a global software company. Participants were split into three groups—engineers newly granted access to GitHub Copilot, a control group using no AI tools, and a comparison group of existing Copilot users—combining daily diary studies with system telemetry. Before the trial began, 86% of experienced Copilot users believed AI coding tools were useful compared to just 44% of non-users, and the single most common reason engineers hadn&amp;rsquo;t tried AI tools yet was simply being too busy.&lt;/p&gt;
&lt;p&gt;After three weeks, enjoyment and perceived usefulness rose significantly among the new users; 84% reported that Copilot positively changed how they worked, cutting down on tedious boilerplate and making coding more enjoyable. Engineers used Copilot not just for code completion, but also as a replacement for web search and as a sounding board for ideation and design. Interestingly, system telemetry showed no statistically significant difference in lines of code, pull request volume, or active coding time, and engineers&amp;rsquo; baseline level of trust in AI-generated code remained unchanged. Matusov advises engineering leaders to measure productivity holistically across satisfaction and flow using the SPACE framework, carve out dedicated time for teams to experiment, and view AI as a way to offload repetitive drudgery so engineers can focus on design and learning.&lt;/p&gt;
&lt;h2 id="overclocking-dbt-discord"&gt;&lt;a href="https://discord.com/blog/overclocking-dbt-discords-custom-solution-in-processing-petabytes-of-data" rel="noopener noreferrer"&gt;Overclocking dbt: Discord&amp;rsquo;s Custom Solution in Processing Petabytes of Data&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Discord data engineer Chris Dong shares how his team scaled &lt;code&gt;dbt&lt;/code&gt; (data build tool) to process petabytes of data across more than 100 developers collaborating on 2,500+ models. Their out-of-the-box setup quickly hit a wall: full project compilation took over 20 minutes, default incremental materialization strategies struggled at petabyte scale, and developers constantly clobbered each other&amp;rsquo;s test tables in shared environments. To isolate environments cleanly, they overrode the &lt;code&gt;generate_alias_name&lt;/code&gt; macro to automatically prefix table names with the developer&amp;rsquo;s username in local dev, the PR number or commit SHA in CI/CD, and the canonical name in production.&lt;/p&gt;
&lt;p&gt;On the performance side, they replaced standard &lt;code&gt;MAX(timestamp)&lt;/code&gt; incremental filtering with explicit &lt;code&gt;start_date&lt;/code&gt; and &lt;code&gt;end_date&lt;/code&gt; variables—noting that the expensive full-table scans came from the &lt;code&gt;MAX(timestamp)&lt;/code&gt; subquery in the old pattern rather than the &lt;code&gt;is_incremental()&lt;/code&gt; macro itself. They also patched partial-parsing hash mismatches and leveraged zero-copy BigQuery partition copies; together, these optimizations sped up runs by 5x. Historical backfills were automated via semantic versioning inside each model&amp;rsquo;s &lt;code&gt;meta&lt;/code&gt; config: bumping the major version triggers a full refresh of the table and all downstream dependents, while bumping the minor version runs a targeted date-chunked backfill. A shared macro library and Buildkite CI/CD pipeline validate query dry-run costs, dependencies, and blast radius before any change lands in production.&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026.&lt;/em&gt;&lt;/p&gt;</description></item><item><title>Newsletter #11</title><link>https://miti99.com/en/post/2025/04/12/</link><pubDate>Sat, 12 Apr 2025 00:00:00 +0700</pubDate><guid>https://miti99.com/en/post/2025/04/12/</guid><description>&lt;p&gt;&lt;em&gt;Enjoy Newsletter #11.&lt;/em&gt;&lt;/p&gt;
&lt;h2 id="enabling-near-real-time-data-analytics-on-the-data-lake"&gt;&lt;a href="https://engineering.grab.com/enabling-near-realtime-data-analytics" rel="noopener noreferrer"&gt;Enabling near real-time data analytics on the data lake&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Grab shares how they brought data lake ingestion latency down to just a few minutes instead of waiting on hourly or daily batch jobs. Their legacy Hive-and-Parquet pipeline worked well for analytical scans but made record-level updates and upserts prohibitively expensive, since modifying a single record required rewriting entire Parquet files. Grab migrated to &lt;strong&gt;Apache Hudi&lt;/strong&gt;, tailoring table types to each source&amp;rsquo;s workload: for high-throughput Kafka event streams, they use &lt;strong&gt;Merge On Read (MOR)&lt;/strong&gt; tables where Apache Flink appends fast Avro row logs and Spark runs periodic background compaction into columnar Parquet files; for lower-throughput sources, they use simpler &lt;strong&gt;Copy On Write (COW)&lt;/strong&gt; tables, accepting a 10–15 minute ingestion latency.&lt;/p&gt;
&lt;p&gt;The article dives into practical production lessons: partitioning high-volume Kafka event tables by event hour to speed up writes and bound compaction scope, using Flink CDC to stream binlog changes directly from relational databases, and choosing Hudi&amp;rsquo;s Bucket Index for deterministic hashing without external index lookups (despite the trade-off that bucket counts are hard to resize later). As a result, business analysts get much fresher operational dashboards, and the fraud detection team can query near-real-time transaction data on the lake without putting extra load on production OLTP databases.&lt;/p&gt;
&lt;h2 id="the-ultimate-guide-to-software-architecture-documentation"&gt;&lt;a href="https://www.workingsoftware.dev/software-architecture-documentation-the-ultimate-guide/" rel="noopener noreferrer"&gt;The Ultimate Guide To Software Architecture Documentation&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This comprehensive guide covers how to write, structure, visualize, and maintain software architecture documentation. The author pushes back against the &amp;ldquo;the code is the documentation&amp;rdquo; myth: source code tells you &lt;em&gt;what&lt;/em&gt; the system does at the lowest level, but cannot tell you &lt;em&gt;why&lt;/em&gt; architectural trade-offs were made, what non-functional quality attributes drove the design, or how the system fits into the broader business context. Good architecture documentation aligns stakeholders around a shared mental model, speeds up onboarding for new engineers, guides teams when building new features, and streamlines communication with external partners and auditors.&lt;/p&gt;
&lt;p&gt;For structure, the author recommends the open-source, process-agnostic &lt;strong&gt;arc42&lt;/strong&gt; template, advising teams not to fill out every section up front on day one, but to grow the document incrementally as decisions are made. For visualization, the &lt;strong&gt;C4 model&lt;/strong&gt; provides a clean hierarchy across four zoom levels: Context, Containers, Components, and Code. Finally, the post champions &amp;ldquo;Documentation as Code&amp;rdquo; and &amp;ldquo;Diagrams as Code&amp;rdquo; using tools like AsciiDoc, PlantUML, and Structurizr so architecture docs live in Git alongside the codebase and evolve in lockstep with pull requests.&lt;/p&gt;
&lt;h2 id="beyond-the-basics-designing-for-a-million-users"&gt;&lt;a href="https://javarevisited.substack.com/p/beyond-the-basics-designing-for-a" rel="noopener noreferrer"&gt;Beyond the Basics: Designing for a Million Users&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This post summarizes the opening chapter of Alex Xu&amp;rsquo;s &lt;em&gt;System Design Interview&lt;/em&gt;, walking through the foundational building blocks for scaling a web application from a handful of users to millions. It starts with a single-server deployment: as traffic grows, latency spikes, a single hardware failure takes the entire site offline, vertical scaling hits cost and physical ceilings, and geographically distant users suffer slow page loads.&lt;/p&gt;
&lt;p&gt;The first step is scaling horizontally rather than vertically, pairing a stateless web tier (moving user session data out into a shared persistent store or cache) with a load balancer to distribute traffic across multiple web servers. Next comes choosing the right database (relational SQL vs. NoSQL), adding read replicas and database sharding, placing a cache tier in front of hot database reads, offloading static assets to a CDN, introducing message queues for asynchronous background processing, using geo-routing for global users, and rounding out the stack with centralized logging, metrics, and automation. It serves as a solid primer for anyone preparing for system design interviews.&lt;/p&gt;
&lt;h2 id="decision-making-pitfalls-for-technical-leaders"&gt;&lt;a href="https://chelseatroy.com/2024/10/16/decision-making-pitfalls-for-technical-leaders/" rel="noopener noreferrer"&gt;Decision-Making Pitfalls for Technical Leaders&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Chelsea Troy points out that the software industry routinely promotes strong individual contributors into technical leadership roles with zero training in how decision-making changes at scale, highlighting three habits that are harmless as an IC but destructive once you hold authority. The first is &lt;strong&gt;assuming context&lt;/strong&gt;: applying a &amp;ldquo;best practice&amp;rdquo; without examining what problem it was designed to solve, leading to mismatched architectures that the leader can&amp;rsquo;t justify to their team or executives. The second is &lt;strong&gt;treating every criterion as an optimization target&lt;/strong&gt;, which causes decisions to drag on forever or get reopened repeatedly, whipsawing downstream teams. The third is &lt;strong&gt;manufacturing false urgency&lt;/strong&gt;, subjecting the team to constant fire drills over foreseeable problems that should have been planned for.&lt;/p&gt;
&lt;p&gt;Her remedies are straightforward: understand the concrete trade-offs of every option within your specific project context, explicitly separate criteria you want to &lt;em&gt;maximize&lt;/em&gt; from constraints that only need to meet a &amp;ldquo;good enough&amp;rdquo; threshold (satisficing), and actively absorb pressure rather than amplifying it—because people rarely do their best thinking in a panic. Most &amp;ldquo;emergencies&amp;rdquo; in software engineering, she argues, are actually leadership failures in disguise.&lt;/p&gt;
&lt;h2 id="how-i-know-i"&gt;&lt;a href="https://www.seangoedecke.com/thoughts-about-engineers/" rel="noopener noreferrer"&gt;How I know I&amp;rsquo;m working with a strong engineer&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Sean Goedecke shares a simple gut-check for telling whether you&amp;rsquo;re working with a strong engineer: count how often two specific thoughts cross your mind while collaborating with them. The first thought is, &lt;em&gt;&amp;ldquo;Oh, good catch—I hadn&amp;rsquo;t thought of that!&amp;quot;&lt;/em&gt;—triggered when they spot a subtle bug, improve the user experience, or simplify a design, rather than when they pedantically nitpick edge cases that don&amp;rsquo;t matter. If you find yourself thinking that constantly around someone, you should probably defer to their judgment on big technical calls.&lt;/p&gt;
&lt;p&gt;The second thought is, &lt;em&gt;&amp;ldquo;Nice, that&amp;rsquo;ll work,&amp;rdquo;&lt;/em&gt; when they propose the exact sensible approach you would have come up with yourself. That reaction signals reliability and steady execution—which the author notes is actually high praise. Engineers can be great in many different ways, of course, but this two-question heuristic is a surprisingly reliable intuitive litmus test.&lt;/p&gt;
&lt;h2 id="ugly-code-and-dumb-things"&gt;&lt;a href="https://lucumr.pocoo.org/2025/2/20/ugly-code/" rel="noopener noreferrer"&gt;Ugly Code and Dumb Things&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Armin Ronacher, creator of Flask, Jinja, and Werkzeug, reflects on two competing passions in software: crafting clean, reusable library abstractions versus hacking together fast, pragmatic solutions for real users. He points to &lt;strong&gt;Flamework&lt;/strong&gt;—an open-source PHP framework capturing Flickr&amp;rsquo;s early engineering ethos of &amp;ldquo;doing the simplest thing that could possibly work.&amp;rdquo; Its code looks messy to modern eyes, concatenating raw SQL strings and using global state, yet it nailed the problems that actually mattered for scaling Flickr: user-based sharding, database replication, and production observability.&lt;/p&gt;
&lt;p&gt;Clean, aesthetically pleasing code doesn&amp;rsquo;t guarantee product success if it doesn&amp;rsquo;t solve a real problem for real people, while &amp;ldquo;ugly&amp;rdquo; code that ships often embodies the exact right compromises needed to iterate quickly. Both mindsets are valid, though they rarely coexist peacefully in the same codebase. The real art, Ronacher concludes, is knowing when to transition from duct-tape pragmatism to solid architectural foundations—because successful projects eventually grow up and need to be re-engineered.&lt;/p&gt;
&lt;h2 id="development-philosophy"&gt;&lt;a href="https://develop.sentry.dev/getting-started/philosophy/" rel="noopener noreferrer"&gt;Development Philosophy&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This page from Sentry&amp;rsquo;s engineering handbook distills over a decade of lessons into their core development philosophy. Sentry emphasizes that code is not art—it is a tool for solving customer problems—so engineers must constantly weigh building clean abstractions against shipping pragmatic &amp;ldquo;duct tape,&amp;rdquo; and ask experienced teammates when unsure where on that spectrum a task falls. Even &amp;ldquo;correctness&amp;rdquo; can be a trap: for instance, forcing rigid static typing gymnastics onto Python can slow down iteration without delivering commensurate value. Duct tape is great for validating ideas quickly, but you can&amp;rsquo;t build a lasting business on it alone—Sentry is equally willing to invest in multi-year foundational bets, such as replacing shell-outs to &lt;code&gt;llvm-symbolizer&lt;/code&gt; with their own custom Rust-based &lt;code&gt;symbolicator&lt;/code&gt; service.&lt;/p&gt;
&lt;p&gt;Other guiding principles include: avoiding overly &amp;ldquo;clever&amp;rdquo; code that future readers (including yourself six months later) will struggle to decipher; encouraging anyone to open a pull request against any repository in the company; standardizing primarily on Python, Rust, and TypeScript; avoiding proprietary database lock-in so Sentry remains easy to self-host; weighing the long-term maintenance cost of every third-party dependency; and keeping &lt;code&gt;main&lt;/code&gt; green and deployable at all times.&lt;/p&gt;
&lt;h2 id="ai-is-stifling-tech-adoption"&gt;&lt;a href="https://vale.rocks/posts/ai-is-stifling-tech-adoption" rel="noopener noreferrer"&gt;AI is Stifling Tech Adoption&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;The author argues that AI coding assistants are quietly stifling the adoption of newer web technologies for two reasons. The first is the &lt;strong&gt;knowledge cutoff trap&lt;/strong&gt;: because foundation models are trained on historical snapshots (at the time of writing, Anthropic&amp;rsquo;s latest models had a cutoff around April 2024), they know little or nothing about newly released frameworks or major version upgrades. That creates a vicious cycle: poor AI support discourages developers from adopting a new tool, fewer users means fewer blog posts and forum answers, and less public writing leaves future model training runs starved of examples. The second culprit is &lt;strong&gt;system prompt bias&lt;/strong&gt;: many AI coding tools explicitly instruct the model behind the scenes to default to React and Tailwind CSS—to the point where Claude once rewrote the author&amp;rsquo;s vanilla HTML/CSS/JS snippet into React despite explicit instructions not to.&lt;/p&gt;
&lt;p&gt;To test this, the author asked four major AI chatbots to &amp;ldquo;build a web app&amp;rdquo; with no stack specified: Claude and ChatGPT defaulted straight to React + Tailwind every time, Gemini used vanilla HTML/CSS/JS but immediately suggested migrating to React, and DeepSeek was more flexible but took several prompts to produce working code. The net effect is that technologies that were dominant right before ChatGPT launched are having their dominance locked in, newcomers are being steered toward a monoculture without realizing it, and AI vendors ought to be far more transparent about the stack biases baked into their system prompts.&lt;/p&gt;
&lt;h2 id="february---the-rest-of-the-story---jvm-weekly-vol-119"&gt;&lt;a href="https://www.jvm-weekly.com/p/february-the-rest-of-the-story-jvm" rel="noopener noreferrer"&gt;February - The Rest of the Story - JVM Weekly vol. 119&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This February 27, 2025 issue of JVM Weekly rounds up notable stories from February that didn&amp;rsquo;t fit into earlier editions. The headline news is that the next LTS release of &lt;strong&gt;Scala 3&lt;/strong&gt;, slated for Q4 2025, will officially drop support for JDK 8; the Scala team is currently debating between JDK 11 and JDK 17 as the new minimum baseline, driven largely by the upcoming removal of memory-access methods in &lt;code&gt;sun.misc.Unsafe&lt;/code&gt;, which forces a redesign of how &lt;code&gt;lazy val&lt;/code&gt; is compiled under the hood. The issue also highlights a set of J.P. Morgan Java Lead interview questions, a series on SOLID principles in Kotlin, a draft JEP under Project Leyden that streamlines creating AOT caches into a single command-line step, and a bytecode-level auto-parallelization prototype that achieved up to a 9x speedup in the author&amp;rsquo;s benchmarks.&lt;/p&gt;
&lt;p&gt;Rounding out the issue is part two of a series on building agentic AI systems with Quarkus—where application control flow is handed over to an LLM—alongside fresh ecosystem releases including KotlinPoet 2.0, Dokka 2.0, the Ktor CLI, Pulumi Java 1.0, and CheerpJ 3.1, a WebAssembly-powered JVM that runs full Java applications directly in the browser.&lt;/p&gt;
&lt;h2 id="manager-antipatterns"&gt;&lt;a href="https://blogs.newardassociates.com/blog/2024/management-antipatterns.html" rel="noopener noreferrer"&gt;Manager Antipatterns&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Ted Neward catalogs twelve common engineering management antipatterns—dysfunctional archetypes that companies repeatedly create through poor hiring, promotion, or org design. The roster includes the &lt;strong&gt;Absentee Manager&lt;/strong&gt; (never around), &lt;strong&gt;The Radical&lt;/strong&gt; (wants to burn everything down and rewrite from scratch), &lt;strong&gt;The Nepotist&lt;/strong&gt; (given the team because they&amp;rsquo;re friends with an executive), &lt;strong&gt;The Mouthpiece&lt;/strong&gt; (merely parrots upper management), &lt;strong&gt;The One-Trick Pony&lt;/strong&gt; (applies the same hammer to every problem), &lt;strong&gt;The Chicken Little&lt;/strong&gt; (panics constantly), &lt;strong&gt;The Pushover&lt;/strong&gt; (avoids all conflict), &lt;strong&gt;The Transplant&lt;/strong&gt; (dropped in from an unrelated domain), &lt;strong&gt;The Perfectionist&lt;/strong&gt; (micromanages every detail), &lt;strong&gt;The Promoted Superstar&lt;/strong&gt; (the best coder forced into management), &lt;strong&gt;The Riddler&lt;/strong&gt; (communicates in cryptic hints instead of clear expectations), and &lt;strong&gt;The Player-Coach&lt;/strong&gt; (stretched thin trying to manage and code simultaneously).&lt;/p&gt;
&lt;p&gt;For each archetype, Neward sketches the behavior and offers practical survival strategies from three vantage points: when you report to that manager, when that manager reports to you, and when you realize &lt;em&gt;you&lt;/em&gt; have become that manager yourself. It&amp;rsquo;s a handy field guide for diagnosing leadership dysfunction wherever you sit on the org chart.&lt;/p&gt;
&lt;h2 id="pause--decision-making-superpower"&gt;&lt;a href="https://read.perspectiveship.com/p/pause" rel="noopener noreferrer"&gt;Pause – Decision-Making Superpower&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;The author argues that even if you master every mental model and decision-making framework in the book, none of them will save you if you react on impulse. Recalling an early-career mistake—giving a snap feature estimate live on a client call and then pulling all-nighters to hit the impossible deadline—the author adopted a personal rule: never give an immediate commitment in the room; always ask to follow up after the meeting. Taking a deliberate pause creates space to step back from emotional hijacking, check for blind spots, and gather missing context—and people rarely push back when you simply tell them it&amp;rsquo;s a personal rule of yours. Nobel laureate Daniel Kahneman famously used the same tactic to stop himself from saying yes to requests on the spot that he would later regret.&lt;/p&gt;
&lt;p&gt;Pausing is especially powerful when you receive an infuriating Slack message or email—wait a few hours or sleep on it before replying—or when fielding high-stakes questions during a crisis, such as right after a layoff. And when you genuinely don&amp;rsquo;t have all the answers yet, state clearly what is within your control and what isn&amp;rsquo;t, and share the concrete steps currently underway so your team knows you are taking their concerns seriously.&lt;/p&gt;
&lt;h2 id="aposd-vs-clean-code"&gt;&lt;a href="https://github.com/johnousterhout/aposd-vs-clean-code/blob/main/README.md" rel="noopener noreferrer"&gt;APOSD vs Clean Code&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;This GitHub repository records a written debate between John Ousterhout, author of &lt;em&gt;A Philosophy of Software Design&lt;/em&gt; (APOSD), and Robert &amp;ldquo;Uncle Bob&amp;rdquo; Martin, author of &lt;em&gt;Clean Code&lt;/em&gt;, hashing out their sharpest disagreements on software design. While both agree that the ultimate goal is making software easier to understand and modify, they clash across three main areas. On &lt;strong&gt;method length&lt;/strong&gt;, &lt;em&gt;Clean Code&lt;/em&gt; famously advocates tiny functions of just a few lines, whereas Ousterhout argues that decomposing functions too aggressively creates &amp;ldquo;shallow&amp;rdquo; interfaces and tangles of tiny coupled methods that force readers to jump back and forth to reconstruct the control flow.&lt;/p&gt;
&lt;p&gt;On &lt;strong&gt;code comments&lt;/strong&gt;, &lt;em&gt;Clean Code&lt;/em&gt; treats most comments as a failure to express intent in code, while APOSD views comments as indispensable for capturing high-level interface contracts and design intent that code alone cannot express. On &lt;strong&gt;Test-Driven Development (TDD)&lt;/strong&gt;, Martin champions strict red-green-refactor micro-cycles before writing implementation code, whereas Ousterhout worries that strict TDD encourages tactical, line-by-line hacking at the expense of upfront design thinking. Reading the two authors challenge each other over concrete code examples is a masterclass in software design trade-offs and well worth reading to form your own view.&lt;/p&gt;
&lt;h2 id="the-software-engineer-spectrum-speed-vs-accuracy"&gt;&lt;a href="https://benhowdle.im/software-engineer-spectrum" rel="noopener noreferrer"&gt;The Software Engineer Spectrum: Speed vs. Accuracy&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Drawing on 15 years of experience ranging from IC engineer to CTO, Ben Howdle observes that every software engineer sits somewhere along a spectrum between &lt;strong&gt;Speed&lt;/strong&gt; and &lt;strong&gt;Accuracy&lt;/strong&gt;. Speed-leaning engineers live by &amp;ldquo;ship now, refine later&amp;rdquo;—excelling at rapid iteration and getting MVPs in front of users, at the risk of accumulating technical debt and missing edge cases. Accuracy-leaning engineers live by &amp;ldquo;make sure it&amp;rsquo;s right before it goes live&amp;rdquo;—excelling at rock-solid reliability, edge-case coverage, and scalable architecture, at the risk of moving slowly and over-engineering. Neither end of the spectrum is inherently superior; what matters is alignment with the company&amp;rsquo;s current stage.&lt;/p&gt;
&lt;p&gt;Different business stages demand different balances: an early-stage seed startup needs Speed to validate product-market fit before the runway runs out; a scaling Series B company needs a balanced mix as testing and architecture start to matter; and a large enterprise or regulated fintech needs Accuracy where downtime or compliance failures are existential. The post offers self-reflection prompts—such as whether you get more frustrated by slow decisions or by shipping imperfect code—and advises engineering leaders to build balanced teams where both archetypes respect each other and can shift along the spectrum as the product matures.&lt;/p&gt;
&lt;h2 id="llm-ecosystem-predictions"&gt;&lt;a href="https://www.moderndescartes.com/essays/llm_predictions/" rel="noopener noreferrer"&gt;LLM Ecosystem Predictions&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Brian Kihoon Lee, an engineer at Databricks, lays out his predictions for the LLM ecosystem over the next 5–10 years. The cost of LLM inference at a fixed quality level is dropping roughly three times faster than Moore&amp;rsquo;s Law—halving every six months—driven by hardware advances, systems optimizations, and algorithmic breakthroughs; because all three favor capital-intensive hyperscalers, only a handful of frontier model providers will survive, competing at razor-thin margins. Just as high-level languages beat C and Java on developer iteration speed, simple agent architectures will beat complex, brittle orchestration frameworks: simply swapping in the latest frontier model every 6–12 months, backed by a solid evaluation suite, will beat months of custom prompt-chain tuning on both quality and cost.&lt;/p&gt;
&lt;p&gt;He also predicts the market will bifurcate between managed enterprise AI platforms and DIY tooling, with an &amp;ldquo;Excel for LLMs&amp;rdquo; emerging to let non-programmers build their own AI workflows. Cost per query will span orders of magnitude—from 100-million-parameter small models to 1-trillion-parameter frontier models—while the economic value unlocked will vary even more wildly: LLMs are poised to replace expensive human coordination overhead inside organizations, personalized tutoring, and customer support with 24/7 availability and consistent quality. By 2030, he argues, the world will look as different from today as the mobile and Internet era looked compared to the 1990s.&lt;/p&gt;
&lt;h2 id="the-right-kind-of-stubborn"&gt;&lt;a href="https://www.paulgraham.com/persistence.html" rel="noopener noreferrer"&gt;The Right Kind of Stubborn&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;Paul Graham distinguishes between two very different traits that often get lumped together: &lt;strong&gt;persistence&lt;/strong&gt; and &lt;strong&gt;obstinacy&lt;/strong&gt;. Both make people hard to stop, but in opposite ways: the persistent person is like a boat whose engine can&amp;rsquo;t be throttled back, while the obstinate person is like a boat whose rudder can&amp;rsquo;t be turned. Persistent people are fiercely attached to the &lt;em&gt;goal&lt;/em&gt; and happily change tactics when something isn&amp;rsquo;t working—in fact, they listen intently to criticism because they want to know if their current approach is flawed; obstinate people are attached to their &lt;em&gt;initial plan&lt;/em&gt;—which is almost always the least-informed version of the idea—and reflexively dismiss feedback. Obstinacy only works on simple problems where the first idea happens to be right by luck.&lt;/p&gt;
&lt;p&gt;True persistence, Graham argues, has a much richer internal structure built on five distinct qualities: &lt;strong&gt;energy&lt;/strong&gt; to keep trying, &lt;strong&gt;imagination&lt;/strong&gt; to invent new approaches when blocked, &lt;strong&gt;resilience&lt;/strong&gt; to keep morale intact after setbacks, &lt;strong&gt;good judgment&lt;/strong&gt; to weigh expected value, and a &lt;strong&gt;specific focus&lt;/strong&gt; on a concrete problem. For less experienced founders and makers, his advice is to start with a small, well-defined goal and expand your ambition as you gain traction. Because it requires all five ingredients at once, the right kind of stubbornness is much rarer than mere obstinacy—and vastly more powerful.&lt;/p&gt;
&lt;h2 id="bonus-interesting-images-from-bytebytego"&gt;Bonus: Interesting images from &lt;a href="https://bytebytego.com/" rel="noopener noreferrer"&gt;ByteByteGo&lt;/a&gt;&lt;/h2&gt;&lt;p&gt;&lt;img src="https://substack-post-media.s3.amazonaws.com/public/images/ae95d38c-41f8-4eb7-885e-f7fafa4ca45d_2250x2624.png" alt="Data Consistency Strategies For Microservices"&gt;&lt;/p&gt;
&lt;hr&gt;
&lt;p&gt;&lt;em&gt;This post was rewritten by Claude Code with Opus 5.5 on 27/09/2026; reviewed and edited by Codex with GPT 6.1 Sol on 09/11/2026, and re-verified by Claude Code with Sonnet 5.5 on 16/11/2026.&lt;/em&gt;&lt;/p&gt;</description></item></channel></rss>