That Little Square Does a Lot: How QR Codes Actually Work

You've scanned hundreds of them — restaurant menus, bus tickets, payment counters, event passes, Instagram profiles.

You point your phone. Wait half a second. Something happens.

But what is that little square, really? And how does pointing a camera at a printed sticker somehow move money between two bank accounts in under a second?

It turns out this is one of the most elegant pieces of everyday engineering. Once you see how it works, you can't unsee it.

TL;DR

A QR code is just text — encoded as a grid of black-and-white squares. Your camera decodes that grid back into text using math. For payments, that text is a special address that tells your UPI app who to pay and how much. Then five institutions — your bank, NPCI, the merchant's bank, and a couple of others — silently coordinate a real-time transfer in milliseconds. The Soundbox speaks because it keeps a permanent cloud connection open, waiting for exactly that moment.

Chapter 1: What Even Is a QR Code?

Let's start with the simplest possible explanation.

A QR code is just text, printed as a picture.

That's it. Everything else is engineering detail on top of that single idea.

Instead of writing upi://pay?pa=chaiwala@upi&am=50, someone converts that string into a grid of black and white squares. When you point a camera at that grid, the phone runs some math and gets the text back out.

Every zone has a job: corner squares for orientation, tiny dots for grid calibration, everything else is data. Source: Wikimedia Commons (CC BY-SA 3.0)

Why squares? Why not just a regular barcode?

A regular barcode — the zebra-striped one on a cereal box — only stores data in one direction, horizontally. It's like a single line of text. That limits it to maybe 20-30 numbers.

A QR code uses both dimensions. Horizontal and vertical. Think of it as a page of text versus a single line. Same physical space, but now you can pack in 4,000+ characters.

This is what made QR codes revolutionary when Denso Wave invented them in 1994 — originally just to track car parts in a Toyota factory. Nobody imagined they'd end up on every restaurant table on Earth.

Chapter 2: The Anatomy — What Are All Those Squares For?

Look closely at any QR code. You'll notice it's not random noise. It has a very specific structure.

It's not random noise — every region is precisely defined by the ISO standard. Source: Wikimedia Commons (CC BY-SA 3.0)

The three big squares in the corners

Those three bold nested squares — top-left, top-right, bottom-left — are not data. They're navigation anchors.

Your phone's camera finds these three squares first, before reading anything else. Once it locates them, it instantly knows:

• Where the QR code starts and ends
• What angle it's tilted at
• How large each data "cell" is
• Whether the image is distorted or curved

This is why you can scan a QR code:

• Sideways
• At an angle
• Upside down
• Printed on a curved bottle
• Slightly crumpled

The three anchor squares let the software mathematically "flatten" the image before reading the data cells.

The tiny dots in the middle — the data

Everything between and around the anchors is actual data: the encoded text, plus extra redundancy bits for error correction.

The quiet zone

Notice how QR codes always have a white border around them? That blank space isn't wasted — it's mandatory. It tells the scanner "the code starts here." Without it, the camera can't distinguish the code from whatever is printed around it.

Chapter 3: What Happens When You Scan?

Here's the sequence that happens the moment you point your camera at a QR code:

1. The camera captures a frame and converts it to black-and-white. No colour needed. Just dark and light.

2. Software scans for the three corner anchors. It's looking for the specific pattern of that nested square. Once it finds three of them, it locks on.

3. It maps the grid. Using the anchor positions and the thin "timing strips" between them (those alternating rows of dots), the software figures out where every data cell is and reads each one as a 0 or 1.

4. It reverses a mathematical mask. During QR code creation, the data gets scrambled with an XOR mask to prevent large uniform patches of black or white (which confuse cameras). The scanner reverses this.

5. Error correction runs. If a few cells were smudged or misread, a Reed-Solomon algorithm reconstructs the original data. This is the same math NASA used for deep-space probes and CDs use for skipping prevention.

6. Output: plain text. The whole thing — from camera frame to decoded string — takes milliseconds.

Chapter 4: The Error Correction Magic

This part deserves its own section because it's genuinely remarkable.

A QR code can be physically damaged — torn, smudged, covered by a logo — and still decode perfectly.

Wikipedia's own QR code has its logo sitting right in the middle — covering real data cells. It still scans perfectly because error correction fills in what's missing. Source: Wikimedia Commons (public domain)

There are four levels of error correction you can choose when generating a QR code:

Payment QR codes often use level H — so even if 30% of the squares are destroyed, the data is fully recoverable.

The math behind this (Reed-Solomon codes) operates on something called a Galois Field — a finite number system where all arithmetic "wraps around." It's the same fundamental idea used in RAID storage arrays and satellite communications. The QR code on your chai stall's counter uses mathematics built for outer space.

Chapter 5: Static vs Dynamic — Two Very Different Beasts

Not all payment QR codes work the same way. There are two fundamentally different types.

The printed sticker on the counter — Static QR

That laminated card at your local vegetable vendor? That's a static QR code.

It was generated once during merchant onboarding, printed, and it never changes. It only stores the merchant's UPI address — something like vendor@oksbi. When you scan it, your UPI app opens and asks you to type the amount yourself.

The good: It's cheap to produce, requires no internet connection to display, and lasts indefinitely.

The problem: You type the amount. You might type ₹100 when you owe ₹1,000. The merchant has no way to pre-confirm the correct amount. Reconciling which payment matched which order is a manual headache.

The screen at a restaurant — Dynamic QR

At a modern restaurant, grocery store, or any POS system, the QR code is generated fresh for every single transaction.

You scan it and your UPI app already shows: "Pay ₹847 to Domino's Koramangala." Pre-filled. No manual entry. One fingerprint, PIN, done.

This QR is generated live by the payment aggregator's backend (PhonePe Business, Razorpay, Paytm for Business, etc.) and encodes:

• Merchant's UPI address
• Exact transaction amount
• A unique order reference ID
• An expiry timestamp (usually 10-15 minutes)

Scan a day-old dynamic QR code and it rejects. This is intentional — expired codes can't be replayed or reused by anyone.

Chapter 6: The Payment Journey — Half a Second, Five Institutions

Here's the part most people never think about: what happens after you tap "Pay."

From this moment to the Soundbox speaking — five organisations talk to each other in under a second.

The chain of events

Your phone reads the QR code and finds this text:

upi://pay?pa=merchant@bank&pn=Store Name&am=150.00&tr=ORD123456

Your phone sees upi:// and immediately knows — same way mailto: opens your email app — to launch your UPI payment app. This is called deep linking.

You enter your PIN. But it never travels in plain text.

Every UPI app — Google Pay, PhonePe, Paytm — embeds a locked-down cryptographic module called the NPCI Common Library. It's a piece of isolated code that every UPI app is legally required to include.

When you type your PIN, it goes into this vault:

• Mixed with device-specific values
• Hashed (converted into an irreversible fingerprint)
• Wrapped in multiple layers of encryption

Think of it this way: your PIN gets locked inside a box. That box gets locked inside another box. The second box's key is held by your bank — and only your bank.

PhonePe never sees your PIN. NPCI never sees your PIN. Nobody in the middle can.

The encrypted package travels a fixed route:

Your phone
  → PhonePe's servers
    → Your bank (the Payer PSP)
      → NPCI UPI Switch
        → Merchant's bank (the Payee PSP)

Each hop verifies the cryptographic integrity before passing it along. Nobody can tamper with the amount or the recipient mid-flight.

NPCI is the air traffic controller.

The NPCI Switch sits at the center. It:

• Decrypts the outer routing layer
• Looks up what bank account merchant@bank belongs to
• Forwards the PIN payload only to your bank (which holds the only key)
• Waits for your bank to confirm the debit
• Tells the merchant's bank to credit the account
• Broadcasts SUCCESS to everyone

Your bank is the only entity that can decrypt your PIN and verify it. If it matches and balance is sufficient, the debit happens. Signal flows back up the chain.

Total time: typically 200–800 milliseconds.

Chapter 7: Why Does the Soundbox Speak Instantly?

This is the detail that surprises most people.

The Soundbox at the counter isn't checking its inbox every second. It isn't waiting for an SMS. It maintains a permanent open connection to the payment aggregator's cloud servers — like a phone call that's always on hold, never hanging up.

The protocol used for this is called MQTT (Message Queuing Telemetry Transport) — originally designed for oil pipeline sensors in remote locations where bandwidth is expensive. It's extremely lightweight, works over 2G/3G, and keeps the connection alive with tiny "are you there?" heartbeat packets.

The moment NPCI confirms the credit:

1. PhonePe's backend fires a notification to the Soundbox's cloud channel
2. MQTT delivers it in milliseconds over the persistent connection
3. The Soundbox lights its green LED
4. It plays audio assembled from pre-stored clips:

[jingle] + ["ek sau pachaas"] + ["rupaye"] + ["praapt hue"]

That slightly robotic quality in the voice? That's because it's stitching together pre-recorded number clips, not synthesising speech from scratch.

Chapter 8: What Stops Someone from Making a Fake QR Code?

This is a real attack. It's called "QR code overlay fraud" — someone prints a QR code pointing to their own account and pastes it over the merchant's legitimate code.

You scan it. Looks identical. Money goes to the wrong person.

That sticker on the counter could, in theory, be replaced by a fraudster's code. Cryptographic signatures make this attack detectable.

To combat this, UPI payment QR codes include a cryptographic signature — a mathematical fingerprint generated using the merchant's private encryption key.

When your UPI app scans a payment code:

1. It reads the full URL
2. It verifies the signature against NPCI's public key registry
3. If the signature doesn't match — even one character was changed — the transaction is blocked immediately

A fraudster can copy a QR code and print it. But they cannot fake the cryptographic signature without the merchant's private key. The math makes forgery detectable.

The Bigger Picture

What began as a Toyota factory tool for tracking car components in 1994 is now the physical-to-digital gateway for one of the world's largest payment networks — handling half a billion users and billions of monthly transactions.

Every time you scan and pay, you're triggering:

• A camera doing real-time computer vision
• Reed-Solomon error correction reconstructing possibly-damaged data
• Deep linking routing your OS to the right app
• A cryptographic vault protecting your PIN from everyone, including the app itself
• Five institutions coordinating an inter-bank transfer in milliseconds
• An IoT device playing spliced audio over a connection that never closes

All in under a second. From a little black-and-white square on a laminated card.

Further Reading

If this sparked curiosity, here's where to go next:

On QR Codes:

• QR Code — Wikipedia — The full history, Denso Wave origins, and technical evolution
• Reed–Solomon error correction — Wikipedia — The math that makes smudged codes readable
• Reed–Solomon codes for coders — Wikiversity — Hands-on if you want to implement it
• ISO/IEC 18004 standard overview — ANSI Blog — The formal spec in readable form

On UPI & Payments:

• Unified Payments Interface — Wikipedia — The full architecture and history
• How UPI powers Scan & Pay — Paytm Blog — The four-party model explained
• Security Analysis of UPI and Payment Apps in India — USENIX — Academic deep dive into UPI security
• UPI Common Library Specification — Razorpay Docs — How the PIN encryption actually works

On the IoT side:

• How Payment Soundbox Works — EazyPay Tech — Full breakdown of the Soundbox hardware and pipeline
• MQTT Protocol — The lightweight protocol that connects Soundboxes to the cloud

Go deeper into system design:

• The System Design Behind PhonePe's UPI Engine — Database sharding, consistent hashing, Raft consensus

The next time the Soundbox speaks, you'll know exactly what just happened — and how many pieces had to work perfectly for that half-second to feel effortless.

Go 1.26.2 was released on 2026-04-07 as a patch release focused on security and stability.

If you are running Go 1.26.x in production, this is the kind of release you should evaluate quickly.

Go 1.26.2 includes security fixes to the go command, compiler, and the archive/tar, crypto/tls, crypto/x509, html/template, and os packages, plus bug fixes across the go command, go fix, compiler, linker, runtime, net, net/http, and net/url.

TL;DR (beginner + technical)

• If you are on 1.26.0 or 1.26.1, plan an upgrade to 1.26.2.
• If your service is internet-facing, treat this as high priority because security-sensitive packages were patched.
• Verify checksums before installation.
• Roll out with canary + SLO guardrails, not a full one-shot deployment.

If you are new: what does "1.26.2" mean?

Go versions follow a major.minor.patch pattern:

• 1 = major line
• 26 = minor release line
• 2 = patch release

Patch releases are usually about fixing behavior, not adding new language features.

Why this patch matters technically

Go 1.26 introduced important runtime and toolchain changes. Point releases like 1.26.2 are where real-world regressions and security backports get addressed.

Use this release as both a security update and a reliability update.

Deep look at the Go1.26.2 milestone

Using the public GitHub milestone API for Go1.26.2:

• Milestone state: closed
• Open issues: 0
• Closed issues: 31

What the issue distribution tells us

From a title/label scan of closed issues, the cluster is practical and production-facing:

• CVE-related titles: 10
• Security backport-labeled titles: 8
• Compiler/runtime-labeled items: 10
• cmd/go-labeled items: 1
• Testing-labeled items: 2
• Documentation/backport consistency items: 4

This is the normal fingerprint of a mature patch release: security + high-impact regressions + release-quality cleanup.

Security and CVE signal

The milestone shows clear CVE and security backport activity, including:

• #78428 security: fix CVE-2026-32283 [1.26 backport]
• #78426 security: fix CVE-2026-32282 [1.26 backport]
• #78424 security: fix CVE-2026-27144 [1.26 backport]
• #78422 security: fix CVE-2026-27140 [1.26 backport]
• #78362 crypto/x509 ... (CVE-2026-32280)
• #78360 crypto/x509 ... (CVE-2026-32281)

Even without changing your app code, patching core trust-boundary packages (tls, x509, templates, and toolchain) reduces real production risk.

Regression fixes you may actually notice

Representative examples from the milestone:

• #78058: cmd/go cache trim could block for 20+ minutes on macOS.
• #78111: net/url parsing regression affecting MongoDB multi-host connection strings.
• #78041: runtime crash on Windows in 1.26.0/1.26.1.
• #78239: linker panic on darwin/arm64.
• #78191: cmd/fix panic in edge cases.

Beginner translation: these are exactly the kinds of bugs that become random CI failures, odd runtime crashes, or flaky production behavior.

Download artifacts and checksum verification

Official download page: go.dev/dl

Selected Go 1.26.2 artifacts from the official download feed:

Why checksum verification matters: it confirms the artifact you downloaded is exactly what Go published.

Example verification (macOS/Linux shell):

curl -LO https://go.dev/dl/go1.26.2.linux-amd64.tar.gz
shasum -a 256 go1.26.2.linux-amd64.tar.gz
# expected: 990e6b4bbba816dc3ee129eaeaf4b42f17c2800b88a2166c265ac1a200262282

Production-safe upgrade playbook

If your team has no formal release process yet, just follow the steps in order and keep a rollback version ready.

Use Go 1.26.2 explicitly in CI images, local dev environments, and build containers. Avoid implicit latest tags.

```dockerfile
FROM golang:1.26.2-alpine
```


Validate what your runner is actually using before tests.

```bash
go version
go env GOTOOLCHAIN GOOS GOARCH
```


Start with your existing suite, then add race detection for services that handle concurrency heavily.

```bash
go test ./...
go test ./... -race
go vet ./...
```


Execute `govulncheck` and compare findings against your previous baseline.

```bash
go install golang.org/x/vuln/cmd/govulncheck@latest
govulncheck ./...
```


Add focused tests for URL parsing, HTTP handling, TLS handshakes, and cert verification in critical flows.


Canary first, then staged rollout with explicit SLO gates (error rate, latency, crash loops, build stability).


Preserve the previous working toolchain image/tag so rollback is a fast switch, not a rebuild.

Copy-paste CI gate (practical baseline)

set -euo pipefail

go version
go env GOTOOLCHAIN GOOS GOARCH

go test ./...
go test ./... -race
go vet ./...

govulncheck ./...

Should you upgrade now?

• Upgrade now (high priority): internet-facing services, heavy TLS/cert usage, or strict security posture.
• Upgrade soon (scheduled): internal services on 1.26.0/1.26.1.
• Plan migration path: older major line users that need broader compatibility testing.

Read more

The canonical summary of packages touched in 1.26.2.

Understand what 1.26 introduced so you can interpret which 1.26.2 fixes are stabilization backports.

Official binaries, source tarball, and checksums for every platform.

Closed issue set showing exactly what was backported into the patch.

Guidance on govulncheck, vulnerability database, and release security policy.

Data provenance

This article and its visuals are based on:

1. Official Go release notes and release history.
2. Official Go download index and checksums.
3. Public GitHub milestone API data for Go1.26.2.

The embedded images in this post are custom summary visuals created from those public sources.

cursor-claude-personas: Give Your AI Coding Assistant a Domain Expert Brain in 30 Seconds

TL;DR — cursor-claude-personas is a free, open-source collection of 38 role-based AI persona packs for Claude Code, Cursor, and VS Code. Copy one folder into your project, reopen your editor, and your AI assistant instantly behaves like a seasoned specialist — whether that's a senior Python developer, a DevOps engineer, a security expert, or a UI/UX designer.

The Problem: Generic AI Gives Generic Code

You have Claude Code or Cursor open. You ask it to add an endpoint. It writes something that works — but it doesn't know your preferred framework patterns, it doesn't follow your domain conventions, and it certainly doesn't remember that you care deeply about SQL index hygiene or GraphQL schema design.

The usual fix is a long system prompt you paste in every time. Or a .cursorrules file you spend hours writing. Or endless back-and-forth corrections.

cursor-claude-personas solves this with a dead-simple copy-paste.

What Is cursor-claude-personas?

cursor-claude-personas is a GitHub repository containing 38 ready-made persona packs for AI coding tools. Each persona is a folder with two hidden subdirectories:

senior-python-developer/
├── .claude/
│   ├── rules/       ← always-on instructions Claude Code reads at session start
│   └── skills/      ← auto-triggered skill docs for specific tasks
└── .cursor/
    ├── rules/       ← always-on instructions Cursor reads at session start
    └── skills/      ← auto-triggered skill docs for specific tasks

When you copy .claude/ or .cursor/ (or both) into the root of your project, your editor loads those configs automatically. No slash commands. No system prompts. The persona is live the moment you start a new chat session.

Supported tools

How to Use It — Step by Step

Step 1: Clone the repo

git clone https://github.com/ratnesh-maurya/cursor-claude-personas.git

You only need to clone it once. Keep it around as your personal AI persona library.

Step 2: Pick the persona that matches your current work

Browse the table in the next section or visit the GitHub Pages site to filter by technology or tag.

Step 3: Copy the persona into your project

Navigate to your cloned copy and run:

# Claude Code only
cp -R cursor-claude-personas/senior-python-developer/.claude /path/to/your-project/

# Cursor only
cp -R cursor-claude-personas/senior-python-developer/.cursor /path/to/your-project/

# Both tools
cp -R cursor-claude-personas/senior-python-developer/.claude /path/to/your-project/
cp -R cursor-claude-personas/senior-python-developer/.cursor /path/to/your-project/

Replace senior-python-developer with any persona slug from the list below.

Step 4: Reopen your editor or start a new session

Claude Code and Cursor reload configuration at session start. From that point on, the AI follows the persona's rules, applies its skills automatically, and behaves as a domain expert — without you typing a single extra word.

Bonus: Enable the auto-skill router

Each persona includes a special rule file called rules--auto-skill-router.mdc (.mdc is the Cursor/Claude rule file format — it is just Markdown with optional YAML frontmatter). This rule tells the AI to automatically select the right skill doc based on what you're working on — so you never need to type /skill commands manually.

It is already included when you copy the persona folders. No extra steps required.

All 38 Personas

Not sure where to start? Try senior-python-developer or senior-frontend-developer — both are polished starting points that work for a wide range of projects.

What Makes This Different From a .cursorrules File?

A .cursorrules file (a single plain-text file at the project root) you write yourself usually contains two or three paragraphs of general instructions. Cursor also supports a .cursor/rules/ directory of individual rule files for more granular control. Both are fine for broad guidance, but neither gives your AI assistant real domain depth out of the box.

cursor-claude-personas packs in:

• Multiple focused rule files — each covering a specific concern (schema design, security, pagination, query patterns, etc.) rather than one sprawling blob of text.
• Skill documents — structured reference files that are automatically surfaced when the AI detects it needs them, without you prompting it.
• Both tool formats simultaneously — the same expertise loads whether you are using Claude Code, Cursor, or VS Code Copilot.
• Zero prompt engineering on your part — you copy two folders and you are done.

The key design principle is depth over breadth: one sharp specialist beats five vague generalists.

Real-World Example

Imagine you are building a Python REST API with PostgreSQL. You copy in the senior-python-developer persona. From that point on, when you write a query, the AI:

• Warns you against WHERE id IN (SELECT ...) and rewrites it as WHERE id = ANY(ARRAY[...]) with a note about the 10× speedup.
• Adds WHERE deleted_at IS NULL partial indexes automatically when it sees soft-delete columns.
• Flags missing foreign key indexes before you even run EXPLAIN ANALYZE.
• Enforces proper connection pool limits and prepared statement patterns.

None of that required you to type a single instruction. It was all encoded in the rules and skills you copied in.

How the Folder Structure Works Under the Hood

Each persona ships with two kinds of config files:

Rules (rules/)

Rules are always-on markdown files the editor reads at the start of every session. They define the persona's defaults, code style, anti-patterns to avoid, and domain-specific best practices. They are named with a rules-- prefix so they sort together and are easy to identify.

Skills (skills/)

Skill documents are structured markdown files with YAML frontmatter. The auto-skill router rule tells the AI to load the relevant skill when it detects you are working on a related task. For example, if you start writing an index migration, the SQL indexing skill is surfaced automatically. If you start wiring up an authentication flow, the auth skill fires.

This means the AI pulls in deep, specific guidance exactly when it is relevant — not all at once, cluttering its context.

Switching Personas Mid-Project

You can swap personas at any time. Just delete the existing .claude/ and .cursor/ folders in your project and copy in a different persona. Reopen your session and the new persona is live.

You can even mix rules from multiple personas manually if you know what you are doing, but most of the time a single focused persona is all you need.

Contributing a New Persona

The project is open source under the MIT license and welcomes contributions. If you want to add a new persona:

1. Fork the repository.
2. Create a new folder named after the role (e.g., data-science-researcher).
3. Add matching .claude/ and .cursor/ subdirectories with rules and skills.
4. Make sure the two directories are in sync — the parity check in CONTRIBUTING.md validates this.
5. Open a pull request.

See CONTRIBUTING.md for branch naming conventions, commit message format, and the full persona checklist.

Where to Get It

• GitHub repository: https://github.com/ratnesh-maurya/cursor-claude-personas
• GitHub Pages site (browse and filter personas): https://ratnesh-maurya.github.io/cursor-claude-personas/
• License: MIT — free for personal and commercial use.

If it saves you time, star the repo. It helps other developers discover it.

Summary

Clone it, copy a persona, and ship better code.

Five Caching Strategies Every Backend Dev Should Know

If you build backend systems, you eventually run into caching.

Someone says “just put Redis in front of it” and suddenly your app is faster… until you start seeing stale data, cache misses, and weird bugs when you deploy.

This post explains five core caching strategies in plain language:

• Cache-Aside
• Read-Through
• Write-Through
• Write-Back
• Write-Around

You’ll see how each one works, what can go wrong, and when to use it.

A cache is just a faster, smaller copy of some data. The hard part is keeping that copy useful: when to fill it, when to read from it, and when to update or skip it.

1. Cache-Aside (Lazy Loading)

Idea: The application talks to the cache and the database. It “checks the cache first, then falls back to the DB” and fills the cache on a miss.

If the key is in the cache (a hit), just return it. Super fast.


If the key is missing, the app queries the database instead.


The app stores the result in the cache and returns it to the user.

Why people love it

• Simple and flexible: logic lives in your code; you control what goes in the cache.
• Resilient: if the cache dies, the app can still read from the DB (just slower).
• Memory-efficient: only data that is actually read gets cached.

What can go wrong

• The first request for a key is slow (cold cache).
• If the DB is updated outside your code path (e.g. admin tool, another service), the cache can serve stale data until TTL expires or you manually invalidate.

When to use

• Great default for read-heavy apps (blogs, product catalogs, many dashboards).
• When you’re just adding caching to an existing codebase and want full control.

2. Read-Through

Idea: The application pretends the cache is the main database.

It calls the cache for every read. On a miss, the cache layer itself fetches from the DB, stores the result, and returns it.

With Cache-Aside, your app code calls both cache and DB. With Read-Through, your app only calls the cache; a loader behind the cache talks to the DB.

Pros

• Cleaner application code: your code only knows “get from cache”.
• Cache libraries or infrastructure can handle loading, retries, and metrics.

Cons

• If some writes go directly to the DB (bypassing the cache API), the cache can easily become stale.
• You need a cache provider or library that supports read-through loading.

When to use

• Larger teams where you want centralized cache logic, not ad-hoc caching scattered everywhere.
• Situations where infra/platform team owns the cache layer.

3. Write-Through

So far we talked about reads. Now let’s talk about writes: how we keep cache and DB in sync.

Write-Through means:

On a write, update the cache and the database synchronously. Only when both succeed is the write considered “done”.

Flow

1. User updates something (e.g. changes quantity in a cart).
2. App writes the new value into the cache.
3. App also writes the same value into the database.
4. Only if both succeed do we return “OK” to the user.

Pros

• Strong consistency: cache and DB always match for that key.
• Reads from the cache always see the latest value.

Cons

• Slower writes: every write has to hit both cache and DB before returning.
• If DB is slow, your API write latency is also slow.

When to use

• When correctness matters more than raw speed:
  • balances in a wallet / bank,
  • inventory in an e‑commerce site,
  • data where a stale read would be really bad.

4. Write-Back (Write-Behind)

Write-Back trades strict consistency for speed.

On a write, update only the cache and return success quickly. Later, an async process flushes those changes to the database.

Flow

1. User sends a write.
2. App writes to cache and returns 200 OK immediately.
3. A background worker or the cache itself periodically writes batched updates into the DB.

Pros

• Very fast writes: app isn’t blocked waiting for DB.
• Great for high-throughput workloads:
  • counters (views, likes),
  • logs / telemetry,
  • social feed events.

Cons

• Risk of data loss: if the cache crashes before flushing to DB, those changes are gone.
• DB is eventually updated, not immediately.

When to use

• Write-heavy workloads where losing a tiny fraction of data is acceptable:
  • metrics, dashboards, click-streams,
  • non-critical logs.

5. Write-Around

Write-Around takes a different approach:

Writes skip the cache and go straight to the DB. The cache is only filled later when data is read.

Flow

1. A write goes directly into the database.
2. The cache is not updated.
3. On the next read, the app (or cache) sees a miss:
   • loads from DB,
   • populates the cache,
   • and returns the value.

Pros

• Prevents “polluting” the cache with data that is written but rarely read.
• Good when you have big write batches (imports, ETL jobs) that users won’t read immediately.

Cons

• The first read after a write is always a cache miss (slower).
• More frequent DB reads if many keys are only read once.

When to use

• Large imports, archival data, logs that might be read only occasionally.
• Systems where memory is tight and you want the cache to focus on truly hot data.

Side-by-side comparison

Here’s a quick comparison of these five strategies using the labels you provided.

How teams typically choose

To connect this with the interactive content you built:

These charts show two things:

• Cache-Aside dominates in real systems because it’s simple and resilient.
• Write-Back gives the fastest writes but carries the most risk; Write-Through is safest but slower on each write.

Which caching strategy should you pick? (a practical rule-set)

If you only remember one thing from this post, remember this: you pick a caching strategy based on the cost of a stale read and the cost of a slow write.

Here’s a fast, real-world decision guide:

• Most read-heavy apps (catalogs, blogs, dashboards):

  • Start with Cache-Aside.
  • Add TTLs and a plan for invalidation if data changes.

• You want a clean abstraction and centralized caching (platform/infra owned):

  • Use Read-Through so product code only depends on the cache API.

• A stale read would be painful (wallet balances, inventory, “did my payment go through?”):

  • Prefer Write-Through (consistency > speed).

• Writes are hot and you can tolerate some eventual consistency (counters, analytics, clickstream):

  • Consider Write-Back, but only with durability guardrails (below).

• You write lots of data that is rarely read (imports, ETL, logs):

  • Use Write-Around to avoid polluting the cache.

Two failure modes teams hit in production (and what to do)

Caching rarely fails in obvious ways. It fails with “the DB is melting” or “why is data stale?”. Two patterns show up everywhere.

1) Cache stampede (thundering herd)

Symptom: a hot key expires (or cache restarts), and suddenly thousands of requests miss at once and slam the DB.

Mitigations:

• Request coalescing / single-flight: only one request recomputes the value; others wait and reuse it.
• Stale-while-revalidate: keep serving a slightly stale value while one background refresh updates it.
• Jittered TTLs: add randomization to expiration times so many keys don’t expire at the same second.

2) Hot key / hotspotting

Symptom: one key becomes so popular (home feed, trending item, auth config) that a single cache shard or a single DB row gets hammered.

Mitigations:

• Shard the key (e.g. trending:v1:0..N) and merge results.
• Use a two-layer cache: per-instance in-memory (tiny TTL) + shared Redis.
• Precompute or cache at a higher level (edge/CDN) if the data is public.

What to monitor so caching doesn’t become “magic”

Whatever strategy you choose, watch these:

• Cache hit rate (overall and by endpoint)
• DB QPS (does caching actually reduce it?)
• p95/p99 latency (cache helps only if tail latency improves)
• Eviction rate / memory pressure (are you constantly evicting hot data?)
• Stale reads / correctness signals (app-specific: inventory mismatches, stale dashboards, etc.)

If you can’t measure these, caching will eventually surprise you.

Putting it all together (for a newbie)

If you’re just starting with caching:

1. Start with Cache-Aside for reads. It’s simple, safe, and easy to reason about.

2. Add Read-Through if you want a cleaner abstraction and your cache provider supports it.

3. For writes:

   • Use Write-Through when you absolutely need the cache and DB to agree on every write.
   • Use Write-Back when you care a lot about write speed and can tolerate a little risk.
   • Use Write-Around when you write lots of data that users won’t read immediately.

4. Whatever you choose, always:

   • Set reasonable TTLs,
   • Have a plan for invalidating keys,
   • And monitor hit rate and latency so you can adjust over time.

With just these five strategies, you can handle most caching problems you’ll run into as a backend engineer.

Chrome 144 Introduces the <geolocation> Element: Declarative Location Access for Modern Web Apps

Chrome 144 introduces a new declarative <geolocation> HTML element that changes how web applications request location access. Instead of calling navigator.geolocation.getCurrentPosition() imperatively from JavaScript, developers can now place a browser-controlled element directly in the DOM.

This post is for frontend engineers, platform engineers, and privacy-conscious developers who want to understand how this new model works and whether they should migrate.

By the end of this article, you'll understand:

• What the <geolocation> element is
• How it works internally
• How it compares to the traditional Geolocation API
• How to implement it with proper fallback support

Why Chrome Introduced <geolocation>

Historically, location access was triggered via JavaScript:

navigator.geolocation.getCurrentPosition(success, error);

This caused several problems:

• Permission prompts appearing on page load
• Users denying requests due to poor timing
• Quiet permission blocking by browsers
• Complicated permission state handling in code

The <geolocation> element solves this by:

• Requiring explicit user interaction
• Providing a visible, browser-controlled UI
• Reducing boilerplate permission code
• Improving recovery from previously denied states

Chrome origin trials showed measurable improvements in successful permission grants and recovery rates.

What Is the <geolocation> Element?

The <geolocation> element is part of the WICG "Permission Elements" proposal. It acts as a declarative wrapper around the existing Geolocation API.

It renders as a browser-controlled button such as:

When clicked:

1. The browser shows the location permission dialog.
2. If granted, it dispatches a location event.
3. The element exposes either:
   • position (success)
   • error (failure)

Attributes Explained

autolocate

Automatically attempts to retrieve location when inserted into the DOM — only if permission was already granted.

<geolocation autolocate></geolocation>

It does not trigger surprise permission prompts.

watch

Continuously tracks position updates, similar to watchPosition().

<geolocation watch></geolocation>

Without watch, it behaves like getCurrentPosition().

accuracymode

Controls precision level.

<geolocation accuracymode="precise"></geolocation>

Possible values:

• "approximate" (default)
• "precise"

When set to "precise", the browser may:

• Change icon (crosshair instead of pin)
• Update label text
• Request higher accuracy internally

Basic Implementation

Minimal example:

<geolocation onlocation="handleLocation(event)"></geolocation>

<script>
function handleLocation(event) {
  if (event.target.position) {
    const { latitude, longitude } = event.target.position.coords;
    console.log("Location:", latitude, longitude);
  } else if (event.target.error) {
    console.error("Error:", event.target.error.message);
  }
}
</script>

No permission logic required.

The browser handles:

• Permission prompts
• Recovery dialogs
• Error states

Adding Progressive Enhancement (Fallback)

You must support non-Chrome browsers.

<geolocation id="geo">
  <button id="fallback-btn">
    Use my location
  </button>
</geolocation>

<script>
if ('HTMLGeolocationElement' in window) {
  const geo = document.getElementById('geo');
  geo.addEventListener('location', () => {
    if (geo.position) {
      console.log(geo.position.coords);
    }
  });
} else {
  document.getElementById('fallback-btn')
    .addEventListener('click', () => {
      navigator.geolocation.getCurrentPosition((pos) => {
        console.log(pos.coords);
      });
    });
}
</script>

This ensures compatibility across browsers.

Browser UI Behavior

When clicked, Chrome displays the standard location permission dialog:

If previously denied, Chrome shows a recovery prompt:

This makes permission recovery significantly easier compared to manual browser settings navigation.

Styling Restrictions (Security Guardrails)

To prevent abuse or clickjacking, Chrome enforces:

• Minimum size
• Opacity must remain 1
• No negative margins
• No deceptive transforms
• Enforced contrast ratio

You can style it, but you cannot make it invisible or misleading.

Example:

geolocation {
  border-radius: 8px;
  padding: 12px;
}

But you cannot:

• Hide it
• Make it 1px wide
• Overlay fake UI elements

<geolocation> vs navigator.geolocation

When Should You Use It?

Use <geolocation> if:

• You need a simple "Use my location" feature
• You want better permission recovery
• You want reduced code complexity
• You care about user trust and privacy

Stick to the traditional API if:

• You need fine-grained timing control
• You require silent background tracking
• You are building real-time geo-heavy apps (ride-sharing, logistics dashboards)

Production Checklist

• Feature-detect HTMLGeolocationElement
• Provide a JS fallback
• Avoid using autolocate unnecessarily
• Test denied → recovery flows
• Validate UX on mobile devices

Conclusion

The <geolocation> element is a significant shift toward declarative permission handling on the web. It reduces developer complexity while improving privacy, trust, and user experience.

For simple location-based features, it should become the default pattern in Chrome-supported environments.

However, since it's currently Chrome-specific and incubating, production systems must still include fallback support.

If you're building modern web applications and care about privacy-first UX, this is worth adopting early — but do it responsibly with progressive enhancement.

How Files Are Stored, Deleted, and Copied Inside Your Computer

Have you ever noticed something interesting?

• Deleting a file happens instantly.
• Copying a file takes time.
• Moving a file is sometimes instant and sometimes slow.

Why does this happen?

If you're new to operating systems and storage internals, this article explains everything in simple terms using real-world analogies, while also giving you a deeper technical understanding of what’s happening behind the scenes.

By the end, you’ll understand:

• How files are stored on disk
• Why delete is fast
• Why copy takes time
• Whether files are really deleted
• How HDD and SSD behave differently

How Files Are Actually Stored on Disk

Your computer does not store files the way you see them in folders.

Internally, storage is divided into:

1. Data Blocks (or clusters) – These contain the actual bytes of your file.
2. Metadata – Information about the file:
   • File name
   • File size
   • Location of data blocks
   • Permissions
   • Created/modified timestamps

A file system (like NTFS, ext4, FAT32) manages this structure.

Example File Systems

• Windows → NTFS
• Linux → ext4
• USB drives → FAT32

Each filesystem maintains:

• A directory entry (file name)
• A metadata record (inode in Linux, MFT entry in NTFS)
• Pointers to the actual disk blocks

You can think of it like a database that tracks where your file data lives on disk.

Real-Life Analogy: A Library

Imagine this:

• The library catalog = filesystem metadata
• The books on shelves = actual file data
• The shelf location number = pointer to disk blocks

When you open a file:

1. The system checks the catalog.
2. It finds the shelf location.
3. It retrieves the book (data blocks).

Now let’s see what happens during delete and copy.

Why Deleting a File Is Instant

When you delete a file, the system usually does not erase the data immediately.

Instead, it:

1. Removes the file name from the directory.
2. Marks its data blocks as “free”.
3. Updates metadata records.

That’s it.

The actual bytes are still physically on disk — but the system marks that space as available for reuse.

Since this is mainly a metadata update, it happens very fast.

Important Insight

Deleting is like removing the book’s entry from the library catalog.

The book is still on the shelf — but nobody knows it exists anymore.

This is why:

• File recovery tools can restore deleted files.
• Secure deletion requires special tools.

Do Files Actually Get Deleted?

Short answer: Not immediately.

They are:

• Marked as free space
• Eventually overwritten by new data

On SSDs:

• The operating system sends a TRIM command
• The SSD later clears unused blocks internally

Deletion is logical first, physical later.

Why Copying a File Takes Time

Copying is completely different.

When copying:

1. The system reads every block from the source file.
2. Loads it into memory (RAM).
3. Writes those blocks to a new location.
4. Creates new metadata for the copied file.

This process depends on:

• Disk speed
• File size
• HDD vs SSD
• CPU and memory performance

Since the entire file’s data must be read and written, copying takes time.

Example

Copying a 10GB file means:

• Reading 10GB
• Writing 10GB

That’s 20GB of total I/O operations.

That’s real physical work being done.

Why Moving Files Is Sometimes Instant

This depends on where you move the file.

Case 1: Same Drive (Same Filesystem)

Move = metadata update only.

The system just:

• Updates the directory entry
• Keeps data blocks unchanged

This is very fast.

Case 2: Different Drive

Move becomes:

• Copy the file
• Delete the original

So it takes time.

That’s why:

• Moving inside C: is instant
• Moving from C: to D: takes time

HDD vs SSD Behavior

HDD (Hard Disk Drive)

• Mechanical spinning disk
• Moving read/write head
• Slower seek time
• Sequential reads are faster

Copying large files can be slower because of mechanical movement.

SSD (Solid State Drive)

• No moving parts
• Much faster random access
• Uses flash memory cells
• Uses TRIM and garbage collection

Deletion marks blocks unused first. The physical erase happens later internally.

What Happens in the OS Layer

At the system-call level (Linux example):

• delete → unlink()
• copy → multiple read() and write() calls
• move (same filesystem) → rename()

Example:

strace rm file.txt

You'll see something like:

unlink("file.txt")

That's simply removing the reference to the file.

Copying involves many read() and write() operations, which take time.

Why Deleting Many Files Can Be Slow

Deleting one file is fast.

Deleting thousands of small files can take time because:

• Each file has metadata.
• Directory entries must be updated.
• Journaling systems must commit changes.

Even metadata operations add up when repeated many times.

Secure Deletion

Normal delete:

• Marks space as free

Secure delete:

• Overwrites data blocks
• Ensures recovery is impossible

On SSDs:

• Use drive-level secure erase
• Or enable full-disk encryption

Key Takeaways

• Delete is fast because it removes metadata, not actual data.
• Copy is slow because it transfers real bytes.
• Move is fast only within the same filesystem.
• Deleted files remain until overwritten.
• SSDs and HDDs behave differently.
• Secure deletion requires special handling.

What to explore next

File operations look simple from the UI, but under the surface the OS is managing metadata, block allocation, caching, journaling, and hardware coordination.

If this interested you, here are some rabbit holes worth going down:

• strace on Linux — run strace rm file.txt or strace cp src dst to see the actual system calls involved
• Filesystem internals — look into how ext4 journaling works, or how ZFS and Btrfs use copy-on-write to make snapshots nearly free
• Flash Translation Layer — how SSDs remap logical blocks to physical NAND pages, and why TRIM exists
• Data recovery tools — tools like testdisk and photorec exploit the fact that "deleted" data is still on disk

The next time a file deletes instantly, you'll know — it wasn't magic. It was just a metadata update.

The Mechanics of Compression: How 100GB Becomes 25GB

In an era of 4K streaming and massive cloud databases, we often take for granted that a 100GB backup can be shrunk down to 25GB. It isn’t magic—and it isn’t just “squeezing” bits. It’s a careful process of spotting patterns, removing redundancy, and using mathematical shorthand so that the same information takes less space. This post explains how that works in plain terms, so you can understand what happens under the hood when you zip a folder or stream a video.

1. The Core Idea: Redundancy Is Waste

At its heart, compression is about efficiency. Most data is repetitive. The same words, phrases, or pixel patterns show up again and again. If the computer has already seen a pattern once, it doesn’t need to store it again in full—it can refer back to it or describe it in fewer bits.

Think of a recipe that says “add salt” five times. You could write “add salt” five times, or you could write “add salt (×5).” The second version carries the same information in less space. Compression algorithms do something similar: they find repetition and describe it more compactly.

So the main job of any compression scheme is to find redundancy and encode it in fewer bits without losing the ability to recover the original data when needed.

A Real-Life Example: Compressing a Book with Repeated Words

Imagine a book where the same words appear again and again. Words like "the," "and," and "to" might show up thousands of times. Instead of writing the full word every time, we could agree on a legend at the start and replace each repeated word with a short sub-symbol. Anyone with the legend can reconstruct the original text exactly.

Suppose we define a small dictionary at the top of the page:

Legend:  @ = the   |   # = and   |   % = to   |   § = of

Now take a normal sentence and "compress" it by substituting:

Original (longer): "The king and the queen went to the castle and stayed there for the rest of the day."

Compressed (shorter): "@ king # @ queen went % @ castle # stayed there for @ rest § @ day."

We still have the same meaning, but we used one character (@, #, %, §) wherever a repeated word appeared. The legend is the "dictionary"—it has to be stored or sent once so the reader can decode. For a whole book, if "the" appears 2,000 times, we save (3 − 1) × 2,000 = 4,000 characters just on that one word. Add "and," "to," "of," and other frequent words, and the book shrinks noticeably without losing a single word.

This is exactly the idea behind dictionary coding and Huffman-style compression: find what repeats, assign it a short code, and use that code everywhere. Real compressors do this automatically; they don't need a human to write the legend.

A real experiment from the wild: Researchers applied this idea to the full text of "Alice's Adventures in Wonderland" by Lewis Carroll. They treated each word as a symbol, counted how often it appeared, and assigned shorter codes to the most frequent words (like "the," "and," "to," "a") and longer codes to rare words (like "wretched," "yawned"). The result: the text went from 164 KB down to 109 KB—about one-third smaller—with no information lost (Huffman-coding English words, Project Nayuki—includes original text from Project Gutenberg, input/output file sizes, and codebook samples). The same technique—replace repeated words with shorter symbols, and keep a mapping so we can decode—is what powers the compression we use every day in ZIP files and beyond.

2. Lossless vs. Lossy: Two Different Goals

Before diving into algorithms, it helps to know that compression splits into two families: lossless and lossy. The choice depends on whether you can afford to lose any information.

Lossless Compression: Every Bit Preserved

Lossless compression means that after you decompress, you get back exactly the original data—every bit. Nothing is discarded.

• How it works: The algorithm finds statistical redundancy (repeated or predictable patterns) and encodes them more efficiently. Decompression reverses the process and reconstructs the original stream.
• When to use it: Whenever you cannot afford to lose information—source code, text files, databases, executables, legal documents, medical data, or images where every pixel must stay exact (e.g. PNG for graphics, FLAC for audio).
• Common formats: ZIP, GZIP, PNG, GIF (for lossless use), FLAC.

So when you compress 100GB of source code or logs to 25GB with a lossless tool, you can later decompress and get back the same 100GB bit-for-bit.

Lossy Compression: Smaller Files, Imperceptible Loss

Lossy compression discards some information on purpose. What you get back is an approximation of the original—close enough for human perception, but not identical.

• How it works: The algorithm keeps what matters for the intended use (e.g. what the eye or ear notices) and drops “invisible” or less important detail—e.g. colors we can’t easily tell apart, or sounds masked by louder ones.
• When to use it: Media where small quality loss is acceptable—photos (JPEG), music (MP3, AAC), video (H.264, VP9). Streaming and storage would be far heavier without lossy compression.
• Common formats: JPEG, MP3, AAC, H.264, VP9.

For the rest of this post we focus on lossless compression—the kind that powers ZIP, GZIP, and the 100GB→25GB backup scenario—and then briefly touch how video combines similar ideas with lossy temporal compression.

3. The Building Blocks: Huffman Coding and LZ77

Most modern lossless compressors (including the ones inside ZIP and GZIP) combine two ideas:

1. Dictionary / back-reference coding (e.g. LZ77): “I’ve seen this phrase before; refer back to it.”
2. Entropy coding (e.g. Huffman): “Frequent symbols get short codes; rare symbols get longer codes.”

Together they form algorithms like DEFLATE (used in ZIP and GZIP): first reduce repetition with something like LZ77, then shorten the remaining stream with Huffman (or similar) coding.

3.1 Huffman Coding: Short Codes for Common Symbols

In normal text encoding (e.g. ASCII or UTF-8), every character uses the same number of bits (e.g. 8 bits per character). So the letter “E” and the letter “Z” both cost 8 bits, even though “E” appears far more often in English text. That’s wasteful: we’re spending the same budget on rare and common symbols.

Huffman coding fixes that by giving variable-length codes: frequent symbols get shorter codes, rare symbols get longer codes. On average, the whole message uses fewer bits.

One crucial constraint: no code is allowed to be a prefix of another. So if “E” is encoded as 10, then no other character’s code can start with 10 (e.g. no 100 or 101). That way, when we read the compressed stream left-to-right, we always know where one symbol ends and the next begins—no need for extra separators (unlike Morse code, which needs pauses between letters).

A simple way to build such codes is with a binary tree: each character is a leaf, and the path from the root to that leaf (e.g. left=0, right=1) is its code. The algorithm builds the tree from the bottom up, repeatedly grouping the two least frequent symbols (or groups) until everything is in one tree. Frequent characters end up near the root (short path); rare ones end up deeper (long path).

Conceptually:

// Standard 8-bit encoding (e.g. ASCII): every character uses 8 bits
"E" = 01000101 (8 bits)
"Z" = 01011010 (8 bits)

// Huffman encoding (example): frequent letters get shorter codes
"E" = 10      (2 bits)   ← common in English
"Z" = 111010  (6 bits)   ← rare, so longer code

So Huffman doesn’t “invent” new information; it represents the same information in fewer bits by exploiting how often each symbol appears.

3.2 LZ77: “I’ve Seen This Before—Point Back to It”

Huffman only shortens symbols based on frequency. It doesn’t remove repeated phrases. That’s where LZ77 (and family) comes in.

LZ77 keeps a sliding window of recently seen data. As it reads the input, it looks for the longest match between:

• what’s coming next (the “look-ahead” buffer), and
• what it has already seen (the “search” buffer).

When it finds a match, it doesn’t output the phrase again. Instead it outputs a back-reference: a pair (distance, length) meaning “go back distance bytes and copy length bytes.” The decoder can then reconstruct the phrase from earlier in the stream.

So the stream becomes a mix of:

• Literal bytes (when there’s no useful match), and
• Back-references (when a phrase repeats).

Example:

Input:  "The quick brown fox jumps over the quick dog"
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        "the quick" appears twice

Compressed idea:
  Literals: "The quick brown fox jumps over "
  Back-reference: (distance=31, length=9)  → copy 9 bytes from 31 bytes ago
  Literals: " dog"

After this step, the stream has fewer bytes but still contains the same information. Then Huffman (or similar) is applied to that stream to shorten the representation of literals and back-reference pairs—that’s DEFLATE in a nutshell.

4. ZIP vs. TAR.GZ: Same Ideas, Different Packaging

Both ZIP and TAR.GZ use lossless compression (often DEFLATE/GZIP-style), but they package files differently. That difference affects compression ratio and how you can access files.

ZIP: One File at a Time

ZIP compresses each file separately and then puts them in one archive, with a central directory so you can list and extract individual files.

• Pros: Random access. You can extract image_402.jpg from a 50GB archive without reading or decompressing the rest. Good for browsing and pulling single files.
• Cons: The compressor never looks across file boundaries. If 100 files are nearly identical, ZIP may store 100 separate compressed versions and miss shared patterns. So total size can be larger than a “solid” archive.

So ZIP is great when you need per-file access and broad compatibility (e.g. Windows, macOS, Linux).

TAR.GZ: One Big Stream (Solid Archive)

TAR.GZ is a two-step process:

1. TAR (Tape Archiver): Concatenate all files into one continuous stream (no compression yet). This is a “solid” archive—one long byte stream.
2. GZIP: Compress that entire stream with DEFLATE (LZ77 + Huffman) as if it were a single file.

Because GZIP sees the whole project as one stream, it can find patterns that span many files. A repeated header in file_A.txt can be used to compress the same header in file_Z.txt. That’s why you can get a much better ratio—e.g. 100GB down to 25GB—when you have lots of similar or repeated content across files.

Trade-off: to get any file out of a TAR.GZ, you typically have to decompress from the beginning up to that file. There’s no cheap “jump to file 402” as in ZIP.

# Step 1: Bundle everything into one stream (size unchanged)
tar -cvf backup.tar ./my_project

# Step 2: Compress that whole stream
gzip backup.tar

# Result: backup.tar.gz — one compressed stream

When to use TAR.GZ: Backups, source tarballs, logs, or any case where you want maximum compression and usually extract the whole archive (or don’t mind decompressing from the start to reach a file). Common on Linux and in DevOps pipelines.

When to prefer ZIP: When you need random access to individual files or maximum compatibility with non-Unix systems.

5. How Video Compression Goes Further: Temporal Redundancy

Video has a special kind of redundancy: between frames, most of the image doesn’t change. Only moving parts (and lighting changes) differ. So instead of storing every frame in full, modern codecs (e.g. H.264) use temporal compression: store one full frame, then for the next frames store mainly what changed.

Roughly:

• I-frames (Intra-frames): Full frames—like a JPEG image. They don’t depend on other frames. Used as anchors so you can seek or recover from errors.
• P-frames (Predictive): Encoded as “differences” from a previous frame (or I-frame). Much smaller than a full frame.
• B-frames (Bi-predictive): Use both past and future frames to predict the current frame, so they can be even smaller.

So conceptually:

Frame 1 (I):  Full image of a sunset
Frame 2 (P):  “Same as Frame 1, but the bird moved 2 pixels left”
Frame 3 (P):  “Same as Frame 2, but the bird moved 2 pixels left”
…
Total stored: one full frame + many small “deltas” and motion vectors

That’s temporal compression. Video also uses spatial compression inside each frame (similar in spirit to JPEG). Together, temporal + spatial (and often lossy choices) let streaming and storage stay practical at 1080p and 4K.

6. Wrapping Up

Compression is the invisible engine behind smaller backups, faster transfers, and watchable video. The main ideas are:

• Redundancy is removed or shortened: repeated phrases become back-references (LZ77), and frequent symbols get short codes (Huffman).
• Lossless (ZIP, GZIP, PNG) keeps every bit; lossy (JPEG, MP3, H.264) trades a bit of quality for much smaller size.
• ZIP compresses file-by-file for random access; TAR.GZ compresses one solid stream for better ratio when you don’t need per-file random access.
• Video adds temporal compression: store full frames rarely, and mostly store changes between frames.

Understanding these mechanics helps you choose the right format—whether you’re optimizing Next.js assets, shipping backups to S3, or tuning video encoding—and demystifies how 100GB can become 25GB without losing a single bit.

written using AI tools

Building This Blog: an MDX‑powered Next.js blog

This blog is a small, opinionated setup for writing long‑form content that feels good to read and fun to build on.

Under the hood it uses static generation, MDX, and a theme‑aware UI that works in both light and dark modes.

Everything after the frontmatter is pure MDX. Components like Callout, Steps, architecture diagrams, flows, and charts are all React components rendered directly in the post body.

Stack I ended up with

I wanted something fast, simple, and easy to reason about over years—not weeks.

The site is a static App Router app. Routes like `/blog/[slug]`, `/til/[slug]`, `/technical-terms/[slug]` are statically generated at build time.


TypeScript keeps the content layer and components honest. Tailwind + CSS variables handle layout and theming.


All long‑form content lives in the repo inside content/, written as `.md` or `.mdx` files with frontmatter.


Views, upvotes, and UTM events are stored in Supabase and surfaced through a custom analytics dashboard.

How content works

All posts are simple files in the repo—no CMS required.

---
title: "Your Post Title"
description: "Post description"
date: "2024-01-20"
author: "Your Name"
tags: ["tag1", "tag2"]
category: "Category"
featured: true
---

Your content here...

• Markdown (.md) posts go through a Remark pipeline and are rendered as HTML.
• MDX (.mdx) posts are compiled at runtime in the page using next-mdx-remote/rsc, with a shared component map.

For posts that need diagrams, flows, or charts, I use .mdx and drop in components like ArchitectureCard, FlowStep, and DemoBarChart.

Authoring flow

Add a new `.md` or `.mdx` file to content/blog/ with frontmatter for SEO, tags, and images.


Run npm run dev and iterate on copy, diagrams, and layout until the post feels polished.


Push to the main branch. The build step generates static HTML for every route and updates search data + OG images.

Architecture of the blog

At a high level, the blog is just three pieces: browser, Next.js, and Supabase.

Readers get fully rendered HTML from the CDN, with minimal JavaScript on top for upvotes, analytics, and small interactions.


Next.js statically generates pages for blog posts, TIL entries, technical terms, and lists. It also exposes a few API routes for analytics.


Supabase stores per‑page views, upvotes, and UTM events that feed the analytics dashboard and charts.

Request–response flow for a page view

Here’s how a single blog page view flows through the system.

The CDN serves a pre‑rendered HTML page that was generated at build time.


Next.js hydrates the page; components like the upvote button, custom cursor, and view counter start working.


A lightweight request records the view / upvote in Supabase without blocking the reader.


Aggregated stats are used to power Recharts visualizations on the analytics page.

Visualizing traffic with charts

The same Recharts setup used in the analytics dashboard is available inside MDX posts.

Charts are plain React components exposed to MDX. They respect the global theme and use the same accent palette as the rest of the site.

Performance before and after

Static export plus small tweaks around assets made a noticeable difference in user‑facing numbers.

Why this setup works for me

1. Files as the source of truth – I can write posts in a text editor, version them in git, and refactor them like any other code.
2. Static by default – Most pages are static HTML, which is great for speed and reliability.
3. MDX when I need power – Architecture diagrams, flows, and charts are just components; I only reach for them when a post really needs them.
4. Theme‑aware UI – Components use the same CSS variables as the rest of the app, so they look good in both light and dark modes.

If you want to explore the implementation details, the full source code is on GitHub. You could clone it, point it at your own content folder, and have a similar blog running in minutes.

Reorder the fields in a Go struct and the size changes — without changing the data it holds. A bool next to an int64 wastes 7 bytes of padding. Across 10 million allocations, that's 67MB of memory you're paying for but never using.

This matters in high-throughput services where struct slices dominate heap usage: event pipelines, in-memory caches, analytics collectors.

How alignment and padding work

Go stores struct fields in a contiguous block of memory. Each field must be aligned to a memory address that's a multiple of its own size — int64 aligns to 8 bytes, int32 to 4, bool to 1. When a smaller field is followed by a larger one, the compiler inserts invisible padding bytes to satisfy the alignment requirement.

type Bad struct {
    Active  bool    // 1 byte
    // 7 bytes padding
    Balance float64 // 8 bytes
    Age     uint8   // 1 byte
    // 7 bytes padding
}
// Total: 24 bytes (only 10 bytes of actual data)

Reorder from largest to smallest:

type Good struct {
    Balance float64 // 8 bytes
    Active  bool    // 1 byte
    Age     uint8   // 1 byte
    // 6 bytes padding (struct itself aligns to 8)
}
// Total: 16 bytes (same 10 bytes of data, 8 bytes less waste)

That's a 33% reduction per struct, just by reordering fields.

Measuring the difference

Use reflect.TypeOf and unsafe.Sizeof to check struct sizes at runtime:

package main

import (
    "fmt"
    "reflect"
    "unsafe"
)

type Bad struct {
    Active  bool
    Balance float64
    Age     uint8
}

type Good struct {
    Balance float64
    Active  bool
    Age     uint8
}

func main() {
    fmt.Println("Bad:", unsafe.Sizeof(Bad{}), "bytes")   // 24
    fmt.Println("Good:", unsafe.Sizeof(Good{}), "bytes") // 16

    // Field-by-field inspection
    t := reflect.TypeOf(Bad{})
    for i := 0; i < t.NumField(); i++ {
        f := t.Field(i)
        fmt.Printf("  %s: size=%d, offset=%d\n", f.Name, f.Type.Size(), f.Offset)
    }
}

How much memory this saves at scale

Here's the math for a real scenario — an analytics service tracking page view events:

type PageView struct {
    // Unoptimized layout
    IsBot      bool      // 1 + 7 padding
    Timestamp  int64     // 8
    StatusCode uint16    // 2 + 6 padding
    Duration   int64     // 8
    UserID     uint32    // 4 + 4 padding
    PathHash   uint64    // 8
}
// Size: 48 bytes

type PageViewOptimized struct {
    // Sorted by alignment: 8 → 4 → 2 → 1
    Timestamp  int64
    Duration   int64
    PathHash   uint64
    UserID     uint32
    StatusCode uint16
    IsBot      bool
}
// Size: 32 bytes

At 10 million structs, the difference is 152MB — enough to matter for your container memory limits and GC pressure.

Automated detection with fieldalignment

You don't need to manually audit every struct. The fieldalignment analyzer from golang.org/x/tools catches suboptimal layouts automatically:

go install golang.org/x/tools/go/analysis/passes/fieldalignment/cmd/fieldalignment@latest
fieldalignment ./...

It reports every struct that could be smaller and suggests the optimal field order. You can also run it as part of golangci-lint by enabling the govet linter with the fieldalignment check.

The alignment rules

The general rule: sort fields from largest alignment to smallest. This minimizes padding because smaller fields can pack together in the leftover space after larger fields.

Structs themselves are padded to a multiple of their largest field's alignment. That's why the Good struct above is 16 bytes (multiple of 8) even though the data only needs 10 bytes.

When not to bother

Field ordering optimization is worth the effort when:

• You allocate millions of the same struct (event pipelines, time-series data, game state)
• The struct is stored in a large slice that stays in memory
• You're hitting container memory limits or seeing heavy GC pauses

It's not worth the effort when:

• The struct is allocated once or a handful of times
• Readability would suffer from rearranging logically grouped fields
• The struct is mostly pointers and strings (already 8-byte aligned)

Run fieldalignment on your codebase as a CI check. Fix the easy wins — the structs that save 8+ bytes per instance — and leave the rest alone. The tool does the thinking for you.

Deploying a static site to S3 manually means running aws s3 sync from your laptop every time you make a change. That gets old fast. GitHub Actions can automate the entire flow: push to main, and the site builds and deploys itself.

This guide walks through the full setup for a Nanoc site, but the S3 + GitHub Actions pattern works for any static site generator.

Source code on GitHub

What you need before starting

• An AWS account with an S3 bucket configured for static website hosting
• An IAM user with scoped permissions (created in the steps below)
• A GitHub repository containing your Nanoc source code

Step 1: Configure the S3 bucket policy

Your bucket needs a policy that allows public read access to its contents. Apply this to the bucket in the S3 console under Permissions > Bucket Policy:

{
  "Version": "2008-10-17",
  "Id": "PolicyForPublicWebsiteContent",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": {
        "AWS": "*"
      },
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::docsite-github-action/*"
    }
  ]
}

Replace docsite-github-action with your actual bucket name.

Step 2: Create a scoped IAM user

Create a dedicated IAM user for GitHub Actions — don't use your root credentials. Attach this policy, which gives the minimum permissions needed for s3 sync:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "AccessToGetBucketLocation",
      "Effect": "Allow",
      "Action": ["s3:GetBucketLocation"],
      "Resource": ["arn:aws:s3:::*"]
    },
    {
      "Sid": "AccessToWebsiteBuckets",
      "Effect": "Allow",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:DeleteObject"
      ],
      "Resource": [
        "arn:aws:s3:::docsite-github-action",
        "arn:aws:s3:::docsite-github-action/*"
      ]
    }
  ]
}

Step 3: Add AWS credentials to GitHub Secrets

In your GitHub repository, go to Settings > Secrets and variables > Actions and add two repository secrets:

• AWS_ACCESS_KEY_ID — your IAM user's access key
• AWS_SECRET_ACCESS_KEY — the corresponding secret key

These are injected into the workflow at runtime. They never appear in logs.

Step 4: Create the GitHub Actions workflow

Create .github/workflows/deploy.yml in your repository:

name: Nanoc Compile and Upload to S3

on:
  push:
    branches:
      - main

jobs:
  deploy:
    runs-on: ubuntu-latest

    steps:
      - name: Checkout Repository
        uses: actions/checkout@v2

      - name: Set up Ruby and Nanoc
        run: |
          sudo apt-get update
          sudo apt-get install -y ruby-full build-essential zlib1g-dev
          sudo gem install bundler
          sudo gem install nanoc
          sudo gem install adsf
          sudo gem install kramdown

      - name: Build Nanoc Website
        run: |
          ls
          cd tutorial && nanoc

      - name: Set AWS credentials
        uses: aws-actions/configure-aws-credentials@v1
        with:
          aws-access-key-id: ${{ secrets.AWS_ACCESS_KEY_ID }}
          aws-secret-access-key: ${{ secrets.AWS_SECRET_ACCESS_KEY }}
          aws-region: ap-south-1

      - name: Push to S3
        run: aws s3 sync tutorial/output/ s3://docsite-github-action

The workflow triggers on every push to main. It installs Ruby and Nanoc, compiles the site, and syncs the output directory to your S3 bucket.

Alternatives to consider

This setup works well for simple static sites, but depending on your needs, other approaches might be a better fit:

S3 alone doesn't give you HTTPS — you'd need CloudFront in front of it for that. If HTTPS and preview deployments matter to you, Vercel or Netlify will save you time.

Common pitfalls

• Forgetting to enable static website hosting on the S3 bucket — without it, S3 serves files as downloads instead of web pages.
• Overly broad IAM permissions — scope the policy to your specific bucket, not s3:* on *.
• Cache invalidation — s3 sync updates files but doesn't invalidate CloudFront caches. If you add CloudFront later, add aws cloudfront create-invalidation to the workflow.
• Region mismatch — set aws-region in the workflow to match your bucket's region.

The complete workflow YAML and policies are in the repo.

A ride-sharing app needs to match riders with drivers in real-time, track locations, process payments, and handle all of this at scale across a city. This is my take on how to architect such a system using microservices — and where a simpler approach might actually be better.

Why microservices for a ride app

The core challenge is that different parts of the system have very different scaling requirements:

• Geo-location tracking needs to handle thousands of location updates per second with sub-100ms latency
• Payment processing needs strong consistency and idempotency but handles far fewer requests
• Notifications are fire-and-forget at high volume
• User authentication is read-heavy with infrequent writes

A monolith would force all of these to scale together. Microservices let you scale the geo-location service to 50 instances while keeping the payment service at 3.

That said, if you're building an MVP or serving a single city, start with a monolith. Uber itself started as a monolith. Extract services only when specific components hit scaling bottlenecks.

Service boundaries

Each service owns its data and exposes a clear API:

User Service — Manages rider and driver accounts, authentication (OAuth2/OIDC), profile data, and session management. Backed by PostgreSQL.

Ride Service — The core coordination service. Handles ride requests, matches riders with nearby available drivers, tracks ride state (requested → matched → in-progress → completed), and calculates fares. Backed by PostgreSQL with Redis for active ride state.

Driver Service — Manages driver availability, approval status, vehicle information, and earnings. Tracks which drivers are online, idle, or on a ride. Backed by PostgreSQL.

Geo-Location Service — The highest-throughput service. Receives GPS coordinates from driver and rider apps every 3–5 seconds, stores them in a spatial index (Redis with geospatial commands or a dedicated service like H3), and answers proximity queries ("find the 5 nearest available drivers within 3km"). This is the service that needs the most aggressive scaling.

Payment Service — Integrates with payment gateways (Razorpay, Stripe). Processes charges after ride completion, handles refunds, and manages driver payouts. Must be idempotent — a network retry should never charge a rider twice.

Notification Service — Sends push notifications (ride matched, driver arriving, ride completed), SMS fallbacks, and email receipts. Consumes events from a message queue rather than being called directly.

How the services communicate

Not all communication should use the same pattern:

The API Gateway sits in front of all services and handles routing, rate limiting, and authentication token validation. Clients never talk directly to internal services.

A critical design decision: the Ride Service publishes a ride.completed event to Kafka. The Payment Service, Notification Service, and Analytics Service all consume this event independently. This means adding a new consumer (say, a driver-rating prompt) doesn't require changing the Ride Service at all.

Scaling and fault tolerance

Load balancing. An L7 load balancer (like AWS ALB or Envoy) distributes requests across service instances. The Geo-Location Service gets its own load balancer with sticky sessions disabled (since requests are stateless).

Auto-scaling. Kubernetes Horizontal Pod Autoscaler watches CPU and custom metrics (like queue depth for the Notification Service). The Geo-Location Service might scale from 10 pods at 2 AM to 80 pods at 6 PM during rush hour.

Circuit breakers. If the Payment Service is down, the Ride Service shouldn't hang waiting for it. A circuit breaker (e.g., via Istio or a library like resilience4j) fails fast after N consecutive errors and falls back to queuing the payment for later processing.

Health checks. Every service exposes /healthz (liveness) and /readyz (readiness) endpoints. Kubernetes restarts crashed pods automatically and stops routing traffic to pods that aren't ready.

Security considerations

• Authentication: OAuth2 with JWTs for user-facing APIs. Service-to-service calls use mTLS within the Kubernetes cluster.
• Data encryption: AES-256 at rest, TLS 1.3 in transit. Payment card data never touches your services — use the payment gateway's tokenization.
• Rate limiting: Applied at the API Gateway level. Geo-location updates are rate-limited per device to prevent abuse.
• Secrets management: HashiCorp Vault or AWS Secrets Manager. No secrets in environment variables or config files.

Deployment

• Containerization: Each service is a Docker image with a multi-stage build (builder → runtime). Images are scanned for vulnerabilities in CI.
• Orchestration: Kubernetes with namespaces per environment (dev, staging, prod). Services declare resource requests and limits.
• CI/CD: GitHub Actions or GitLab CI runs tests, builds images, pushes to a container registry, and deploys via Helm charts or ArgoCD.
• Observability: Prometheus + Grafana for metrics, Jaeger for distributed tracing, Fluentd/Loki for centralized logs. Every service emits structured JSON logs with a correlation ID.

What I'd do differently at different scales

The biggest mistake is building for the third row when you're at the first row. Start simple, measure, and extract when the pain is real.

Most startups need to send notifications — order confirmations, alerts, password resets — but don't want to run their own messaging infrastructure. Amazon Simple Notification Service (SNS) solves this: a fully managed pub/sub service where you pay per message, not per server.

Here's what makes it worth evaluating, where it falls short, and how it compares to alternatives.

The startup messaging problem

Early-stage teams face a specific tension: they need reliable message delivery across email, SMS, and push notifications, but can't justify the cost or operational overhead of self-hosted messaging systems. The requirements typically include:

• Pay-as-you-go pricing — no monthly minimums, no long-term contracts
• Multi-channel delivery — email, SMS, mobile push, HTTP webhooks, Lambda triggers
• Automatic retries — temporary failures shouldn't lose messages
• Global reach — users in multiple regions need low-latency delivery

SNS addresses all four. But so do other services, which is why the trade-offs matter.

How SNS keeps costs low

SNS uses a pay-per-message model. The first million SNS API requests per month are free. After that, it's $0.50 per million requests. SMS and email have separate per-message pricing that varies by destination country.

For a startup sending 100K push notifications and 10K emails per month, the SNS cost is effectively zero (within the free tier). Compare that to a dedicated email service like SendGrid or Mailgun, which typically start at $15–20/month for similar volumes.

The catch: SNS doesn't do rich email templates, drip campaigns, or analytics. It's a delivery pipe, not a marketing platform.

How SNS ensures delivery

Three mechanisms prevent message loss:

Retry logic. When a delivery attempt fails (subscriber endpoint is down, network timeout), SNS automatically retries with exponential backoff. The retry policy is configurable per delivery protocol.

Dead letter queues. Messages that exhaust all retry attempts are routed to an SQS dead letter queue instead of being silently dropped. You can inspect these later, replay them, or trigger alerts.

Cross-AZ replication. Messages are replicated across multiple availability zones within a region before SNS acknowledges the publish request. This protects against hardware failures in a single data center.

SNS vs SQS vs EventBridge: when to use what

This is the decision most teams get wrong. All three are AWS messaging services, but they solve different problems:

The common pattern is SNS + SQS together: SNS fans out an event to multiple SQS queues, each consumed by a different microservice. This gives you both broadcast and buffering.

If you only need one consumer, skip SNS and use SQS directly. If you need content-based routing (e.g., "send order events to the billing service, send inventory events to the warehouse service"), EventBridge is the better choice.

Real-world usage

Netflix uses SNS to send push notifications about new content releases. When a new season drops, SNS fans out the notification to millions of subscriber endpoints simultaneously.

Amazon itself uses SNS for order lifecycle notifications — placed, shipped, delivered — routing events to email, SMS, and mobile push depending on customer preferences.

Walmart uses SNS for order fulfillment and in-store pickup notifications, integrating with their logistics systems to trigger real-time updates.

Where SNS falls short

• No rich content. SNS messages are plain text (or JSON for application-to-application). If you need HTML email templates, open/click tracking, or A/B testing, you need SES or a third-party email service.
• No guaranteed ordering. Standard SNS topics don't guarantee message order. FIFO topics do, but they're limited to 300 messages/second per topic.
• Vendor lock-in. SNS integrates deeply with AWS services (Lambda, SQS, CloudWatch). Migrating to a different cloud later means rewriting all your pub/sub logic.
• SMS costs add up. International SMS delivery can be expensive ($0.02–0.15 per message depending on country), and you need to manage opt-in compliance yourself.

When to pick something else

• Transactional email with templates: Use Amazon SES or SendGrid
• Marketing automation (drip campaigns, segmentation): Use Brevo, Mailchimp, or Customer.io
• Real-time chat or presence: Use WebSockets or a service like Ably/Pusher
• Cross-cloud pub/sub: Use Google Cloud Pub/Sub, Confluent Kafka, or NATS

Getting started

If you decide SNS fits, the setup is straightforward:

1. Create an SNS topic in the AWS Console or via CloudFormation/CDK
2. Add subscribers (email, SQS queue, Lambda function, HTTP endpoint)
3. Publish messages via the AWS SDK from your application code
4. Configure a dead letter queue to catch failed deliveries
5. Set up CloudWatch alarms on NumberOfNotificationsFailed

The AWS SNS documentation covers each step with working examples in Python, Node.js, and Java.

S3 has three overlapping access control systems — bucket policies, IAM policies, and ACLs — and the interaction between them confuses most people the first time. Here's how each one works, when to use which, and the JSON to copy for the most common scenarios.

The three access control layers

Every S3 request is evaluated against all applicable policies. If any of them explicitly deny the request, it's denied. Otherwise, at least one policy must explicitly allow it.

The rule of thumb: Use IAM policies for your own users, bucket policies for external access or bucket-wide rules, and ignore ACLs unless you're dealing with legacy configurations.

Anatomy of an S3 policy

Every policy is a JSON document with these fields:

• Version — always "2012-10-17" (the current policy language version)
• Statement — an array of permission rules, each containing:
  • Effect — "Allow" or "Deny"
  • Principal — who the rule applies to ("*" for everyone, or a specific ARN)
  • Action — which S3 operations (s3:GetObject, s3:PutObject, etc.)
  • Resource — which bucket/objects (specified as an ARN)
  • Condition (optional) — extra constraints like IP range, encryption type, or request origin

Common policies with working JSON

Public read-only access

Makes all objects in a bucket publicly readable. Use this for static website hosting or public assets:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "PublicReadGetObject",
      "Effect": "Allow",
      "Principal": "*",
      "Action": "s3:GetObject",
      "Resource": "arn:aws:s3:::my-bucket/*"
    }
  ]
}

This allows anyone to read objects but not list the bucket contents, upload, or delete. The /* in the Resource means all objects inside the bucket.

Deny uploads without encryption

Forces all uploaded objects to use server-side encryption. Note: this uses Deny + a StringNotEquals condition, not Allow:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyUnencryptedUploads",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::my-bucket/*",
      "Condition": {
        "StringNotEquals": {
          "s3:x-amz-server-side-encryption": "AES256"
        }
      }
    }
  ]
}

This is the correct pattern. A common mistake is using Allow with a StringEquals condition — that permits encrypted uploads but doesn't block unencrypted ones if another policy allows s3:PutObject.

Scoped access for a specific IAM user

Grants a single user read and write access to a bucket:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "UserReadWrite",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/deploy-bot"
      },
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::my-bucket/*"
    },
    {
      "Sid": "UserListBucket",
      "Effect": "Allow",
      "Principal": {
        "AWS": "arn:aws:iam::123456789012:user/deploy-bot"
      },
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::my-bucket"
    }
  ]
}

Note the two statements: object-level actions use my-bucket/* (objects inside), while ListBucket uses my-bucket (the bucket itself). Mixing these up is a common source of "Access Denied" errors.

Restrict access by IP range

Allows access only from your office or VPN IP range:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "RestrictToOfficeIP",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::my-bucket",
        "arn:aws:s3:::my-bucket/*"
      ],
      "Condition": {
        "NotIpAddress": {
          "aws:SourceIp": "203.0.113.0/24"
        }
      }
    }
  ]
}

Bucket policy vs IAM policy: when to use which

In general: if the question is "who can access this bucket?", use a bucket policy. If the question is "what can this user/role do?", use an IAM policy.

Common mistakes

• Forgetting S3 Block Public Access. Even if your bucket policy allows public reads, the account-level Block Public Access setting overrides it. Check this first when public access isn't working.
• Resource ARN mismatch. s3:ListBucket needs the bucket ARN (arn:aws:s3:::my-bucket), while s3:GetObject needs the object ARN (arn:aws:s3:::my-bucket/*). This trips up almost everyone.
• Using ACLs. AWS recommends disabling ACLs on new buckets (S3 Object Ownership: "Bucket owner enforced"). Bucket policies and IAM policies cover every use case that ACLs used to handle, with better auditability.
• Overly broad wildcards. "Action": "s3:*" with "Resource": "*" is an admin-level policy. Scope both to the specific actions and bucket you need.

The AWS Policy Generator can help you build policies interactively if you're not sure about the syntax.