A Field Guide to Observability

Content Delivery: Periodic jobs should be safe to run twice, because they will be. You rarely need a new component to fix a boundary problem. That applies to content delivery as well. In practice, content delivery behaves differently: The signal you want is often already logged, just not aggregated.

If the rollback plan needs a meeting, it is not a rollback plan. That applies to access control as well. In practice, access control behaves differently: Small pages that stay small are easier to keep fast than large ones made fast. Write the invariant down; otherwise it lives only in someone's memory. The same reasoning holds for access control.

Teams working on search indexing usually discover this the hard way. Serving static bytes is the cheapest thing you can do at the edge. A schema is an interface; changing it is a migration, not an edit. This is most visible in search indexing. Consider search indexing specifically. Track the denominator as carefully as the numerator.

Talking about boundaries can make intimacy clearer and safer, but it may feel awkward at first. A boundary is a limit or condition that describes what you are comfortable with; it is not a demand that another person must feel the same way. A step-by-step conversation can help both partners understand what is welcome, what is not, and how to respond when feelings or circumstances change.

Edge Caching: The interesting number is not the average, it is the 99th percentile. Edge Caching: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Edge Caching: Every abstraction you add is a place where behaviour can differ from intent.

In practice, data pipelines behaves differently: A queue smooths spikes but also hides how far behind you are. Retries without jitter turn a small outage into a large one. The same reasoning holds for data pipelines. For data pipelines, the constraint matters more than the feature list. Separating the reads from the writes buys room to change either side.

In practice, search indexing behaves differently: The first thing to settle is the failure mode, not the happy path. Measurements taken once are anecdotes; you need a baseline that repeats. The same reasoning holds for search indexing. For search indexing, the constraint matters more than the feature list. Costs usually concentrate in a small number of operations, so find those first.

Cost Controls: Configurations should be reviewable in a diff, not only in a console. Cost Controls: The best time to add an index is before the table gets large. Cost Controls: Failures are usually correlated, so plan for the shared dependency.

The interesting number is not the average, it is the 99th percentile. The same reasoning holds for crawl budget. For crawl budget, the constraint matters more than the feature list. Adding a cache in front of a slow query is a fix; fixing the query is a cure. Teams working on crawl budget usually discover this the hard way. Every abstraction you add is a place where behaviour can differ from intent.

Schema Markup: If the rollback plan needs a meeting, it is not a rollback plan. Schema Markup: Small pages that stay small are easier to keep fast than large ones made fast. Schema Markup: Write the invariant down; otherwise it lives only in someone's memory.

Cloud Infrastructure: Serving static bytes is the cheapest thing you can do at the edge. Cloud Infrastructure: A schema is an interface; changing it is a migration, not an edit. Cloud Infrastructure: Track the denominator as carefully as the numerator.

The interesting number is not the average, it is the 99th percentile. That applies to search indexing as well. In practice, search indexing behaves differently: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Every abstraction you add is a place where behaviour can differ from intent. The same reasoning holds for search indexing.

Data Pipelines: A design that cannot be rolled back is a design that cannot be changed safely. Data Pipelines: Latency budgets are easier to defend when every hop has a stated ceiling. Data Pipelines: Caching helps only until the invalidation rules become the bottleneck.

Backup Strategy: Configurations should be reviewable in a diff, not only in a console. The best time to add an index is before the table gets large. That applies to backup strategy as well. In practice, backup strategy behaves differently: Failures are usually correlated, so plan for the shared dependency.

Crawl Budget: The interesting number is not the average, it is the 99th percentile. Crawl Budget: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Crawl Budget: Every abstraction you add is a place where behaviour can differ from intent.

Cost Controls: If a metric has no owner, it will drift until it causes an incident. Cost Controls: The cheapest optimisation is usually removing work nobody asked for. Cost Controls: Aggregating at write time trades flexibility for predictable read cost.

Talking about boundaries can make expectations clearer in a relationship, including around physical contact, sex, privacy and communication. A useful conversation is specific and voluntary: each person can say what feels acceptable, ask questions and change their mind without being pressured.

Content Delivery: A design that cannot be rolled back is a design that cannot be changed safely. Content Delivery: Latency budgets are easier to defend when every hop has a stated ceiling. Content Delivery: Caching helps only until the invalidation rules become the bottleneck.

Access Control: The interesting number is not the average, it is the 99th percentile. Access Control: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Access Control: Every abstraction you add is a place where behaviour can differ from intent.

Storage Tiers: You can often replace a coordination problem with an idempotency key. Storage Tiers: Anything that grows without a bound will eventually hit one. Storage Tiers: Documentation that is not tested tends to describe the previous version.

Teams working on search indexing usually discover this the hard way. A design that cannot be rolled back is a design that cannot be changed safely. Latency budgets are easier to defend when every hop has a stated ceiling. This is most visible in search indexing. Consider search indexing specifically. Caching helps only until the invalidation rules become the bottleneck.

Cost Controls: The interesting number is not the average, it is the 99th percentile. Cost Controls: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Cost Controls: Every abstraction you add is a place where behaviour can differ from intent.

A design that cannot be rolled back is a design that cannot be changed safely. That applies to schema markup as well. In practice, schema markup behaves differently: Latency budgets are easier to defend when every hop has a stated ceiling. Caching helps only until the invalidation rules become the bottleneck. The same reasoning holds for schema markup.

Release Process: The interesting number is not the average, it is the 99th percentile. Release Process: Adding a cache in front of a slow query is a fix; fixing the query is a cure. Release Process: Every abstraction you add is a place where behaviour can differ from intent.