Skip to content

The One Query That Takes Down Launch Day

Most launch-day outages come down to a single database lookup nobody indexed. Here is how to find yours in an afternoon.

3 min read

Systems fail on good days. The campaign lands, registration opens, the post goes further than expected — and the thing that had run quietly for a year falls over in four minutes.

In our experience the cause is usually singular and boring: one database lookup, running on every page load, scanning a whole table because nobody added an index.

Why it hides so well

At normal traffic that query costs a few milliseconds. Nobody notices, no dashboard flags it, and it stays invisible for as long as the load stays flat. Under a spike it becomes the queue that everything else waits behind, and the symptom shows up somewhere else entirely — a slow checkout, a timing-out report, a page that never finishes loading.

That is why teams spend launch day looking at the frontend. The page is fine. It is waiting.

Finding it before the date

You do not need a load test to start. You need your slowest queries, which almost every database can tell you:

  • Turn on slow query logging, or read the statement statistics your database already collects
  • Sort by total time, not by worst single run — the query called ten thousand times matters more than the one that took two seconds once
  • Run the top offender through your database's query planner and look for a full table scan
  • Add the index, measure again, and keep the before and after numbers

Expect the first fix to reveal a second bottleneck. That is normal: the slowest thing was hiding everything behind it. Three rounds is a typical outcome of doing this properly, not a sign that something was built badly.

Then confirm it with load

Reading query stats tells you where the risk is. Only a load test against a copy of production, with a realistic amount of data, tells you where the ceiling is. Testing against an empty database proves nothing — the scan that kills you does not exist when the table has twelve rows.

The full approach, including pre-warming capacity for a known start time, is on high-traffic campaign platform. What it looked like on a real campaign is in the case study.

Solutions

Services

  • Backend Engineering

    The part nobody sees, and the part that decides whether your product survives a busy Monday.

  • Cloud & DevOps

    Deploys that are boring, costs you can predict, and a system that stays up on your biggest day.

Topics

  • Performance & Scale

    What breaks under load, why it breaks there, and how to find out before customers do.