Why did my metric drop? A step-by-step way to find out

To find out why a metric dropped, first confirm the drop sits outside the metric’s normal range and isn’t a data problem. Then slice the metric by each dimension to find the segment that holds most of the change, check whether a rate fell or the mix shifted, and list what shipped just before the drop started.

The manual routine, step by step

Most analysts run some version of the same five steps. As an example, say checkout conversion reads 3.1% on a Tuesday when it usually sits near 3.4%.

1. Spot it. Someone sees the number on a dashboard or gets asked about it in Slack.
2. Check that it’s real. Confirm the data loaded completely, then compare the value with the metric’s normal for that day of the week.
3. Slice it. Break the metric down by each dimension, such as device, channel, country and plan. Find the segment where the change concentrates, then break that segment down further.
4. Ask what shipped. Check the deploy log, release notes, the marketing calendar and recent pricing changes for anything that landed just before the drop.
5. Write it up. Put the numbers, the segment and the timing into a few sentences a stakeholder can act on.

How do you check that a drop is real?

Start with the data, because a broken pipeline looks exactly like a business problem. Check that the latest load finished, that row counts match the same day last week, that nulls haven’t jumped in a key column, and that no dimension value has disappeared. A tracking change that drops one platform from the events table can make conversion fall overnight.

Then compare the number with its own normal. Tuesday against Monday tells you little if Monday is always your busiest day. Compare against the same weekday over several weeks, and allow for any trend. Many alarming drops are ordinary weekly variance, as our guide to why threshold alerts fire every Monday explains.

How do you find which segment moved?

Slice the metric one dimension at a time and compare each segment’s change with the total change. Then look past size. A segment can account for most of a drop simply because it carries most of your traffic. The segment you want is the one that moved unusually compared with its own history.

Segment (example)	Share of sessions	Usual conversion	Tuesday conversion
Desktop web	45%	2.7%	2.6%
iOS app	30%	4.8%	3.9%
Android app	25%	3.0%	3.0%

In this example, desktop is the largest segment, and its 0.1-point dip is within its normal range. iOS fell 0.9 points and holds most of the drop. Next, slice iOS by app version and country to see whether the change narrows.

For a ratio metric like conversion, also check whether each segment’s share of traffic changed. If every segment converts at its usual rate but more traffic arrives from a low-converting channel, the total still falls. That is a mix effect, and you fix it differently from a rate effect. See rate effect vs mix effect for a worked example.

What shipped right before the drop?

Once you know where the change sits, pin down when it started, as precisely as your data allows. Daily data gives you the day. Hourly data can give you the hour, and that matters. A deploy at 9:00 followed by a drop at 9:40 is a much tighter match than “something on Tuesday.”

Then list what happened in the hours before. Deploys, flag changes, pricing changes, campaigns and incidents all count. This step is usually the slowest, because the information sits in GitHub, a project tracker, a marketing calendar and people’s heads. The analyst ends up asking four teams and waiting for replies.

An event near the start of a drop is a coincidence in time. Check it first, and remember that timing alone proves nothing.

Where the manual routine breaks down

In practice the routine fails in predictable ways.

• It’s slow. Slicing, drilling and asking around can take most of a day, and the answer arrives after the damage is done.
• It only covers metrics someone watches. A drop in a metric nobody checks goes unnoticed.
• Alerts get muted. Fixed thresholds fire on normal weekly swings, so teams switch them off and then miss the real change.
• The biggest segment looks guilty. Without comparing each segment with its own history, analysts tend to name the largest one.
• Slicing multiplies. Five dimensions with ten values each give fifty segments, and far more combinations below them.
• Timing turns into cause. “Release 2.14 went out, then activation fell” becomes “release 2.14 broke activation” by the time it reaches a leadership channel.

How Metron automates each step

Metron is self-hosted root-cause analysis for business metrics. It runs the same investigation inside your environment, querying your PostgreSQL or BigQuery warehouse with a read-only role. An investigation starts when Metron detects a change or when someone asks a question, such as why revenue dropped last Tuesday. Both return the same evidence.

Manual step	What Metron does
Spot it	When Metron checks a metric, it removes the weekly pattern and the trend, then judges the latest value against that metric’s own normal range.
Check it’s real	Data quality checks for freshness, row counts, nulls and missing dimension values run before detection. A metric whose load failed never reaches the detector, so it can’t raise a false business alarm.
Slice it	Metron tests each segment of each dimension against that segment’s own history and names the segment that holds most of the change. It drills into combinations, such as US mobile, when that narrows the change, and reports how much of the change the named segments leave unexplained.
Rate or mix	For ratio metrics, Metron says whether a segment’s rate changed or the mix of volume shifted between segments.
Ask what shipped	Deploys and releases arrive from GitHub by webhook. You log other events, such as a price change, yourself. Metron lists the events that fall close to the start of the change, with how long before it each one landed.
Write it up	A language model you supply words the findings in plain language. It sees the findings only, never your tables. Follow-up questions are answered from the same evidence.

Here is an example of a finished write-up.

Revenue is 11.8% below expected. 72% of the decline comes from US mobile. Within that, iOS conversion fell from 4.8% to 3.9%. Deployment checkout-v4.21 went out 38 minutes before the decline starts.

Attribution: high confidence. Temporal association: medium confidence. Causality: not established from warehouse data.

Metron scores where the change sits and when it started separately, and it never claims cause. The homepage shows how Metron scores each claim. Write-ups can go out by webhook or email, and mark each finding useful or not. Metron stores that verdict with the evidence. It does not use it to retrain or tune detection.

Metron is in private beta, and we’re setting up a small number of teams by hand. If you run PostgreSQL or BigQuery and have a metric your team can’t explain, request beta access and tell us which one.