Why did activation rate drop? An illustrative walkthrough

Activation rate usually drops for one of two reasons. One group of signups stopped activating, or the signup mix shifted toward a group that activates less. To find which, confirm the drop is real, find the segment that holds it, then check what shipped just before it began. The walkthrough below follows one illustrative investigation through those steps.

This is an illustrative example. The product, the numbers and the release are made up to show how a Metron investigation reads. They are not customer data.

The setup

A SaaS team defines one metric in Metron, on the signups table in its warehouse.

Field	Value
Metric	activation_rate
Value	Activated signups over all signups, count(activated) / count(signups)
Time	signed_up_at
Slice by	signup_method, plan, country
Events	GitHub releases, sent by webhook

In Metron’s metric builder, that value is a value expression of CASE WHEN activated THEN 1 ELSE 0 END and a denominator expression of 1. Metron adds both up for each time period and divides. The denominator is what makes this a ratio metric.

Step 1: Detect. Is the drop real?

On Wednesday afternoon, someone on the team runs a check. Metron reports:

Activation rate is 9.4% below expected since Wed 10:00 UTC.

Before it judges anything, Metron removes the metric’s weekly pattern and its trend. The 9.4% is measured against what this metric normally does at this point in the week, so a quiet midweek stretch doesn’t count. For more on that, read how seasonality causes false alarms.

Metron narrows the start to the hour because signups carry timestamps across the day. When a table only supports daily timing, Metron says the timing is accurate to the day.

Step 2: Locate. Where does the drop sit?

Metron tests each segment of signup_method, plan and country against that segment’s own history. One segment holds most of the change.

Segment	Share of the drop
Google SSO signups	68%
Email signups	17%
SAML signups	9%

68% of the drop is signups through Google SSO. Their activation fell from 41% to 29%.

Metron labels this a rate effect. Google SSO signups activate less often than they did. If the signup mix had shifted toward a segment that activates less while every segment’s rate held, Metron would report a mix effect, and the team would look at acquisition instead of the product. The glossary entry on rate effect vs mix effect walks through the difference.

Step 3: Line up events

Releases reach Metron from GitHub by webhook. Metron looks for events close to the start of the change and lists them nearest first, with the time gap.

Release auth-service 2.14 went out 52 minutes before the drop began.

That sentence describes timing only. Metron attaches no likelihood to the release and does not call it the cause.

Step 4: The write-up

The language model the team supplies receives the findings and nothing else. It never sees the signups table. It words the result in a few sentences, and Metron checks that every number and name in them appears in the findings.

Activation rate is 9.4% below expected since Wed 10:00 UTC. 68% of the drop is signups through Google SSO. Their activation fell from 41% to 29%. Release auth-service 2.14 went out 52 minutes before the drop began.

Each claim carries its own confidence.

Claim	Confidence
Where: Google SSO signups	High
When: auth-service 2.14, 52 minutes before	Medium
Why: auth-service 2.14 caused the drop	Not claimed

The team has webhook delivery switched on for this metric, so the write-up lands in their channel. Someone asks a follow-up, and Metron answers from the same evidence.

Are email signups affected? No. Email signups are down 1.6%, inside their normal range. The drop sits in Google SSO signups.

What the team does next

Metron hands over a narrow question. Did something in auth-service 2.14 break activation for Google SSO signups? The team answers it.

1. Read the release. Open the auth-service 2.14 changes and look for anything on the Google SSO path, such as the callback, the requested scopes or session creation after sign-in.
2. Check the edges with follow-ups. Ask whether the drop is limited to one plan or one country. A drop spread evenly across plans and countries points back at the shared login path.
3. Fix or roll back, then check again. After the fix ships, run a check and see whether Google SSO activation returns to its normal range.
4. Record a verdict. Mark the investigation useful, or pick another verdict, such as false positive or wrong attribution, if it didn’t hold up. Metron stores the verdict with the evidence. It does not retrain detection.
5. Log what you learn. If the cause turns out to sit outside the release, such as a change on the identity provider’s side, log it as an event with the time it happened. A rerun of this investigation then lists it next to the drop.

If auth-service 2.14 turns out to be clean, the location still holds. The drop sits in Google SSO signups, and the search starts there instead of with every team that shipped something that week. For the general method, read why did my metric drop.

Metron is in private beta, and we set up each team by hand. If you have an activation metric your team can’t explain, request beta access and tell us which warehouse it lives in.