What thousands of A/B tests reveal about websites that convert and retain

Casey Hill on Churn FM, on what leading companies keep after a test concludes, and why retention starts before the signup.

Show

Churn FM, episode 296

Host

Andrew Michael

Guest

Casey Hill, DoWhatWorks

Runtime

45:08

What is covered

01 What brands keep, not what they test The database tracks which version survives after a test concludes. That is a different and more useful signal than what a team chose to try.

02 Where the tracking engine looks Company size, page type, industry and geography all get factored in, because the same change does not behave the same way across those cuts.

03 Conversion and retention are the same conversation A page that oversells produces signups that churn. Setting accurate expectations before the click is a retention decision as much as a conversion one.

Churn FM is a show about retention, which makes it a useful place to argue that acquisition and retention are not separate problems.

Casey brings experience from ActiveCampaign, teaching at Stanford, and years of research into how companies like Slack and Shopify actually run their sites.

A page that oversells produces signups that churn.

What to take from it

Ask what a brand kept after the test, not what they tried. Those are different questions.
Segment your own results the way the data does: by page type and company size, not in aggregate.
Treat expectation setting as a retention lever, not just a conversion one.

Episode Listen to episode 296 on Churn FM