Running 10 tests a month sounds productive, but if none of them are tied to a real business question, you're just taking swings for the sake of swinging. In this episode Adam and Alex dig into why "how many tests should I run?" is often the wrong question, and what to ask instead when you're building an experimentation program on Shopify.

In this episode of IntelliJAMS, Adam and Alex explore the difference between test output and test outcomes, how to run concurrent tests without confounding your results, and why elevating the conversation from "number of tests" to "strategic priorities" changes everything.

Timestamps:
0:00 - Intro
0:27 - The question everyone's asking right now
0:52 - Why "how many tests" is the wrong question
2:23 - Outputs vs. outcomes: reframing productivity
3:10 - Test the plan, don't plan the tests
3:51 - You can't predict how many tests it'll take
4:54 - Running concurrent tests without confounding results
5:53 - Scoping metrics to the right part of the funnel
6:48 - Every test needs a hypothesis
7:30 - One test often leads to five new questions
8:15 - Real example: ending a checkout upsell test early
9:11 - How to push back on "I need 10 tests this month"
11:49 - Elevating the conversation from output to strategy
13:28 - Final thoughts: see the forest for the trees

Topics covered:

Why counting tests is an output metric, not an outcomes metric

How to reframe testing around strategic business goals

Running concurrent A/B tests on Shopify without confounding data

Segmenting tests by funnel stage and visitor type

How one test can lead to five new questions

Pushing back when stakeholders demand a test quota

Real-world example: killing a checkout upsell test 24 hours in

The "test the plan, don't plan the tests" framework

Want to sharpen your experimentation skills? Join GEM Academy for free courses and a community of brands sharing what works: https://www.skool.com/intelligems-academy-1535

Website: https://intelligems.io
Blog: https://intelligems.io/blog
LinkedIn: https://www.linkedin.com/company/intelligems