Getting Started
To use Cazy AB, install it from the Shopify App Store and open it inside your Shopify admin. The first thing you'll see is the onboarding checklist — follow it to connect your store in under five minutes.
After installing, navigate to Online Store → Themes → Customize, open the App Embeds panel, and enable the Cazy AB Testing embed. This activates the storefront script that runs your experiments. Save your theme and you're live.
You can confirm the script is running by visiting your storefront, opening your browser console, and checking for a log line that starts with [Cazy AB].
Enable the app embed on a development theme first to verify everything is working before activating on your live theme.
Creating Your First Experiment
From the Cazy AB dashboard, click Create Experiment. Choose an experiment type — start with Content if you want to test page copy, or Shipping if you want to test free-shipping thresholds.
Give your experiment a name and description. Click AI Generate next to the Hypothesis field and Cazy AB will write a testable hypothesis for you based on what you're testing.
Define your variants. The first variant is always the Control (your current experience). Add a second variant with the change you want to test. Set the traffic split — 50/50 is the most common starting point.
Set your primary goal (Conversion Rate or Revenue per Visitor), then click Create Experiment. Your experiment will be saved as a draft. Open it and click Start when you're ready to go live.
Run one experiment at a time to start. Overlapping experiments can affect each other's results if they target the same pages.
Understanding Results
The results panel shows your key metrics per variant: Conversion Rate (the percentage of visitors who completed your goal), Revenue per Visitor (total revenue divided by unique visitors), and Uplift (the percentage improvement over the control).
The p-value measures statistical confidence. When the p-value drops below 0.05, your result is statistically significant — meaning you can be 95% confident the difference between variants is real and not random noise. Cazy AB highlights significant results automatically.
The SRM check (Sample Ratio Mismatch) warns you if visitors are not being split in the expected ratio. An SRM usually means something in your setup is redirecting or filtering visitors before they can be bucketed. Fix the issue and restart the experiment.
Wait until each variant has at least 100 visitors before reading results. Early data is noisy and can point you in the wrong direction.
AI Simulation
AI Simulation is available on the Starter plan and above. It predicts which variant is likely to win before you have enough real data to reach statistical significance.
To run a simulation, open any experiment and click Run AI Simulation. The simulation takes 30–60 seconds. When it completes, you'll see predicted conversion rates, revenue per visitor, and a probability-to-be-best score for each variant.
Simulations use industry benchmarks for your product category and model realistic visitor behaviour across multiple cohorts. The more experiments you complete, the more accurate future simulations become — Cazy AB calibrates itself using your actual results.
Use simulation to decide whether an experiment is worth running at full traffic, not to replace it. Always let real data confirm the winner.
Billing & Plans
Cazy AB charges a flat monthly fee through Shopify's billing system. You'll never be charged outside of Shopify.
The Free plan gives you 1 experiment and 1,000 tracked visitors per month — enough to test one idea at a time. The Starter plan ($19/month) unlocks 5 experiments, 10,000 visitors, and AI simulation. The Growth plan ($99/month) removes experiment limits and supports 100,000 visitors. The Enterprise plan ($299/month) has no limits and includes dedicated support.
To upgrade, go to Settings → Billing inside the app. You'll be redirected to Shopify's payment confirmation screen. Downgrades take effect immediately — any experiments over your new plan's limit will be paused automatically.
Visitor counts are tracked monthly and reset on the 1st of each month. Archived experiments do not count against your experiment limit.