Facebook Community
Change Log
Help Center
How to A/B Test Promotional Pop-Ups on Shopify [Using Two Methods]
Pop-ups could be a "game-changer" OR "deal-breaker" depending on how you use them.
Because when used creatively, pop-ups can drive email sign-ups and revenue from promotional emails. And on the flip side, they can annoy your site visitors to the extent that they’d leave your site.
So, how do you do pop-ups right? A/B testing is the solution that can help you find the answer.
In this blog post, we’ll walk you through a detailed guide on how to A/B test promotional pop-ups with two different methods.
Fundamentals of A/B Testing Promotional Pop-Ups
What is A/B Testing in Shopify?
A/B testing is a technique in which two different versions of a webpage are compared by distributing them to different visitors to identify which one performs better for a defined business goal.
For example, you can create two versions of a product page by changing one of the elements and analyzing the impact of that change on your conversion rate.
What is a Promotional Pop-Up in Shopify?
In your Shopify store, you can configure a promotional pop-up that appears on top of the webpage to grab the visitors' attention with a discount offer or any other reward and get them to sign up with their email and/or phone number.
Once the contact details are collected, along with the consent for marketing emails, the brand can start promotional email marketing campaigns to such subscribers to generate more sales.
These promotional pop-ups can be triggered automatically or based on user behavior upon certain events.
For example, an automatic pop-up would appear after 10 seconds of the site loading. Example of a pop-up based on user behavior: exit intent pop-up — this appears just when the user hover overs cursor near the exit tab on the browser.
A/B Test Promotional Pop-Ups: How Does it Work?
Let’s take a hypothetical example to understand this.
If you have a promotion pop-up but aren’t getting enough signups, you could A/B test the pop-up offer: “10% discount on your first order” vs. “$10 cashback on your first order”.
As you can see in the example, the variant pop-up version achieves 10% conversion rate compared to the original version, which got 5% conversion rate. Thus, you can say that the pop-up A/B test was successful as we found a winning variant.
Similarly, you can conduct various types of A/B tests for your promotional pop-up by experimenting with different variables/elements, e.g., the pop-up design, copy, image, CTA button, and so on.
Why A/B Test Promotional Pop-Ups?
Here are some of the key benefits of A/B testing and enhancing your promotional pop-ups:
-
Optimize your pop-up for conversion
A pop-up is an important tool in your e-commerce store. It helps increase the conversion rate as well as strengthen your email marketing game. So, if you’re not getting enough signups through your pop-up, you can optimize it through A/B testing.
-
Gain email subscribers
You can perform A/B tests that are specifically designed to increase the number of signups through your pop-up. The more visitors you get to interact with your pop-up, the more you’ll find email marketing subscribers.
-
Increase your sales and revenue
When the promotional pop-up is done right, it boosts your sales and revenue. And when your pop-up fails to achieve these goals, A/B testing comes in as a handy tool to identify and fix the issue.
-
Enhance the user experience (UX)
While pop-ups are beneficial for your store’s revenue, they can also annoy some customers. Using A/B testing, you can enhance the design of your pop-ups from a UX perspective.
How to A/B Test Promotional Pop-Ups in Shopify
We’ll cover two different methods to A/B test your Shopify pop-ups:
-
A/B test a pop-up using GemPages & GemX
-
A/B test a pop-up using a dedicated pop-up app
Now, let’s go through the step-by-step guide for each method:
Method #1: A/B Test a Pop-Up Using GemX
GemX is built specifically for Shopify merchants to help them increase conversion rate with advanced A/B tests. We’ve built GemX with our years of expertise and experience in the Shopify CRO domain.
You can install GemX and get 14-day free trial!
Step 1: Define the Hypothesis for the test
First things first: your A/B test has to start with a hypothesis.
The hypothesis defines what exactly you’re going to test and why.
Let’s understand this with a sample hypothesis:
-
Theory: Based on the low pop-up signup data of the last six months, we believe that promoting a flat $10 discount instead of a 10% discount to all new visitors will entice them to sign up for the newsletter/email marketing through the pop-up.
-
Validation: We will know this when we see an increase in the signup rate of the promotional pop-up and an increase in the email subscriber count.
-
Outcome: This will be good for our business because it will help us market our products to a larger audience and increase our overall revenue.
You can use this template to create your own A/B test hypothesis.
Step 3: Create Two Versions of Your Template
Now that you have the hypothesis ready, it’s time to create a variant pop-up that would display the $10 flat discount.
For example, here’s our original (Control) landing page that has a promotional pop-up with 10% discount offer.
This pop-up is designed with the GemPages Landing Page Builder app, one of the most loved Shopify page builder apps with a rating of 4.9 out of 5 stars from over 3,400 reviews.
Using GemPages, we also created the variant that promotes a flat $10 discount offer.
Pro Tip: As an A/B testing best practice, you should change only one variable/element at a time to precisely analyze the impact of the change. If you change more than one variable in a single test (e.g., pop-up image + CTA button), it becomes almost impossible to identify which change impacted the results.
Step 4: Configure and Run the A/B Test
So, we have both control and variant versions ready to be tested.
Now, go to the GemX app’s dashboard and click on the “Create new experiment” button.
You’ll see two options for the types of A/B test you could perform with GemX:
-
Template Testing: This A/B testing feature lets you test a single-page template. Thus, you can A/B test a certain element on the page template as defined in the hypothesis.
-
Multipage Testing: As the name suggests, “Multipage Testing” covers a broader scope with multiple pages. Thus, GemX delivers the capability to A/B test two complete store experiences.
For this scenario, we’ll go with the “Template testing” feature because we’re just testing a single page template.
Next, select the “Control” and “Variant” templates inside the GemX experiment. If you’re using GemX along with GemPages, you could create a variant GemX by clicking “Create Variant based on Control”.
Next, GemX allows you to set up advanced conditions to make your A/B test more specific based on some important criteria.
For example, since we’re testing a promotional pop-up, we can cover “New” visitors instead of all visitors (New + Returning). Also, we can exclude the email and SMS traffic because those are already our subscribers.
By any chance, if you still need help, GemX provides help options like the “Setup guide”, “Chat with us”, and “Book a call with us”.
Once everything’s set up, click on the “Start experiment” button, and your test will be live immediately.
Step 5: Analyze the Test Results and Implement Changes
Now, monitor your test as you start getting traffic to the store/landing page. You can click on “More actions” to view analytics and end the experiment.
Before you end the test, you need to make sure of two important aspects:
-
Statistical significance of 90-95%
For your A/B test to be considered reliable, you need to ensure that you’re achieving 95% of statistical significance (or 90% at a minimum). This is required to validate that the test result is not achieved by fluke.
-
Required timeframe of at least two weeks
You should run the test for at least two weeks to gather reliable insights that are not influenced by an occasional spike or dip in the conversion rate.
Once you’ve taken care of these important criteria, you can end the test and declare your winner. If the test is successful and you’ve found the winner, you can implement the changes and continue monitoring its performance.
Pro Tip: You may not always find the winner template with the first experiment — and that’s okay. You could continue the tests by experimenting with different elements until you find the winner.
Step 6: Document the A/B Test Insights
Regardless of the test results, you should always make it a practice to document the observations and insights generated through your tests.
Whether the test was successful or a failed one, you’d still find observations about customer behavior. These insights could be quite helpful in your future marketing campaigns.
You could use a database tool like Notion to save and organize all your test results along with observations in a centralized database.
Method #2: A/B Test a Pop-Up Using a Dedicated Pop-Up App
Now, let’s discuss an alternate method, as some brands use a dedicated pop-up app.
Side note: Even if you’re using another third-party pop-up app, in some cases, GemX can still help you with a workaround to run a pop-up A/B test. So, you may feel free to reach out to the GemX team to figure this out.
Step 1: Choose and Install the Pop-Up App
First of all, you need to ensure that your pop-up app has the A/B testing feature.
For this tutorial, we’ll use one such app — Popupsmart. The app has a rating of 4.9 out of 5 stars from 70+ reviews.
Once the app is installed, you need to enable the app embed inside your Shopify theme.
Step 2: Define the Hypothesis for the Test
Once again, the process of creating the hypothesis will remain the same.
Define the theory, validation, and outcome based on your business scenario, goals, and areas of improvement.
Step 3: Create Your Promotional Pop-Up
Go to the Popupsmart dashboard, and click “Create first campaign” (or alternatively, you can click on the “Create popup” button on the top right corner).
Create your pop-up with Popupsmart’s visual editor.
Once you have your original (Control) pop-up ready, you can create its variant version as well.
Step 4: Configure and Run the A/B Test
While Popupsmart has a free plan, in order to run an A/B test on your pop-ups, you need to have the Advanced plan, which is $69 per month.
In “Popup Campaigns”, you’ll find all your pop-up campaigns. You can create an A/B test for the respective pop-up campaign.
Step 5: Analyze the Test Results and Implement Changes
Once you’ve published the A/B test campaign, the process remains similar to what we discussed in Method #1. Monitor the performance of your variants and declare the winner once you’ve achieved statistical significance and allowed the required timeframe (minimum of two weeks).
Step 6: Document the A/B Test Insights
Last but not least, document the observations and insights from your A/B test.
And that’s it! We’ve covered both methods, and you can use either one depending on your tech stack and requirements.
Final Thoughts on A/B Testing Promotional Pop-Ups
Finally, we’d like to leave you with this reminder — use promotional pop-ups, but mindfully.
As we mentioned at the beginning of this article, you don’t want your visitors to leave your website due to an annoying pop-up. Monitor how your site visitors interact with the pop-up.
Based on customer behavior insights, you can use A/B testing to enhance how the pop-up plays its role in the overall shopping experience.
And don’t forget to install GemPages and GemX, if you haven’t already, to boost the effectiveness of your store design, CRO, and A/B testing campaigns.
To learn more about other eCommerce marketing strategies, tools, and best practices — check out more resources on the GemPages Blog. Also, join the GemPages Facebook community to network and learn from like-minded entrepreneurs and experts.
FAQs About A/B Testing Promotional Pop-Ups
