Facebook Community
Change Log
Help Center
How to A/B Test Promotional Pop-Ups on Shopify [Using Two Methods]
Pop-ups could be a "game-changer" OR "deal-breaker" depending on how you use them.
Because when used creatively, pop-ups can drive email sign-ups and revenue from promotional emails. And on the flip side, they can annoy your site visitors to the extent that they’d leave your site.
So, how do you do pop-ups right? A/B testing is the solution that can help you find the answer.
In this blog post, we’ll walk you through a detailed guide on how to A/B test promotional pop-ups with two different methods.
Fundamentals of A/B Testing Promotional Pop-Ups
What is A/B Testing in Shopify?
A/B testing is a technique in which two different versions of a webpage are compared by distributing them to different visitors to identify which one performs better for a defined business goal.
For example, you can create two versions of a product page by changing one of the elements and analyzing the impact of that change on your conversion rate.
What is a Promotional Pop-Up in Shopify?
In your Shopify store, you can configure a promotional pop-up that appears on top of the webpage to grab the visitors' attention with a discount offer or any other reward and get them to sign up with their email and/or phone number.
Once the contact details are collected, along with the consent for marketing emails, the brand can start promotional email marketing campaigns to such subscribers to generate more sales.
These promotional pop-ups can be triggered automatically or based on user behavior upon certain events.
For example, an automatic pop-up would appear after 10 seconds of the site loading. Example of a pop-up based on user behavior: exit intent pop-up — this appears just when the user hover overs cursor near the exit tab on the browser.
A/B Test Promotional Pop-Ups: How Does it Work?
Let’s take a hypothetical example to understand this.
If you have a promotion pop-up but aren’t getting enough signups, you could A/B test the pop-up offer: “10% discount on your first order” vs. “$10 cashback on your first order”.
Why A/B Test Promotional Pop-Ups?
Here are some of the key benefits of A/B testing and enhancing your promotional pop-ups:
-
Optimize your pop-up for conversion
A pop-up is an important tool in your e-commerce store. It helps increase the conversion rate as well as strengthen your email marketing game. So, if you’re not getting enough signups through your pop-up, you can optimize it through A/B testing.
-
Gain email subscribers
You can perform A/B tests that are specifically designed to increase the number of signups through your pop-up. The more visitors you get to interact with your pop-up, the more you’ll find email marketing subscribers.
-
Increase your sales and revenue
When the promotional pop-up is done right, it boosts your sales and revenue. And when your pop-up fails to achieve these goals, A/B testing comes in as a handy tool to identify and fix the issue.
-
Enhance the user experience (UX)
While pop-ups are beneficial for your store’s revenue, they can also annoy some customers. Using A/B testing, you can enhance the design of your pop-ups from a UX perspective.
How to A/B Test Promotional Pop-Ups in Shopify
We’ll cover two different methods to A/B test your Shopify pop-ups:
-
A/B test a pop-up using the “GemX: CRO & A/B Testing” app
-
A/B test a pop-up using a dedicated pop-up app
Now, let’s go through the step-by-step guide for each method:
Method #1: A/B Test a Pop-Up Using GemX
GemX is built specifically for Shopify merchants to help them increase conversion rate with advanced A/B tests. We’ve built GemX with our years of expertise and experience in the Shopify CRO domain.
You can install the GemX app with a 14-day free trial.
Step 1: Define the Hypothesis for the test
First things first: your A/B test has to start with a hypothesis.
The hypothesis defines what exactly you’re going to test and why.
Let’s understand this with a sample hypothesis:
-
Theory: Based on the low pop-up signup data of the last six months, we believe that promoting a flat $10 discount instead of a 10% discount to all new visitors will entice them to sign up for the newsletter/email marketing through the pop-up.
-
Validation: We will know this when we see an increase in the signup rate of the promotional pop-up and an increase in the email subscriber count.
-
Outcome: This will be good for our business because it will help us market our products to a larger audience and increase our overall revenue.
You can use this template to create your own A/B test hypothesis.
Step 3: Create Two Versions of Your Template
Now that you have the hypothesis ready, it’s time to create a variant pop-up that would display the $10 flat discount.
For example, here’s our original (Control) landing page that has a promotional pop-up with 10% discount offer.
Alt-text: Pop-up element inside GemPages editor with a 10% discount offer
This pop-up is designed with the GemPages Landing Page Builder app, one of the most loved Shopify page builder apps with a rating of 4.9 out of 5 stars from over 3,400 reviews.
Using GemPages, we also created the variant that promotes a flat $10 discount offer.
Alt-text: Pop-up element inside GemPages editor with a flat $10 discount offer
Pro Tip: As an A/B testing best practice, you should change only one variable/element at a time to precisely analyze the impact of the change. If you change more than one variable in a single test (e.g., pop-up image + CTA button), it becomes almost impossible to identify which change impacted the results.
Step 4: Configure and Run the A/B Test
So, we have both control and variant versions ready to be tested.
Now, go to the GemX app’s dashboard and click on the “Create new experiment” button.
Alt-text: GemX app dashboard
You’ll see two options for the types of A/B test you could perform with GemX:
-
Template Testing: This A/B testing feature lets you test a single-page template. Thus, you can A/B test a certain element on the page template as defined in the hypothesis.
-
Multipage Testing: As the name suggests, “Multipage Testing” covers a broader scope with multiple pages. Thus, GemX delivers the capability to A/B test two complete store experiences.
For this scenario, we’ll go with the “Template testing” feature because we’re just testing a single page template.
Alt-text: A/B testing features inside the GemX app dashboard
Next, select the “Control” and “Variant” templates inside the GemX experiment. If you’re using GemX along with GemPages, you could create a variant GemX by clicking “Create Variant based on Control”.
Alt-text: A/B testing configuration inside the GemX app
Next, GemX allows you to set up advanced conditions to make your A/B test more specific based on some important criteria.
For example, since we’re testing a promotional pop-up, we can cover “New” visitors instead of all visitors (New + Returning). Also, we can exclude the email and SMS traffic because those are already our subscribers.
Alt-text: A/B testing configuration inside the GemX app
By any chance, if you still need help, GemX provides help options like the “Setup guide”, “Chat with us”, and “Book a call with us”.
Alt-text: A/B testing configuration with support options inside the GemX app
Once everything’s set up, click on the “Start experiment” button, and your test will be live immediately.
Alt-text: Starting an A/B test inside the GemX app
