Introducing A/B Testing for Pack + Hydrogen Storefronts

Get the full rundown on Pack's native split-testing feature and how marketers are using it to drive conversions and enrich their site experiences.

For marketers, looping in a developer to conduct even the most basic A/B tests can be frustrating—and even with a developer involved, there tend to be limits on what you can test. Our new A/B testing feature for Pack + Hydrogen storefronts eases this frustration, benefiting marketers and developers alike.

Pack's native A/B testing feature empowers marketers to identify content that captivates and converts without impacting site performance or relying on continuous developer assistance.

Unlock the benefits of streamlined A/B testing on Shopify Hydrogen

Pack's A/B testing feature is far from a minimum viable product for Shopify marketers. We combined Hydrogen's powerful server-side content rendering with Pack's drag-and-drop editor to deliver a blazing-fast, integrated experience that other Shopify split testing tools simply can't match. 

Let's take a closer look at the benefits of Pack's split testing feature.

Pre-rendered content maintains site speeds during testing

Why do A/B tests on a Shopify theme often slow down page load times—which impacts conversions and muddies test results?

Most A/B testing tools for Shopify themes load the testing experience when content starts loading on your shoppers' device (client-side). With Shopify Hydrogen, your content and test data load on the server, before data is even sent to your shopper's device. This significantly speeds up load times — resulting in higher conversions and cleaner test results.

Server-side testing can reduce the risk of bugs

Injecting an A/B testing script onto your theme can be prone to bugs that can be difficult to resolve and negatively impact the customer experience—especially when they conflict with other apps. To resolve these bugs, a developer generally gets brought in, reducing their bandwidth for other vital tasks like page building and optimization. With Pack’s server-side A/B testing, in-depth testing and optimizations don’t are less likely to impact your site functionality. 

Vast testing opportunities improve the customer experience

With Pack, you can test every element or feature on your site from copy, images, fonts, navigation — even your free shipping meter. If it's on your site, Pack can help your team test it without friction. Our visual editor and native tracking tools make it easy to create page variants and track events without developer assistance. 

Reduce reliance on third-party apps

Shopify apps provide immense value, but using too many slows down your site. The more apps you add, the higher your monthly costs and the more time your team spends acclimating to different tools. A/B testing is native to Pack, so it reduces your need for third-party apps. What’s more, even the most non-technical team members can use it efficiently. 

How Pack's native A/B testing feature works

A/B testing on Pack + Hydrogen storefronts takes just five simple steps:

1. Set up your test

In Pack's admin, you can create a new test, specify a title, handle, description, and target your test to a specific audience—using a URL, query, path, or UTM parameter.

You can add an unlimited number of test variants, and set a goal for your test. Finally, you can determine how much site traffic you want to send to each variant. To ensure the test is running smoothly, we recommend starting small — directing 1% to 10% of users to your new variant(s). If no red flags arise, gradually increase the amount of traffic until you reach an even 50/50 split.

Note: Although you can add unlimited test variants, we recommend using only one variant at a time. This will give you the cleanest test data possible and reduce the time spent uncovering actionable results.

2. Edit page variants

With Pack's visual Customizer, you can toggle on testing mode, flip between your test variants, and make edits to your test visually. Whether you're testing copy, imagery, or toggling features on and off, Pack allows you to quickly test anything that you can edit within the Customizer.

3. Preview & run your test

After previewing your test variants, you can turn on a test with a click, and pause it at any time.

4. Analyze your results

With your testing infrastructure in place, it's time to watch the data flow into Pack's admin, and/or your preferred analytics platform. You'll want to continue running the test until you've reached a statistically significant data pool (around 1,000 site visitors at minimum with a 90% interaction rate). Then, you can identify the variants with the highest conversion rate without worrying about the validity of the results.

5. Ship your winning page variants

Once you know which variants were most successful, it's time to conclude the test, promote your winning variant, and watch your metrics soar. 

Split testing on a Shopify theme vs. a Pack + Hydrogen storefront

If your team is considering the switch from a Shopify theme to a Pack + Hydrogen storefront, you’re probably wondering how the testing experiences differ. So, let's review the basic mechanics, pros, and cons of testing on a Shopify theme vs. a Pack + Hydrogen storefront.

Split testing on a Shopify theme

Shopify themes are built on Liquid, a development framework that's great for static content and doesn't rely heavily on user interaction.

Pros of testing on a Shopify theme

• Ease of use: Many A/B testing apps can be installed on a Shopify theme with a single click.

Cons of testing on a Shopify theme

• Resource-heavy: A/B testing tools can range from a few hundred dollars per month to a few thousand. Additionally, your team will have to continuously manage the app to keep up with product changes. 

• Not built for dynamic content: While it's possible to build dynamic elements into your Shopify theme, you won't have much control over when and how content loads. Testing is inherently dynamic, so A/B testing tools may negatively impact the customer experience on a theme. For example, page flashes often happen during loading because of how quickly the page needs to refresh to load dynamic content. 

• Bug-prone: The majority of A/B testing tools and Shopify apps rely on scripts that are injected into your theme. Those scripts frequently conflict with other scripts on your site, creating bugs that harm site functionality.

When to use Shopify theme A/B testing

If you aren't running into site speed issues and want to test basic, static visuals, a theme will suit your needs just fine. There are a handful of apps on the Shopify marketplace that are quick to set up and can help you start to collect data and optimize your site experience.

Split testing on a Pack + Hydrogen Hydrogen storefront

Pack + Hydrogen storefronts are built on Remix, a development framework that supports dynamic content. Remix gives you full control over how and when page elements load. Instead of requiring a hard page refresh to change the content on your page, the server can determine what content to load in microseconds, giving each user a truly unique experience.

Pros of testing on a Pack + Hydrogen storefront

• Cleaner test results: No laggy pages, fewer bugs, and no page flashes to impact your customer experience and contaminate test results. 

• Testing freedom: Test any page element or site feature, even dynamic page elements (e.g. carousel images). 

• Simple test creation: Our drag-and-drop visual Customizer makes it easy to create tests, allocate traffic, and turn them on or off.

• Low cost of ownership: Eliminating an expensive A/B testing tool can help you justify the cost of a Pack + Hydrogen storefront.

• Light dev involvement: The only time you’ll need to pull in a developer is to set up event tracking or to test complex site features. From there, testing basic content (images, copy, etc.) can be handled without developer assistance.

Cons of testing on a Pack + Hydrogen storefront

• Hydrogen storefront required: Switching from a theme to a Hydrogen storefront can seem daunting. The good news is that, with Pack, most brands can make the switch in two to three months with no more lift than building a custom theme.

• Beta stage: For now, Pack's A/B testing feature is a work in progress. But we're actively expanding the tool’s functionality so you can take full advantage of the potential for dynamic content. Plus, we're investing in other features — like the ability to serve up personalized copy and images outside of a test.

When to use a Hydrogen storefront for A/B testing

If you're creating and testing dynamic, personalized content, it's time to graduate to a Pack + Hydrogen storefront. With a streamlined customer experience, you'll get cleaner data for highly informed decisions about your storefront.

Ready to take your split testing capabilities to the next level?

Join our waitlist to get access to our A/B testing sandbox, or send our developer documentation to your technical team.

Have questions or have feedback for the team? Reach out to contact@packdigital.com anytime.