How a one word change increased product demo conversions by 139%

The following is a repost from the GoCardless Blog where I explained one of our successful A/B tests. I also started a good discussion about the merits of the A/B test over at growthhackers.com.

---

This post looks at an A/B test with a simple copy change and how it improved conversion rates by 139%. The idea behind the A/B test was to give users immediate access to the product, via a recorded demo, instead of having them receive a personal phone call from our sales team later on.

Theory

Using a framework, when A/B testing potential improvements to landing pages, is helpful. Sean Ellis has a simple one for understanding the broad levers that can been affected: Conversion Rate = Desire - Friction. Desire and Friction can be further broken out and the LIFT Model by WiderFunnel describes this well:

Hypothesis

Our goal was to improve the conversion rate for demo requests so that customers could access the content they were interested in as soon as possible and with minimal friction. We wondered if the “Request a demo” button might be causing some users anxiety (as described by the LIFT model) and, as such, artificially lowering conversion rates. We then tested whether the wording “Watch a demo” would outperform the original “Request a demo” wording.

We had further reason to believe that immediate access to a recorded demo would be beneficial as only 1/5th of leads ended up watching the live demo which has been scheduled by our sales team.

Website Modifications

The original GoCardless user experience with the “Request a demo” was the following:

• Call to action (CTA) on the homepage was “Request a demo”
• Users were taken to a request a demo form to fill out and submit
• Upon completing the form, users were given a date and time of an upcoming live demo that they could participate in.

We altered the user experience by giving users immediate access to a recorded demo:

• Call to action (CTA) on the homepage is “Watch a demo”
• Users are taken to a “Watch a demo” form to fill out and submit
• Upon completing the form, users are shown a 10 minute recorded demo in their browser

Most of these changes were implemented within Optimizely’s Multi-page Experiments feature (aka Conversion Funnel Testing). However, we did build the recorded demo page ourselves as we didn’t need to A/B test this page directly.

Although some time and effort went into thinking about reducing friction, the work required to implement and instrument these changes was very easy because of Optimizely and Mixpanel. Optimizely has a very useful single toggle option for sending super properties to Mixpanel so that we can track what happens deep within our funnel.

Results

Given our acquisition channel characteristics, we ran the A/B test for a full 7 days. We then looked for a statistically significant winner with at least a 95% confidence level. Optimizely’s report panel below shows that the “Watch” version consistently outperformed the original version:

We also ran the numbers through Mixpanel’s split test calculator based on event data we track in our conversion funnels:

This shows that the “Watch a demo” version is more than twice as effective as the “Request a demo” version (139% increase in conversion). With this simple copy change, derived from the idea of reducing friction for new users, we’ve dramatically increased the number of users who watch a product demo and are therefore more likely to become customers.

10 Steps For Running A/B Tests On Mobile Apps

There are a number of mobile architectures that support effective A/B testing within mobile apps. They range from rapid prototyping ones based on HTML5 components to feature flag based ones that trigger different versions of native components. The trade-offs are between in-app performance, testing iteration time and the native look and feel within the app. The main concern for effective A/B testing is to produce as many valid experiments as possible in the shortest amount of time. Therefore the longer this process takes, the longer it will take to discover what version(s) of the app perform best for various user segments. Whichever strategy is used, A/B tests should not be dependent on infrequent App Store releases to be the most effective.

After setting up a new A/B testing framework, its important to run an A/A test and determine if it is calibrated correctly. This type of A/A test should also be run every so often to make sure the A/B testing framework still works as expected and produces the correct statistical results.

Once a basic A/B testing framework is setup, here are the steps to run an effective A/B test:

1. Define a goal that can be accurately measured. The effort in this step will reap dividends later in reducing the number of failed or ineffective tests. 
2. Brainstorm ideas for how to satisfy the goal. These can come from a variety of places such as qualitative customer feedback, employee suggestions, behavioural economic theories, gut feelings about product improvements, etc. 
3. Prioritize the list of ideas above based on the ease of implementation, the estimation of improvement potential and the relative position in the funnel. 
4. Setup the necessary event-based analytics tracking for an individual user's flow through the entire app. These events should be wired together to produce a funnel so that it is clear what the conversion rates are at each step. Depending on what is being tested, the user’s flow should begin from their entry point in the app (direct launch, push notification or website launch) through to the point of purchase and/or post-purchase follow-up. Another important strategy is to measure not only the success of the step being tested, but also the overall engagement of a user. 
5. Capture a baseline set of metrics for how the app currently performs for various user segments before any testing is run. 
6. Build the minimum viable test (MVT) and make sure to test it with a small set of beta users prior to releasing it in order to validate the initial metrics. 
7. Decide on the proportion of users that will be exposed to the A/B test (e.g. new users, returning users, users who haven't purchased yet, 10% of all users, etc.) 
8. Run the A/B test until the results become statistically significant for the required confidence level (usually 95%). Also ensure that the A/B test occurs during a time period that is considered "usual" activity (e.g. don’t A/B test on a Sunday if users don’t often purchase on a Sunday). 
9. Calculate which version of the test performs better. If the newly tested version is superior, make it the default version of the mobile app and release it into production for all users. 
• If the newly tested version either performs poorly or no conclusion can be reached, record the details and possibly re-assess later. 
10. Observe any other tangential effects that the A/B test may have caused such as increased support calls/emails, decreased retention, engineering complexity, etc. It may also be helpful to present some users with a brief survey asking them about their new experience in the mobile app. The results from this survey will add valuable qualitative feedback to the A/B test’s quantitative results. 
11. Repeat the process by running another A/B test.

Ultimately, executing A/B tests is about simplicity and speed. The faster the tests can be run and statistically significant winners declared, the more growth a product will see over time. 

The steps given above for running A/B tests relate to users who have already downloaded the mobile app. A/B testing can also be performed on users coming from specific growth channels. Due to mobile's inherently closed ecosystem, attribution is more complicated on mobile apps. However once it is setup correctly, it is possible to track users from specific growth channels so that each channel’s revenue potential can be calculated and optimized.