Wednesday 17 September 2014

Improving On-Site Search Functionality For

Back in 2013 when started gaining prominence and was regularly used by thousands of growth engineers, I started noticing that their on-site search functionality needed some improvement as relevant content was hard to find. I knew the small team were working hard to design and deliver important features that were growing the community, so I wanted to help out by articulating the problem and devising a solution that would be easy for them to implement and could almost immediately be leveraged by the community. 

Here is the short presentation I created and sent to Sean Ellis. It was subsequently used to improve their on-site search and is still used today by the community (note that instead of using Swiftype they went with a similar product called Algolia):

Improving Search on from Andrew Thompson

The Swiftype based search for that I setup when I created the above presentation is still working today (give it a try):

Sunday 20 July 2014

Just launched my new company called Insignum

Yesterday I launched my new company called Insignum. It delivers anomaly detection and notification for your analytics across Mixpanel, KISSmetrics and Google Analytics.

The product provides data-driven startups with automated analytics intelligence to find deviations in expected customer behaviour as they happen. Using machine learning algorithms, it continuously searches your data and when an anomaly is detected it will notify you so that you can act on the insight.

If you have any feedback about the product I'd be very interested in hearing about it. Insignum's Twitter account is @insignum_io.

Tuesday 3 June 2014

How a one word change increased product demo conversions by 139%

The following is a repost from the GoCardless Blog where I explained one of our successful A/B tests. I also started a good discussion about the merits of the A/B test over at

This post looks at an A/B test with a simple copy change and how it improved conversion rates by 139%. The idea behind the A/B test was to give users immediate access to the product, via a recorded demo, instead of having them receive a personal phone call from our sales team later on.


Using a framework, when A/B testing potential improvements to landing pages, is helpful. Sean Ellis has a simple one for understanding the broad levers that can been affected: Conversion Rate = Desire - Friction. Desire and Friction can be further broken out and the LIFT Model by WiderFunnel describes this well:

LIFT Framework for CRO


Our goal was to improve the conversion rate for demo requests so that customers could access the content they were interested in as soon as possible and with minimal friction. We wondered if the “Request a demo” button might be causing some users anxiety (as described by the LIFT model) and, as such, artificially lowering conversion rates. We then tested whether the wording “Watch a demo” would outperform the original “Request a demo” wording.
We had further reason to believe that immediate access to a recorded demo would be beneficial as only 1/5th of leads ended up watching the live demo which has been scheduled by our sales team.

Website Modifications

The original GoCardless user experience with the “Request a demo” was the following:
  • Call to action (CTA) on the homepage was “Request a demo”
  • Users were taken to a request a demo form to fill out and submit
  • Upon completing the form, users were given a date and time of an upcoming live demo that they could participate in.

GoCardless Homepage With Request Copy

GoCardless Demo Page With Request Copy

We altered the user experience by giving users immediate access to a recorded demo:
  • Call to action (CTA) on the homepage is “Watch a demo”
  • Users are taken to a “Watch a demo” form to fill out and submit
  • Upon completing the form, users are shown a 10 minute recorded demo in their browser
GoCardless Homepage With Watch Copy

GoCardless Requesting A Demo Page With Watch Copy

GoCardless Watch A Demo Page

Most of these changes were implemented within Optimizely’s Multi-page Experiments feature (aka Conversion Funnel Testing). However, we did build the recorded demo page ourselves as we didn’t need to A/B test this page directly.

Although some time and effort went into thinking about reducing friction, the work required to implement and instrument these changes was very easy because of Optimizely and Mixpanel. Optimizely has a very useful single toggle option for sending super properties to Mixpanel so that we can track what happens deep within our funnel.


Given our acquisition channel characteristics, we ran the A/B test for a full 7 days. We then looked for a statistically significant winner with at least a 95% confidence level. Optimizely’s report panel below shows that the “Watch” version consistently outperformed the original version:

Results From Optimizely Report

We also ran the numbers through Mixpanel’s split test calculator based on event data we track in our conversion funnels:

Results From Mixpanel's Split Test Calculator

This shows that the “Watch a demo” version is more than twice as effective as the “Request a demo” version (139% increase in conversion). With this simple copy change, derived from the idea of reducing friction for new users, we’ve dramatically increased the number of users who watch a product demo and are therefore more likely to become customers.