Tuesday, December 17, 2013

10 Steps For Running A/B Tests On Mobile Apps


There are a number of mobile architectures that support effective A/B testing within mobile apps. They range from rapid prototyping ones based on HTML5 components to feature flag based ones that trigger different versions of native components. The trade-offs are between in-app performance, testing iteration time and the native look and feel within the app. The main concern for effective A/B testing is to produce as many valid experiments as possible in the shortest amount of time. Therefore the longer this process takes, the longer it will take to discover what version(s) of the app perform best for various user segments. Whichever strategy is used, A/B tests should not be dependent on infrequent App Store releases to be the most effective.

After setting up a new A/B testing framework, its important to run an A/A test and determine if it is calibrated correctly. This type of A/A test should also be run every so often to make sure the A/B testing framework still works as expected and produces the correct statistical results.

Once a basic A/B testing framework is setup, here are the steps to run an effective A/B test:
  1. Define a goal that can be accurately measured. The effort in this step will reap dividends later in reducing the number of failed or ineffective tests. 
  2. Brainstorm ideas for how to satisfy the goal. These can come from a variety of places such as qualitative customer feedback, employee suggestions, behavioural economic theories, gut feelings about product improvements, etc. 
  3. Prioritize the list of ideas above based on the ease of implementation, the estimation of improvement potential and the relative position in the funnel. 
  4. Setup the necessary event-based analytics tracking for an individual user's flow through the entire app. These events should be wired together to produce a funnel so that it is clear what the conversion rates are at each step. Depending on what is being tested, the user’s flow should begin from their entry point in the app (direct launch, push notification or website launch) through to the point of purchase and/or post-purchase follow-up. Another important strategy is to measure not only the success of the step being tested, but also the overall engagement of a user. 
  5. Capture a baseline set of metrics for how the app currently performs for various user segments before any testing is run. 
  6. Build the minimum viable test (MVT) and make sure to test it with a small set of beta users prior to releasing it in order to validate the initial metrics. 
  7. Decide on the proportion of users that will be exposed to the A/B test (e.g. new users, returning users, users who haven't purchased yet, 10% of all users, etc.) 
  8. Run the A/B test until the results become statistically significant for the required confidence level (usually 95%). Also ensure that the A/B test occurs during a time period that is considered "usual" activity (e.g. don’t A/B test on a Sunday if users don’t often purchase on a Sunday). 
  9. Calculate which version of the test performs better. If the newly tested version is superior, make it the default version of the mobile app and release it into production for all users. 
    • If the newly tested version either performs poorly or no conclusion can be reached, record the details and possibly re-assess later. 
  10. Observe any other tangential effects that the A/B test may have caused such as increased support calls/emails, decreased retention, engineering complexity, etc. It may also be helpful to present some users with a brief survey asking them about their new experience in the mobile app. The results from this survey will add valuable qualitative feedback to the A/B test’s quantitative results. 
  11. Repeat the process by running another A/B test.
Ultimately, executing A/B tests is about simplicity and speed. The faster the tests can be run and statistically significant winners declared, the more growth a product will see over time. 

The steps given above for running A/B tests relate to users who have already downloaded the mobile app. A/B testing can also be performed on users coming from specific growth channels. Due to mobile's inherently closed ecosystem, attribution is more complicated on mobile apps. However once it is setup correctly, it is possible to track users from specific growth channels so that each channel’s revenue potential can be calculated and optimized.

Monday, November 11, 2013

Xamarin Features RESAAS Mobile App













One of the things I am passionate about at RESAAS is our mobile app for iPhone and Android. We are often exploring how our customers use the app differently from the browser experience and then optimizing the experience for that exact use case.

Xamarin, the company behind the cross-platform development framework that uses C#, recently featured the RESAAS App on their website: http://xamarin.com/apps/app/resaas_the_real_estate_social_network.

I've written previously about our App being showcased on the Appcelerator Titanium blog (when we used their framework instead of Xamarin) as well as our initial app release back in April 2013.

Sunday, November 10, 2013

A Growth Hacking Case Study on Starbucks SRCH


In 2011, while working at Blast Radius, a global digital agency, I was responsible for the technical development of the 'Starbucks SRCH Scavenger Hunt'. The following video describes the campaign.

video

Given my recent venture into the world of growth hacking and the way it now informs my thinking, I took another look at the Starbucks SRCH Scavenger Hunt from a growth hacking perspective.

Here, I will present it as a retrospective case study using the publicly available data. Here are the numbers quoted by Blast Radius in their post:
  • 7000 Starbucks locations advertised the initial QR code for launch
  • 300k visits over 3 weeks
  • 23k registrations (97% played at least one clue)
  • Avg. time for 1st person to solve a clue was 21 min, indicating extremely high engagement with the brand
  • Over 20k posts from social channels regarding SRCH
  • Media coverage from Mashable, USA Today, CNN, PSFK and more

1. Use a Simple Framework

I've posted before about Dave McLure's Startup Metrics for Pirates: AARRR or Chamath Palihapitiya's growth framework. Neil Patel and Bronson Taylor have also created an even simpler three stage framework influenced by Dave's ideas: Get Visitors, Activate Members and Retain Users. Either framework can be used to independently measure and analyze each stage that a user progresses through as they go from having never heard about the product to being fully engaged and possibly paying for a premium version. In this case, I'm choosing to use Chamath's four stage growth framework as it ignores the revenue stage (due to Facebook's business model which makes sense for SRCH as well because it was also a free product):









2. Start Acquiring Users

Paid media was not used for this project so all inbound traffic for SRCH's acquisition (300k visits) came from the following three sources: 
  1. Existing Starbucks Customers (via their 7000 retail locations)
  2. Traditional Media (Mashable, USA Today, CNN... etc)
  3. Social Media (Mostly Twitter & Facebook)
















Things To Consider:
  • Unique Users: Using "visits" to quantify the acquisition stage is ill-advised. Visits, page views, downloads... etc are usually just vanity metrics and were most likely quoted in this specific instance to bolster numbers. What should be measured at this stage is the exact number of unique users to the landing page(s).
  • Conversion Rates: According to Terifs data analysis, Starbucks had, on average, somewhere around 500 daily customers at their retail locations in 2010-2011. Given this insight, if 7000 retail locations had approximately 500 daily customers over a 2 week period while they might have advertised the SRCH Scavenger Hunt, then there was a potential audience of 49 million customers (without factoring in repeat customers which might  in fact be quite high). As an example, if we split the 300k visits three ways across each acquisition channel (stores, traditional media & social media), then we can estimate that the Starbucks locations brought in approximately 100k visits alone. Thus 100k visits/49M potential customers translates to a conversion ratio of only 0.2%. It is interesting to consider that this is in line with conversion rates for digital display advertising (i.e. banner ads) which are known to have very low click-through rates (CTR) compared to other advertising methods. So when thinking about the logistics and development costs required to setup advertising across 7000 Starbucks stores coupled with the conversion rate of those in-store ads, which approximate the conversion rate of banner ads, it may have been more beneficial to spend time optimizing the in-store advertising of SRCH or switching to paid media to drive those visitors to Starbuck's landing pages.

3. Measure Each Stage In Detail

One of the most valuable things to do in any project is to measure each stage along the growth framework (a.k.a. funnel) and figure out the conversion rate at each stage. This shows where users are dropping off and also allows segmentation of the traffic/users so that insightful questions can be asked like "Which types of users are activating more often?" or "What source did our most engaged users come from?" or "Where should we start optimizing first?".


NOTE: The only data available is from Blast Radius may not be accurately measuring the most representative proxy for each stage. 

Here are some things to consider when building these types of funnels and analyzing the results:

  • Counting Conversion: The funnel should be measuring each user independently and any action they perform should only be counted once. Thus, if a single user sent out multiple social media posts, the virality stage should only count one of those social posts since that user initially "converted" to that stage of the funnel (i.e. converting multiple times is still just a single conversion). The reason this immediately stood out to me was the 87% conversion from the engagement to virality stage. From my experience, this number is quite high, and I assume that it measures the number of total social posts but not necessarily the ones from engaged users only.
  • Defining Engagement: The engagement stage took into account whether the "user played at least once", which may or may not be the right proxy for what should be considered an engaged user. Measuring engagement is by far the hardest stage to measure and each business should measure it differently and constantly re-assess whether they are measuring the right thing. Many industry leaders have discovered what their leading indicators of engagement are, but these are hard to figure out without a comprehensive understanding of the customer and tested theories based on data analysis.
  • Funnel Creation: Given the growth framework above it is very helpful to map each stage to a funnel step in an event based analytics tool such as Mixpanel or Kissmetrics. I've written a post before about using Dave McLure's AARRR framework with Mixpanel but here is a mocked-up version of the growth framework above mapped to a Mixpanel Funnel:


3. Optimize The Funnel

Given the data above, the best place to start optimizing would be higher up in the funnel where the largest drop-off was experienced (i.e. landed users who don't sign-up). The reason for this is that a one percent increase in signed-up users has a much larger effect on the overall completion rate than the same percentage increase in engaged users. One thing to be careful of with this approach is that diminishing returns start setting in the moment you begin optimizing a step. At some point the effort required to discover a change that has a tangible effect is no longer worth the cost. Here are some ideas that could have been used for optimizing each step of the funnel:

  • Optimizing Acquisition: Inbound traffic came from 3 channels as mentioned above. Figuring out which of those channels brought in the "best" users (most highly engaged) using Mixpanel's segmentation features (or Google Analytics), could focus efforts by reallocating resources to focus on the acquisition channel that performed the best and had the greatest potential for increases. For example, optimizing the retail in-store advertising about SRCH during the Scavenger Hunt would have been complex (in terms of logistics and timing to rollout any changes) but this could be tested at a single store and if sufficient increases were noticed to justify changes across the other 7000 stores, the improved advertising could be rolled-out. Essentially testing a variety of in-store combinations of advertising placements, colours, QR codes vs. actual links... etc. could be rapidly performed to see what single or set of changes drove more traffic.
  • Optimizing Activation:  The conversion page could be A/B tested (using something like Optimizely) for activation to determine if there are any changes that would boost sign-ups. Social sign-up, wording, images, colours, layout can all be A/B tested provided there is enough inbound traffic to support the tests. (See Neil and Bronson's suggestions for conversion growth hacks). Changes should be statistically significant, as measured with a A/B split test calculator.
  • Optimizing Engagement: This is the core of a user's experience. As can be seen, there are a number of steps that a user must go through to get to this point but once they are here they should be given what some call a "must-have experience"or "aha-moment" if they are ever to come back and continue to use the product. Not having this is the difference between whether or not the product has a product-market fit. Without it, no growth hacking will be that effective over time as the product will just bleed users over and over again until there are no more users left to acquire. Therefore, optimizing for engagement comes only after product-market fit has been found. If there is a clear understanding of how users are engaging with the product and there is a desire to boost engagement, a number of tactics are available. For the Starbucks SRCH Scavenger Hunt, email, SMS or push notifications could be used to alert users when the next clue has been released or when the first user solves a clue. SRCH was a game after all so building in a gamification system built upon competing users could boost engagement with existing users.
  • Optimizing Virality: Increasing the amount of users who post something about the product to their social graph requires trust, a value proposition and reducing friction. Thus, testing a number of combinations such as where in the flow should the user be prompted to post, what copy should be used to encourage a user to post and what copy should be used for the auto-populated post text. Additionally adding in some clear value added benefit (i.e. exclusive access, more game features... etc) for the user posting could also increase the number of user's who decide to post something to their social graph.

Friday, September 6, 2013

Examples of Mobile First Development

We all know that developing for mobile is different than developing for the desktop but how exactly is it different? I'm deeply interested in how a mobile product needs to be structured in a fundamentally different way than a desktop product in order to thrive. It cannot simply be a slimmed down, feature minimal version of a the desktop or tablet version. It should not feel as if its missing critical features or useful add-ons simply because they couldn't fit into the mobile format.

Various concepts have emerged for how to approach mobile app development. One revolves around mobile apps being remote controls for real life. Another is about mobile apps being useful for a user while they wait for something (i.e. in a line-up or in an elevator). In order for that to happen a user should be able to launch the app and perform some task within 30 seconds to a minute. MG Siegler of CrunchFund also had this to say about building for mobile:
"Don’t build an app based on your website. Build the app that acts as if websites never existed in the first place. Build the app for the person who has never used a desktop computer. Because they’re coming. Soon."
Some companies have built very compelling business models that fit well with this mobile first, quick and effective/remote control paradigm. Users are responding well by engaging with and being retained by these mobile apps due to their simplicity. The following examples show how mobile apps can reduce inherently complicated tasks down to very simple actions which are fundamentally different from anything we've seen previously on the desktop.

1. Hotel Tonight


Hotel Tonight is a mobile-only app where users can book last-minute hotel deals. Hotel Tonight has simplified booking a hotel room down to only a few essential actions without degrading the experience to the point where it feels limited. Here is the user flow from initial launch through to booking confirmation:



1. During launch the mobile app retrieves the hotels on offer given the user's current location. Although this may take a few extra seconds, it presents the user with the exact information they are interested in right when the app loads (the user is not required to type anything).

2. The user is then presented with the hotel selections in a scrollable list with the 3 most essential details displayed: photo, price & location. The app also provides some other useful data like the type of hotel experience (Solid, Basic, Luxe, Charming, Hip) and a rating by other Hotel Tonight guests. This information makes it very easy to select an appropriate hotel for the night.

3. Once a hotel has been initially selected, some further details can be reviewed such as additional photos, information about the hotel itself and its exact location on a map along with the final price.

4. The final screen allows the user to easily confirm the dates, price and credit card to be used for the transaction.

From start to finish this process only requires 3 simple actions: a single hotel selection, an initial booking of the room and finally a confirmation of the booking details. Hotel Tonight has given travellers the ability to easily choose and book a hotel room from their mobile phones. The whole process from start to finish feels uninhibited by the mobile form factor it actually thrives within it. Its because of this that the mobile app thrives and continues to delight users.


2. Car2go


Car2go is a vehicle sharing service paid for by the minute where vehicles can be picked-up and dropped off in different locations. The primary way to find and reserve a vehicle is via its mobile app. Given the nature of the service offered, it needs to take seconds (not minutes) to book a vehicle via the app. Here is the user flow from initial launch through to vehicle reservation:


1. The mobile app launches fairly quickly, determines the user's current location in order to position the map correctly and starts retrieving vehicle locations. Again this presents the user with the exact information that are looking for without any interaction after launching the app.

2. Vehicles begin populating on the map and the user can then zoom in and out to find the vehicle closet to them or simply select a vehicle.

3. Once a vehicle has been selected, the blue vehicle marker expands to show 3 additional bits of information: license number, distance away and gas available (indicated as a percentage). Although this information may be helpful in some circumstances it can be presented in a different and this third step can potentially be removed completely:
  • The license number is unimportant to the vast majority of users unless a user has forgotten something in a vehicle and is trying to find it (but this is a very rare case).
  • Knowing the distance to a vehicle isn't as helpful as knowing the approximate time it would take to walk. A vehicle may be 562m away but how long would that take if the user walked there? A separate interaction with the marker (such as a "2-second hold") could display the walking time from the user's current location therefore removing the need for the distance measurement in the expanded marker.
  • The gas available is the most helpful but it could be more easily displayed so that it can be compared with all other vehicles. By displaying something visually on each vehicle's non-expanded marker (possible a textual percentage of even a level indication), this again would avoid the need to click on the marker to find out how much gas is available.

4. The final screen allows the user to review the vehicle's gas available, cleanliness, street address before confirming the reservation.

Again from start to finish this process only requires 3 simple actions: a single vehicle selection, reviewing some of the vehicle's specific information and finally a confirmation of the reservation details. As outlined above this 3-step process can be further simplified to just 2 steps by removing the expandable marker step and simple going straight to the reservation confirmation screen. Having said this, even with 3 steps this mobile app by Car2go is very efficient to use - a vehicle can be reserved in under 30 seconds.

3. Uber


Uber seamlessly connects a user needing transportation with a taxi driver. Uber, the company, does not own any vehicles and does not have any drivers on staff, instead they provide ride-logistics to both users and drivers in order to match supply and demand more effectively. Ultimately Uber wants to get users a taxi in the shortest amount of time and give them the best experience while helping drivers anticipate demand and therefore maximize the earnings per shift. Here is the user flow from initial launch through to taxi request:



1. During launch the mobile app determines the user's location in order to display the correct map along with taxis on the following screen. 

2. Taxis begin populating on the map but they also update in real-time as taxis move about through the streets or get requested by other users and are no longer available. This gives the user immediate feedback about each taxi's relative speed and direction along with the approximate supply of taxis in the given area. All a user has to do is position their pickup location, review the approximate wait time and then tap the "SET PICKUP LOCATION" button.

3. The final screen allows the user to review the pickup location, credit card to use and approximate wait time before confirming the request.

From start to finish this process only requires 2 simple actions: choosing a pickup location and confirming the taxi pickup request. Uber has made the process of booking a taxi on their mobile app as simple as possible and it works phenomenally well for users.

Thursday, September 5, 2013

Building Product Roadmaps with Kanban

The following back-of-the-napkin drawing is a great illustration for how to think about aligning an entire organization around building the best product(s). Kanban fits well with this illustration as it can be used continuously to build an evolving product roadmap with the aid of everyone across the company in a repeatable and consistent way.


Here are the major steps:
  1. Source: New product ideas should be sourced from anyone inside the company. Anyone from any department should be able to provide small or large ideas that could eventually make their way into the product.
  2. Filter: Have a single person (usually the product manager) filter out and prioritize what ideas should be worked on. Filtering should be performed methodically based on both quantitative and qualitative measures so that the same selection criteria can be user over and over again.
  3. Recycle: Compost all the ideas that weren't selected and provide specific reasons for why they weren't selected. This openness and transparency helps the originator of the idea(s) understand the how to improve future ones so that they are more likely to be selected.
  4. Display: Display what product ideas will be worked on and in what order. This is where a Kanban board can come in very handy. Everyone in the organization should have access to this board where they can see the progress of previously selected product ideas, who is working on them and potentially add additional requirements as they come up.
The key is to make all of these steps publicly visible within the company and give everyone the ability to comment, argue and build on existing ideas. Kanban is about transparency, it gives everyone in an organization the knowledge about what ideas are being proposed, which ones are being selected and dropped, the reason why that happened and, finally, what should be worked on in the near future. This very simple yet transparent process helps to align an entire organization towards a common goal via a repeatable and consistent process.