Saturday, August 19, 2017

Joining Appear Here as CTO


Brand and Product





Appear Here's very clear and focused mission resonates with me because they want to create a world where anyone with an idea can find space to make it happen. They achieve this by bringing together a community of brands, entrepreneurs and creatives in the world's best cities. They already have 80,000+ global brands and creative entrepreneurs, 10,000,000+ sq ft of spaces and 3 offices globally. Brands and entrepreneurs can use the Appear Here platform to find and book short-term spaces such as high street boutiques, market, shopping centres and shop shares in London, Paris and New York.


Company Culture


From the very first interview to signing my contract and officially joining, the team at Appear Here have built a company that continuously upholds these values:


We're Hiring


We're hiring talented people to help make this ambitious vision a reality. We're currently looking for Software Engineers and we will soon be hiring for other product and engineering roles as we grow.  If you're interested in joining me and the rest of the Appear Here team, feel free to reach out to me directly (via Linkedin or Twitter) and we'll have a coffee to chat through what we're up to.

Tuesday, July 11, 2017

Building a "Job Finder" skill for my Amazon Alexa Device

Growth


According to a recent NPR and Edison Research Smart Audio Report, 5% of the 1,620 surveyed respondents owned an Alexa device (Echo or Echo Dot) and 2% owned a Google Home device. Even more revealing was the fact that 42% of Smart Speaker owners said that "their Smart Speakers are essential to their everyday lives". By any stretch, Smart Speakers have reached a significant level of product/market fit and in a little over 2 years Amazon Alexa devices are leading the pack and growing at a rapid pace:

Skill


I decided to build my first Alexa skill to better understand the underlying Alexa Skills Kit that powers all Amazon Echo devices to give users this new intuitive voice interface. I was particularly interested in voice-user interface (VUI) design and how easy it would be to go from idea to building and then deploying a new skill. For the actual implementation, I decided to use the new Flask-Ask framework by John Wheeler to build the skill along with using Zappa to deploy my Python code into AWS Lambda

I settled on a simple idea for a "job finder" skill that allows users to answer 2 simple questions that Alexa asks them about which city and job role they were interested in. The skill would then return the most relevant job listings that had been posted in the last 24 hours according to the answers given. I imagined users waking up in the morning with their coffee in hand and asking their Alexa "job finder" skill to get the latest updates for their ongoing job search. If the skill responded with some particularly timely and relevant jobs, the user could then decide to immediately open their laptop and apply for those jobs or, if not, the user could simply continue to enjoy their coffee and resume the job search later on in the day.

Implementation


I spent a few evenings coding up the "job finder" skill's Python backend and configuring the interaction model (the intent schema and sample utterances) to get it working locally. I ended up with a single job_finder.py file for the bulk of my application logic which is only 116 lines of code and a template.yaml file that contains a number of the templated launch, answer and re-prompt responses which are required for the skill. All-in-all the learning curve was reasonable and most of the learning came from understanding the new type of interaction model set out by the Alexa Skills Kit.

Demo


As of writing this post the "job finder" skill is being reviewed by Amazon's Alexa's team and therefore is unavailable at present to enable on your own Amazon Echo device (the skill should be certified in a week or less). In the meantime I've recorded a demo using Amazon Alexa Testing Tool called Echosim.io:


Learnings


  • I've capped the number of jobs Alexa responds with to a maximum of 5. I think this is a manageable amount but given that Alexa might misunderstand the job role given by the user, it could be frustrating to have to listen to all 5 jobs if they are based on an incorrect interpretation. Therefore I would potentially improve the skill's VUI to either have Alexa repeat the job role specified by the user before all 5 jobs are listed out OR I would pause Alexa after the first job listed and ask the user a yes/no question on whether Alexa should continue or not.
  • I've purposefully kept each job listing very brief so it's easy for users to consume while listening but it might be useful for users to get more detailed information. I could improve the skill's VUI by asking a follow-up question at the end of listing the 5 jobs to know whether or not the user wanted more details about any or all of the jobs listed.
  • I'm using the AMAZON.ProfessionalType intent slot and this is currently only supported in the US. Therefore when the skill is certified by the Amazon team it will only be available on US based Alexa devices. I've asked about UK support for the AMAZON.ProfessionalType intent slots but have not yet received information about when UK support for it will be available. 
  • Analytics are very important for knowing what is and is not working from a user's perspective. I often use tools like Google Analytics for websites and Mixpanel/Amplitude for mobile apps which have been very insightful to better understand user behaviour. For my Alexa skill, I'd like to add in something like Dashbot analytics to get a window into what users are and are not doing with the skill.
  • Most importantly I would like to find ways to improve the VUI for subsequent invocations of the "job finder" skill. If a user has already performed a prior job search then I'd like to be able to have a search saved against their user profile and then have the skill look it up and ask the user whether or not it should be re-used. This will reduce the number of questions Alexa asks the user upon each new "job finder" skill invocation. Such functionality can be accomplished by identifying users by their UserId obtained from the session and then querying a database for prior jobs searches to re-use.



Update


  1. I mentioned implementing analytics within the skill previously. I've since integrated VoiceInsights analytics from VoiceLabs.co to begin receiving data for individual and aggregate user sessions across metrics such as usage, pathing, speech and retention. I'm finding it more valuable compared to Alexa's skill metrics data that are provided by default.
  2. My "Job Finder" skill is now live in both the US and UK markets:




Thursday, June 29, 2017

My podcast with AWS Startup Stories about hiring for startups

On June 23rd, 2017 AWS released the final podcast in a 6 episode series titled AWS Startup Stories. The series of podcasts cover a range of topics from startup fundraising (Crowdcube, LocalGlobe) to scaling a startup (Monzo, Starling Bank, Once dating app) and growing and retaining a team (Makers Academy, Yoyo Wallet).

In episode 6 Will Bentinck, Head of Careers at Makers Academy and myself, VP Engineering at Yoyo Wallet discuss the ins and outs of hiring for startups and how to retain talent long-term. The 30 minute podcast was hosted by Darren Mowry, Head of Startup Business Development and Venture Capital at AWS EMEA.

Monday, May 1, 2017

Yoyo Wallet Powers The New Caffè Nero App

Summary


  1. We built the Caffè Nero App on top of the existing Yoyo platform
  2. We achieved #1 in the Food & Beverage category on launch day ahead of Starbucks and UberEats
  3. We have the highest rated app in its category (mostly 5 star reviews)
  4. We modified our architecture and built a fully automated CI/CD system for rapid, regular releases

Launch


On April 10th, 2017 my team and I at Yoyo Wallet launched the Caffè Nero App across the UK and Ireland enabling a digital payment and loyalty experience for millions of Caffè Nero's customers.

The product engineering team at Yoyo Wallet (consisting of Android, iOS and backend engineers along with product mangers and designers) were responsible for designing, building and shipping the Caffè Nero App for the April deadline, whilst still continuing to improve and maintain the Yoyo Wallet App which is live across the UK and Europe at many Universities (e.g. Imperial, Oxford Brookes, York), Corporates (e.g. JP Morgan, Guardian, Accenture) and high street stores (e.g. Planet Organic, independent coffee shops).

Marketing



Nero kicked off a number of marketing campaigns after the release of the app to raise its profile, including posters and collateral in-store (see above) and email campaigns. 

Feedback


On launch day and the days since we've seen a surge in registrations for the app which resulted in the app obtaining the top ranked spot in the Apple App Store for the Free Food & Drink category (out ranking other apps such as Starbucks, Costa, UberEats, Nando's and Just Eat to name a few). Off the back of customers downloading, registering and using with the app to pay and collect stamps, we've received mostly 5 star reviews making us the top reviewed app in the category (As of April 27th we've received a total of 395 iOS and 416 Android reviews with ratings of 5.0 and 4.7 respectively). Along with positive user rankings, we've also maintained a very high quality threshold shown by our crash-free session rates which are at 99.97% and 99.94% for the latest versions of the iOS and Android apps respectively (measured via Fabric's SDK).


Features


We've abstracted away much of the complexity of the app behind a simple UI/UX that customers engage with and rate highly. Behind this simple UI/UX the app has the following features:

  • Multi-mode registration (Email, Facebook or Yoyo login)
  • Camera based credit card scanning to more easily gather card details (A/B testing proved the uplift in conversion)
  • Secure payment and loyalty in a single scan of the app OR loyalty-only so customers can also pay with their own cash/card
  • Rule-based loyalty that triggers in real-time based on each items in a customer's basket and notifies them via silent push notifications of their loyalty reward(s)
  • An activity feed that aggregates receipts and loyalty reward information 
  • Store locator for finding the closest store
  • Apple and Android Wallet integration for loyalty-only transactions
  • Settings: account management, support and promo codes

Architecture


Given that Yoyo is powering the Caffè Nero app, as an engineering team we were very conscious of leveraging both our existing mobile and backend platform architecture in the right way to minimise code/feature duplication across 2 (and eventually more) mobile frontends. Both sets of mobile apps call the same Yoyo API endpoints running on Amazon Web Services (AWS) and much of the mobile code base, for both iOS and Android, is re-used inside our own internal SDK as well as the core business logic of the app which we call "App Core". Only the UI of the mobile apps (which is visually different) and a small bit of custom business logic are unique to each app and therefore cannot be re-used:






















Shipping


At Yoyo we ship new versions of ALL our mobile apps to their respective app stores every week without fail on Tuesdays at 3pm for iOS and Tuesdays and Thursdays at 3pm for Android (we follow a release train process). This requires building, packaging and deploying a total of 6 .ipa and .apk binaries across our internal test, staging and production environments along with our 3 testing channels (alpha, beta and live). To manage this complexity effectively we've automated the entire build, package and deployment process for multiple apps by using our own custom mobile CI/CD pipeline built on Bitrise, FabricTestflight and our own scripts. 

Here is an example of what our mobile CI/CD pipeline does each day multiple times:
  1. An iOS engineer fixes a bug in the SDK and merges his/her change to master
  2. Bitrise immediately kicks off a new CI build cycle for both the Yoyo Wallet and Caffè Nero versions of the iOS app
  3. Once complete both iOS .ipa binaries are pushed out via Fabric to all alpha testers
  4. The next time all alpha testers open their Yoyo Wallet or Caffè Nero app on their iPhones (most internal Yoyo staff are alpha testers except the sales team) they will be asked to download the latest build
  5. Once downloaded, alpha testers will be using the latest and greatest version of the app and are more easily able to identify bugs or usability issues
This new, fully automated, CI/CD pipeline was implemented late in 2016 and has dramatically increased our mobile team's ability to rapidly release new versions of the mobile app to our users. The chart below shows this increase over time as we've improved our ability to ship to production using automation:




It's worth mentioning that we also use a company called Applause for crowdsourced continuous testing of our mobile apps. The service that Applause provides is incredibly helpful and cost-effective in terms of finding many usability issues and bugs across the plethora of Android devices and configurations that are available in the marketplace today. However we do not use their crowdsourced continuous testing service as a quality assurance gatekeeper for pending production app deployments (there is a very specific reason for why we do not use them in this way which I will explain further). 

Whenever there is a human gatekeeper of quality, I find that developers and their teams often rely on that person or set of people as the main quality assurance mechanism instead of themselves (they are less thorough with their code changes and code reviews and often write fewer automated tests). They effectively offload the burden of responsibility for quality assurance to someone else for the change(s) they are making. This is orthogonal to the idea that each developer is the best person to know what could be most affected by their changes and should therefore assure the quality themselves. At Yoyo we wanted to keep quality high by using Applause but did not want the service to become a crutch for developers to rely on. Therefore we only use Applause in parallel with existing deployments to production. Applause testing never stops any deployment, it merely identifies existing issues and bugs that need to be fixed post deployment and does a very good job at this task.

Learnings


  • Watch out for trademark checks: One unexpected thing to happen in the 11th hour was when we submitted the Android app to Google Play and it was rejected. We had completed and submitted the iOS app a week earlier because we knew the review process was more stringent and always took longer with Apple. We weren't worrying about our submission to Google Play as it was always seamless due to less checks and balances (or so we thought). This was not what happened in our case, Google flagged the fact that a 3rd party developer (i.e. us as in Yoyo Wallet) were submitting an app with someone else's trademarked branding and so the app was rejected. We had to scramble in the 11th hour over a weekend to get documents signed by Caffè Nero to assure Google that we, as a 3rd party developer, had the right to submit an app that contained Caffè Nero branding. This did delay our Android submission by a few days but luckily we still had enough head room for it to ship on-time.
  • Public wifi with logins can cause headaches: A phone's OS will automatically connect to a public wifi that it has previously connected to when it's within range but the user usually still has to accept the terms and/or re-login to gain full internet connectivity. If the latter does not occur then the user usually assumes they have internet connectivity when the app is open due to the wifi symbol displaying on their phone when in fact they do not have internet connectivity. There are a few ways to solve this by either (1) if the public wifi is within your control make sure certain outbound connections (via IP or domain) are given access even without the need to accept the terms or re-login to the public wifi OR (2) the app displays a "no internet connectivity" warning message so users expect limited functionality until they have fully connected to the internet. 
  • Improving mobile architecture and deployment infrastructure upfront was worth it: Given that we had the Yoyo Wallet app and were effectively white labelling it (modifying the UI for Caffè Nero as well as adding some new features), we wanted to approach this intelligently to minimise long-term technical debt. We knew that white labelling Yoyo Wallet could get rather messy quite quickly with multiple sets of apps being created and maintained long-term. Yoyo's mobile team spent time upfront to improve the mobile architecture on both iOS and Android to separate out the App SDK and App Core dependency layers which could be re-used across apps while the UI layer could be modified heavily. We also focused on improving the deployment infrastructure using Bitrise CI and compile time configuration flags within the app so that we could automatically build the different UI versions of the app with the reusable App SDK and App Core dependencies. Although this deployment infrastructure took time upfront to build and configure correctly, it reaped large dividends later in the project as the deadline loomed (it continues to reap ongoing benefits every day with each new commit). Mobile engineers now barely need to think about the Yoyo Wallet or Caffè Nero app deployments, we focus on improving the UI or business logic of the app and the deployments continue to happen automatically each day and in exactly the same way since the start of the Caffè Nero app project.

Conclusion


The entire team at Yoyo Wallet have done an incredible job building Caffè Nero's mobile app on time for the April 10th launch. Not only did they meet the deadline but have built an app that is available to millions of Caffè Nero customers. Many of these customers already love the product based on their ratings and are actively asking for more functionality to be added in near the future which is exactly the kind of response a product engineering team hopes for when releasing a major new product into the market.



Download the Caffè Nero App today:

Thursday, August 27, 2015

Links To Previous Posts

Technical

Product

Process

Growth

Sunday, July 12, 2015

10 Steps To Priming Kanban

Jesper Boeg has written one of the best and most succinct PDF docs on Kanban (43 pages in all). He breaks Kanban down into 10 steps that can be implemented piecemeal until your team has a comprehensive Kanban process governing all aspects of software development.

Thursday, May 28, 2015

Solarwinds Press Release for Pingdom, Librato and Papertrail

Solarwinds, the Texas based enterprise IT management company that purchased Pingdom for an undisclosed amount, Librato for $40M and Papertrail for $41M over the last couple years, recently got in touch with me because my company, Insignum, uses all three of their products to manage our machine learning infrastructure in the cloud.

Solarwinds were attending Velocity 2015 in Santa Clara and asked for my take on their products to include in a press release.  Here is the excerpt that includes my comments:
“All of these SolarWinds Cloud products work well together as they're solving slightly different problems at each level of our technology stack,” said Andrew Thompson, co-founder and CEO of Insignum, an automated analytics intelligence company. “Pingdom is our go-to solution for understanding the overall health and stability of the web pages our customers engage with day-to-day, while Librato gives us an understanding of how all our infrastructure is performing to deliver value to users, and then when something breaks, Papertrail becomes the service that we use to drill into the exact details of the problem so we can devise a solution – its true value is simplicity.”

Wednesday, September 17, 2014

Improving On-Site Search Functionality For GrowthHackers.com

Back in 2013 when GrowthHackers.com started gaining prominence and was regularly used by thousands of growth engineers, I started noticing that their on-site search functionality needed some improvement as relevant content was hard to find. I knew the small GrowthHackers.com team were working hard to design and deliver important features that were growing the community, so I wanted to help out by articulating the problem and devising a solution that would be easy for them to implement and could almost immediately be leveraged by the GrowthHackers.com community. 

Here is the short presentation I created and sent to Sean Ellis. It was subsequently used to improve their on-site search and is still used today by the GrowthHackers.com community (note that instead of using Swiftype they went with a similar product called Algolia):

Improving Search on GrowthHackers.com from Andrew Thompson

The Swiftype based search for GrowthHackers.com that I setup when I created the above presentation is still working today (give it a try):

Sunday, July 20, 2014

Just launched my new company called Insignum

Yesterday I launched my new company called Insignum. It delivers anomaly detection and notification for your analytics across Mixpanel, KISSmetrics and Google Analytics.

The product provides data-driven startups with automated analytics intelligence to find deviations in expected customer behaviour as they happen. Using machine learning algorithms, it continuously searches your data and when an anomaly is detected it will notify you so that you can act on the insight.


If you have any feedback about the product I'd be very interested in hearing about it. Insignum's Twitter account is @insignum_io.

Tuesday, June 3, 2014

How a one word change increased product demo conversions by 139%

The following is a repost from the GoCardless Blog where I explained one of our successful A/B tests. I also started a good discussion about the merits of the A/B test over at growthhackers.com.



This post looks at an A/B test with a simple copy change and how it improved conversion rates by 139%. The idea behind the A/B test was to give users immediate access to the product, via a recorded demo, instead of having them receive a personal phone call from our sales team later on.

Theory

Using a framework, when A/B testing potential improvements to landing pages, is helpful. Sean Ellis has a simple one for understanding the broad levers that can been affected: Conversion Rate = Desire - Friction. Desire and Friction can be further broken out and the LIFT Model by WiderFunnel describes this well:

LIFT Framework for CRO

Hypothesis

Our goal was to improve the conversion rate for demo requests so that customers could access the content they were interested in as soon as possible and with minimal friction. We wondered if the “Request a demo” button might be causing some users anxiety (as described by the LIFT model) and, as such, artificially lowering conversion rates. We then tested whether the wording “Watch a demo” would outperform the original “Request a demo” wording.
We had further reason to believe that immediate access to a recorded demo would be beneficial as only 1/5th of leads ended up watching the live demo which has been scheduled by our sales team.

Website Modifications

The original GoCardless user experience with the “Request a demo” was the following:
  • Call to action (CTA) on the homepage was “Request a demo”
  • Users were taken to a request a demo form to fill out and submit
  • Upon completing the form, users were given a date and time of an upcoming live demo that they could participate in.

GoCardless Homepage With Request Copy

GoCardless Demo Page With Request Copy

We altered the user experience by giving users immediate access to a recorded demo:
  • Call to action (CTA) on the homepage is “Watch a demo”
  • Users are taken to a “Watch a demo” form to fill out and submit
  • Upon completing the form, users are shown a 10 minute recorded demo in their browser
GoCardless Homepage With Watch Copy

GoCardless Requesting A Demo Page With Watch Copy

GoCardless Watch A Demo Page

Most of these changes were implemented within Optimizely’s Multi-page Experiments feature (aka Conversion Funnel Testing). However, we did build the recorded demo page ourselves as we didn’t need to A/B test this page directly.

Although some time and effort went into thinking about reducing friction, the work required to implement and instrument these changes was very easy because of Optimizely and Mixpanel. Optimizely has a very useful single toggle option for sending super properties to Mixpanel so that we can track what happens deep within our funnel.

Results

Given our acquisition channel characteristics, we ran the A/B test for a full 7 days. We then looked for a statistically significant winner with at least a 95% confidence level. Optimizely’s report panel below shows that the “Watch” version consistently outperformed the original version:

Results From Optimizely Report

We also ran the numbers through Mixpanel’s split test calculator based on event data we track in our conversion funnels:

Results From Mixpanel's Split Test Calculator

This shows that the “Watch a demo” version is more than twice as effective as the “Request a demo” version (139% increase in conversion). With this simple copy change, derived from the idea of reducing friction for new users, we’ve dramatically increased the number of users who watch a product demo and are therefore more likely to become customers.