Friday 30 August 2013

The Startup Curve and How Not to Die

Paul Graham's famous startup curve maps the trials and tribulations that a startup can go through on the road to success.

Of course, not every startup ends up hockey sticking after the "Wiggles of False Hope". Most startups end up failing (90% of tech startups according to Allmand Law), but Paul also ended his "How Not to Die" essay with these words:
"So I'll tell you now: bad shit is coming. It always is in a startup. The odds of getting from launch to liquidity without some kind of disaster happening are one in a thousand. So don't get demoralized. When the disaster strikes, just say to yourself, ok, this was what Paul was talking about. What did he say to do? Oh, yeah. Don't give up."

Thursday 29 August 2013

3 Responsibilities of Product Management

Adam Nash (prior VP of Product Management at LinkedIn) describes what it takes to be a great product leader. He boils down a product manager's job description to three key responsibilities.

Responsibility #1: Product Strategy

Adam describes product strategy this way: "it’s the product manager’s job to articulate two simple things":

1. What game are we playing?
    • What is the vision of the product?
    • What value does it provide customers?
    • What is the differentiated advantage over competitors?
2. How do we keep score?

Clearly answering these questions synchronizes teams across the organization and helps them understand how to win effectively in the market. 

Responsibility #2: Prioritization

Different processes exist to handle the prioritization of features and tasks: Waterfall, RUP, Agile, Kanban.. etc. But without a solid product strategy, prioritization becomes very difficult to do effectively. Adam describes a framework for product planning which he calls the Three Feature Buckets:
  1. Customer RequestsThese are features that your customers are actively requesting. 
  2. Metrics MoversThese are features that will move your target business & product metrics significantly.
  3. Customer DelightThese are features that customers haven’t necessarily asked for, but literally delight them when they see them.
Features may fit into more than one bucket but rarely fit in all three. The benefit of classifying each feature this way is so that a team can be intellectually honest with themselves about why they should implement a particular feature.

Responsibility #3: Execution

Execution is all about shipping the product and getting it in front of users so that they can derive value from it. Sometimes this means ASAP and sometimes it requires timing the market effectively. Teams have many different approaches on how they execute ranging from light weight idea, test and deploy methodologies to full-blown specifications, sign-off, development, QA and release cycles. Adam describes the 4 parts of execution that are critical to its success:
  1. Product specification: The necessary level of detail to ensure clarity about what the team is building.
  2. Edge case decisions: Very often, unexpected and complicated edge cases come up. Typically, the product manager is on the line to quickly triage those decisions for potentially ramifications to other parts of the product.
  3. Project management: There are always expectations for time/benefit trade-offs with any feature. A lot of these calls end up being forced during a production cycle, and the product manager has to be a couple steps ahead of potential issues to ensure that the final product strikes the right balance of time to market and success in the market.
  4. Analytics: In the end, the team largely depends on the product manager to have run the numbers, and have the detail on what pieces of the feature are critical to hitting the goals for the feature. They also expect the product manager to have a deep understanding of the performance of existing features (and competitor features), if any.

Top 5 Growth Hacks to Consider

1. The Minimal Homepage

Dropbox, Pinterest, Quora are famous for their minimal homepage. When you start out and envision designing a new homepage for your product with all of its features, it's so counter intuitive and hard to decide to have a homepage with only one sentence, one photo and one call to action above the fold.

Anyone who hasn't measured this effect before will argue and argue that a minimal homepage like the one above will not convert as well because people just won't understand much about the product yet and will therefore not sign-up. It's always a good idea to A/B test your own minimal homepage against other types like short or long firm ones but time and time again minimal wins out (and sometimes significantly). The reason for this is that the page is simple for anyone to understand and the call-to-action is really clear because its the only one. We've all been to a homepage with 10 different products or large amounts of verbiage and its incredibly hard to know where to start or what to click on.

2. Send Push Notifications to Increase Retention

Using email to remind users about your product and re-engage them is a commonly known growth hack. Email is definitely an old school channel (not as sexy as social) but it can be very effective once you've taken the time to master it. Adam Nash (former VP of Product Management at Linkedin) had this to say about emails as a traffic source:
Email scales, and it’s inherently personal in its best form.  It’s asynchronous, it can support rich content, and it can be rapidly A/B tested and optimized across an amazing number of dimensions.  The best product emails get excellent conversion rates, in fact, the social web has led to the discovery that person to person communication gets conversion person over 10x higher than traditional product emails.
Having said this email is a saturated channel. Most companies are not doing email well but they're doing it none-the-less which adds to the sheer volume we all receive in our inbox on a daily basis. The Law of Shitty Clichthroughs is a common problem we face when we all rush into a channel and saturate it but recently it's been made slightly worse with Gmail's new tabbed inbox changes. MailChimp has some great data that shows a 0.5% to 1.0% drop in open-rates because of this change.

Don't stop sending email, as its still a great channel, but try sending push notifications if you have a mobile App. Relevant push notifications have a significant affect on retention rates. To get some ideas of how push notifications compare to email open and click-through rates, this post gives us some insight about their effectiveness:

  • 30%-60% open rates
  • 4%-10% interaction rates (with spikes as high as 40%)

Pro Tip: Use Mixpanel (event based analytics) to register users for your website and/or mobile app. Once you've done that you can send them emails, push notifications or text messages manually or automatically based on specific events a user performs within your application. Not only that but you can build intuitive funnels to track the exact effective of your email, push or text message campaigns.

3. Kill a Feature

This growth hack is less about a quick win and more about getting to the core of what might actually help your product grow long term. It can also be used to make sure that your product is continually refined and made simpler to use instead of leaving all those unused features hanging around which cause product bloat and therefore user confusion/frustration.

The above image is from a presentation by Dave MccLure. It's pretty self explanatory. Andrew Chen talks about this in a similar way when he says that "Does your product suck? If so stop adding features and 'zoom in' instead".

4. Embeddable Widgets

<iframe width="560" height="315" src="//" 
frameborder="0" allowfullscreen></iframe>

Embedded video, like the one above (along with its <iframe> HTML code) was YouTube’s famous growth hack that helped them scale to hundreds of millions of users. In return for free video hosting, blogs and websites promoted the YouTube brand by embedding YouTube-hosted videos on their sites. It was a win-win for everyone involved and YouTube capitalized by acquiring users organically.

Many companies know about this growth hack and have subsequently created their own widgets (Vimeo, SlideShare, Healthtap... etc) that can be used in blogs or websites. Sometimes this works well (SlideShare for instance) but it can also fall flat with your user base. Having an embeddable widget is necessary but not sufficient to harness this growth hack's potential. There has to be a strong incentive for users to spend the time and energy to embed your widget in their site. There should be a strong need to share the content delivered by the widget in order for this growth hack to bear significant fruit.

5. Two-sided Referral Incentives

During Dropbox's growth, the company tried a number of marketing channels like long-tail search and paid advertising but they didn't work out that well. Dropbox then began experimenting with a referral program and it worked really well. The structure of that referral program is the most interesting part. We've all registered for a service before and been asked to invite friends and told that if they join we'll get some special offer. It feels pretty spammy to give a company your friend's email address and you get all the benefit. Dropbox knew this so they devised their referral program to be two-sided. If you invite your friend and they join BOTH of you will get the benefit. Each person receives 250 MB of additional storage. The psychology of this is great, it no longer feels spammy to invite your friends, you actually feel like you're helping them out by giving them more than they otherwise would have had without your invite.

Tuesday 27 August 2013

Classifying Knowledge for Engineering and Product Development

When designing and building a software product there are many things that individuals or teams know and there are also things that they don't know... this is pretty obvious. However this simplified distinction is missing some of the inherent complexity about knowledge that we can or cannot know. Donald Rumsfeld describes this larger epistemological concept, which can be quite complex, in a pretty straight forward way:
There are known knowns; there are things we know that we know. There are known unknowns; that is to say, there are things that we now know we don't know. But there are also unknown unknowns – there are things we do not know we don't know.
These three ways of classifying knowledge can be very helpful during product development for both engineering and product teams. Not only does it help to communicate the various types of things that we may or may not know but also helps to showcase that there are things that we don't even know that we don't know - i.e. concepts, tools, processes, best practises that might exist already but we don't even know about them and therefore don't know to look for them (which I will explain in more detail below).

1. Known Knowns

"Known knowns" are those things that each of us has previous experience with or those things that are held as best practises within the industry that we know about already. Here are some for engineering and product teams:
  • Engineering: Computer languages that are closer to the bare metal of a machine (e.g. C/C++) are generally faster then those languages that run on a virtual machine (e.g. Java, .NET, Python... etc) given the same algorithm and data.
  • Product: When designing conversion pages/screens, each additional step (button click, user input, screen swipe.. etc) in-between the user's entry point and the objective for them to accomplish, results in some trivial or significant amount of drop-off. This can be easily seen in a conversion funnel by measuring and then observing the individual percentage drop-offs between each successive step.
In the engineering case you usually don't need to write the exact same algorithm in multiple languages and measure the execution time but you can be confident that as a general rule of thumb a C/C++ algorithm will be slightly (or maybe even significantly) faster than a virtual machine based language running the same algorithm. Testing always makes sure this assumption is valid and the results can sometimes conclude the opposite but engineers perform so many performance or functional assumptions when writing code that validation for each and every one is not realistic.

In the product case you can be almost certain that the removal of a single non-essential step will increase the conversion rate by some amount (what exactly that amount would be is something that would have to be measured). For example, many conversion flows for social networks include the option to import contacts in order to find your friends/colleagues/acquaintances... etc so that you can connect or follow them (Facebook, Linkedin and Twitter do this). But in other products its less about satisfying the core functionality of the product in terms of connections and more about it being a virality growth hack to acquire more users. Importing contacts in the first case is obviously vital to engage and retain users in a social network but in the second case the step could be removed at the cost of acquiring less users (i.e. no virality). Removal of this step would almost certainly increase the overall conversion rate (due to the step no longer causing drop-off) but usually the increased amount of users, due to the contact import, out weighs the amount of users who drop-off at the contact import step. Therefore it's usually desirable to leave the contact import step in the conversion flow even though is decreases overall conversion rates by a slight amount.

As a final note, the better you are and the more experience you have, the more "known knowns" you accumulate. Because of this you work more efficiently since you have access to a wider array of expertise that you otherwise wouldn't have had. According to Malcolm Gladwell's "10,000-Hour Rule" written about in his book, Outliers, it may take 5 years or more (40 hours/week over 5 years is 10,000 hours) to accumulate enough knowledge to become an expert in any given field.

2. Known Unknowns

"Known unknowns" are those things that we may have some theory for or gut feeling about but don't actually know what the correct approach or answer might be. It might even be that you know that something exists but don't know much else about it. Essentially its being aware of your own ignorance about something. Many good senior engineers or business analysts face these types of issues on a day to day basis and are pretty good at either asking questions and listening to others, performing research to uncover the right approach/answer or, if not available, they experiment or prototype until it becomes more evident what the right approach/answer might be. Once this is done "known unknowns" turn into "known knowns" which is obviously a good thing.

For many things "known unknowns" are similar across engineering and product teams. A software engineer or business analyst might know that something exists which could help improve the product (say a new Python library or integration with another product) but not really know much about it. All they would have to do is begin researching it and given sufficient time they will figure out what they need to.

There is of course a specific area where these teams differ. The worlds of engineering and product start to diverge for "known unknowns" when predictability is accounted for. Software systems are inherently predictable, they generate the same output given the same input - they're built this way to manage complexity. There are things that seem random but they are usually the result of unexpected user input, actual random number/string generators and/or partial system failures that cause intermittent behaviour. The advantage of predictable systems is that various solutions can be re-used across domains and if something has been used once before chances are it can be used again in a different context and the benefit of a lower learning curve can be leveraged. There is also the benefit of a tangible "done criteria" for software systems. If a module within an application was supposed to parse a CSV file and insert each row into a database, then it's pretty easy to determine when all the work is done either by observing the results or writing some unit/integration tests to verify things. The consequence of this can be seen with all the open-source libraries and tools available that are built by one set of engineers and then used by thousands of others engineers for their own applications in a different context.

For product teams the inherent challenge is that users behaviour differently and are very hard to predict. Users behave differently from across different products but even more unsettling is that their behaviour changes over time on the same product. Essentially users are unpredictable and things that have worked in the past may not work in the future. It's analogous to the Law of Shitty Clickthroughs: what works today may not always work tomorrow as users learn and respond to your own product and all the other products that they use. On top of this its never as easy to simply mimic/copy an existing product's features and expect the same results. Engineering teams can use the same open-source library and be pretty confident that the results will be the same. However product teams who decide that a given feature might be worth mimicking/copying for their own product are really gambling on the idea that their users are the same type of users as the other product - and this is never usually the case. For example: If you're building some type of social network you're most likely observing what Twitter, Facebook, Linkedin, Pinterest... etc are doing but simply turning your UI into a Pinterest type feel or adding hashtags may not have the same results as those products had due to the fundamental difference that their users are not the same as yours.

Having said of this, engineering teams are able to turn "known unknowns" into "known knowns" often times more easily than product teams by researching and spec'ing out the design. Even before anything is built it's usually apparent that the given solution will work with enough engineering time (scalability aside). For product teams its a constant challenge to start with "known unknowns" and turn them into "known knowns" after thoroughly researching possible options or coming up with theories of how users might behave. Until the intended product feature addition or change is fully implemented (or, if you're lucky, maybe just some small component of it that users interact with), its almost impossible to know how users will behave.

Conclusion: For product teams "known unknowns" usually stay that way until you've actually shipped your product and see how users are using it. It's only at this stage that "known unknowns" become "known knowns".

3. Unknown Unknowns

This is the scary stuff. "Unknown unknowns" are the things you don't even know that you don't know about (actually they're not that scary because you're actually completely ignorant about them). They're not even on your radar and you don't even know they exist. The dangerous part with "unknown unknowns" is that even with enough time or energy they won't magically turn themselves into "known unknowns". In the engineering case this is like not even knowing that an entire field of academic study has worked on and solved a particular problem with an elegant algorithm that if you just knew about you could use in your application and solve you and your team hundreds of hours working on your own solution. In the product case its like being completely ignorant that another business somewhere in the world has a competing product that is vastly better than yours even though you and your team might have done some extensive market research.

So what do you do with "unknown unknowns"? Well here is the only thing Mike Gagnon thinks you can do: 
The best you can do with “unknown unknowns” is be aware that the category exists and maintain an open mind. This way when information presents itself to you, you can cognizantly realize that it was an “unknown unknown” and then you’ll either be in the “known knowns” or “known unknowns” category.
This is why being a prolific reader is so important. It feeds your mind with a ton of new information, ideas, concepts and solutions that you had no idea existed previously. Everything you read, are told about or experience first hand has the opportunity to identify "unknown unknowns" which at the very moment no longer remain as "unknown unknowns".

My grade twelve english teacher once told our class that we should not spend the rest of our lives reading books that agree with our current belief system but rather we should deliberately read ones that disagree with it so that we grow and change for the rest of our lives. I don't think I really had any idea what he was talking about back then but some how more than decade later I still remember those words and I now understand what he means.

4. Unknown Knowns

This one wasn't mentioned by Donald Rumsfeld but its worth mentioning briefly. Scholars of logic will have noticed that there are two words with two possibilities each which translates into 4 possible outcomes. Mr. Rumsfeld mentioned three of them and obviously didn't talk about the forth but its worth asking whether or not "unknown knowns" are even possible.

It turns out that the well known psychoanalytic philosopher Slavoj Žižek extrapolated a forth category which he obviously called "unknown knowns". He said that "unknown knowns" are those things that we intentionally refuse to acknowledge that we know. Slavoj even wrote an essay on the matter but it speaks more about politics than the underlying concept of unknowingly knowing something. Essentially its those things that we actually do know but we either vehemently deny them or even go so far as to suppress the knowledge we have about them.

The engineering and product teams that I've managed have been highly collaborative and open to exploring new ideas even if that meant redoing and/or throwing out inferior work. So when one person found a solution that could significantly change or alter currently held assumptions, it was always encouraged to bring that to the team. In light of this there was never much incentive to suppress knowledge about something across the team in order to avoid telling the truth even if it hurt to tell it.

There could of course be a case where any one individual would have some incentive to suppress the knowledge they had about something and therefore harbour an unknown known. This is obviously a reality but building the right culture with the right people helps immensely in avoiding this behaviour outright or at the very least discourages it.