Tuesday 11 December 2012

Prototyping with Google's Prediction API, Python, SQLite and some JSON APIs

:: Introduction

I've been playing around lately with the car2go API v2.1 as I was curious about how easy it would be to ingest it's vehicle location data over time and then try to accurately predict certain outcomes using machine learning algorithms.

I've had quite a bit of success in the past using Google's Prediction API for spam comment detection but this "spam" problem was using classification values and not regression values. Regression values  would have to be used in this type of geo-coordinates based problem.

The last time I used the Python programming language was when I worked at Toshiba in Edinburgh, UK a number of years ago. I was interested to get back to using the language as I remember it being a really elegant language and it also seemed like a great fit to prototype things out. I was using a Macbook Air, which had Python pre-installed, so this made it pretty easy to get up and running. I knew I'd need to query the API, parse out the necessary values and store these to a datastore of some kind. I opted to use Mac OS X's built in SQLite database at it integrated well with Python and it provided a great interface from the command line to test out various SQL queries.

:: Inserting Records into SQLite with Python

The following Python program illustrates how to connect to a SQLite database called car2go.db and insert a single data record into the Vehicles table (Note that the Vehicles table schema needs to be pre-defined and created before the code will execute successfully):

#!/usr/bin/python
# -*- coding: utf-8 -*-

import sqlite3 as sqlite
import sys

con = None

try:
    con = sqlite.connect('car2go.db')

    cur = con.cursor()
    
    cur.execute("INSERT INTO Vehicles VALUES(<INSERT_VALUES_BASED_ON_SCHEMA>)")
    
except sqlite.Error, e:
    
    if con:
        con.rollback()
        
    print "Error %s:" % e.args[0]
    sys.exit(1)
    
finally:
    
    if con:
        con.close()

As you can see this code is really lightweight and easy to get up and running when you have a machine with Python installed. 

:: Requesting API Data and Parsing Values with Python

The next step was to query the JSON based API and parse out any necessary vehicle values:

#!/usr/bin/python
# -*- coding: utf-8 -*- import json import urllib2 import sys # Note that you'll need to replace <SECRET_KEY> & <LOCATION> with the appropriate values car2goVehiclesApi = 'https://www.car2go.com/api/v2.1/vehicles&oauth_consumer_key=<SECRET_KEY>&loc=<LOCATION>&format=json' # request vehicle data in json format webReq = urllib2.urlopen(car2goVehiclesApi) vehiclesJson = json.load(webReq)

The vehiclesJson object now has the JSON data from the web request and can be queried directly given the appropriate keys/indexes. And example to output the address of the first vehicle is:

print vehiclesJson['placemarks'][0]['address']

:: Outputting SQLite Query Data to CSV

Once these basics where done I then had the foundation to expand upon it further. I eventually extracted all the necessary JSON vehicle data from the web request and inserted it into my Vehicles table in my car2go.db SQLite database. From there I crafted the specific SQL query I wanted and output that query data to a CSV file using the following sqlite3 command syntax (simply run the SQL query after setting these values):

.headers off
.output vehicleData.txt
.mode csv

:: Using the Google Prediction API

Setting up your Google Prediction API the first time can be slightly tricky so be careful to follow Google's directions carefully. I won't go into these prerequisite details here as Google has already done a great job explaining them here (and will most likely keep them up-to-date in the future). 

After setting everything up the vehicleData.txt file was then uploaded to one of my Google Cloud Storage buckets where it could be queried directly by Google's Prediction API. Note that depending on the data you upload you may need to wrap the strings in double quotes according to Google's specified training data format. After uploading, the first task was to train my model which can take anywhere from a few seconds to a few minutes. Once this is complete (querying for the status will inform you when its complete) you can begin to ask for regression predictions from the predict HTTP request. Note that depending on what you are predicting the outputValue or the outputMulti[].score values can be retrieved from the JSON response and used to interpret your intended outcome.

(Note that I was using Google's Prediction API v1.5 for this prototype).

:: Conclusion

I'd highly recommend giving the Python language a try (if you haven't already) for your next prototype project. Integrating it with a lightweight database, JSON web requests and Google's Prediction API was pretty easy and my overall impressions with the language and available libraries are that its still a pleasure to work with.

Tuesday 27 November 2012

Mobile Development Trends

:: Facebook on Android vs. iPhone Trend

A colleague of mine, Marwan Haddad, sent me an insightful article today about a poster that was recently seen at Facebook:


Many engineering teams are currently engaged in mobile development and because of this, mobile user trends are very important to keep track of so that each of us ends up building the right mobile applications today and hopefully as far into "tomorrow" as possible. Trends are shifting quickly as new users enter the smartphone market, decide to switch devices or simply change their overall mobile behaviour.

Facebook's poster (or really the IDC) is projecting quite far into the future as the date when this poster was mostly likely seen was around November 2012 (which puts us somewhere in the middle of the graph). The projection spans approximately 3 years from today till somewhere in 2016. In the technology business that's a very very long time to project anything as few people really know what will happen in 1 year from now let alone 3 (the iPhone 3GS was released just over 3 years ago and now look where we are with Android phones dominating). Maybe Windows Phone, the forthcoming Firefox OS or even a yet unnamed player will be lucky enough to steal some market from Google or Apple but really only time will tell. All I know right now is that the competition among these tech giants is fierce and as a result its producing an upward spiral of innovation within the smartphone market which all of us will benefit from for years to come.

The really interesting nugget of information within that Facebook poster projection is the dominating trend that Android currently has over iPhone and looks to have and increase in the months/years to come. If you or your company is still solely focused on iOS development at this point it may be worth reconsidering where the bulk of your customers are today let alone tomorrow. There was much talk way back when about Open vs. Closed Platforms and many predicted that even with Apple's initial dominance Android's "open system" would eventually prevail and it looks like that prediction is happening in full force (for the foreseeable future at least).

:: Kleiner Perkins' Mary Meeker on Mobile Trends

For an even deeper analysis of various Internet and mobile trends, Mary Meeker's year end trend report is an absolute gold mine of information. I've included a screenshot of slide #16 below where she notes that Android phone adoption was nearly 6x that of the iPhone... WOW.


:: TechCrunch's MG Siegler on Building for Mobile 

TechCrunch's MG Siegler wrote a post on Dec 11, 2012 titled Mobile Burst where he argues that "the mobile device is a different beast than a computer". He believes that companies/developers should focus a large amount of their resources on building the right type of user experience that is not only optimize for, but created exclusively for mobile use. In that post he links to his own personal blog post where he makes some very pointent statements:
  • "Don’t build an app based on your website. Build the app that acts as if websites never existed in the first place. Build the app for the person who has never used a desktop computer. Because they’re coming. Soon."
  • "What matters is that in the next five years every person on this planet is going to be using a mobile device. And these devices are going to be used far more than any traditional computer ever has been and ever will be."
So whether you're building mobile applications with HTML5 or going native, make sure to build the right experience for that specific mobile form factor instead of replicating or mimicking what you've previously built for the desktop.

:: Updates

Following on from MG Siegler's article above, here is an interview with Vibhu Norby on TechCrunch TV who disagrees that Mobile is unanimously the future of all development.

Megan Quinn of Kleiner Perkins Caufield & Byers (who formally worked at Google and Square) was interviewed on TechCrunch TV and had some things to say about the emerging mobile landscape:
  • "mobile is not an evolution this is a revolution."
  • "every human activity is being re-imagined through the prism of a mobile device"
She went on to say that the 3 areas of mobile development that are seeing huge potential are: Education, Consumer Health and the Internet of Things.

Another Mobile Internet assessment from Business Insider called "The Future of Mobile".


Saturday 17 November 2012

RESAAS Wins Prestigious Microsoft Award

Yesterday my company, RESAAS, was announced as the winner of the 2012 Microsoft Impact Award in the category of Windows Azure Platform ISV Partner of the Year.

Our press release of can be found on Bloomberg and Microsoft has a few more details about our category and who we were up against on the Microsoft Partner Network.

Tuesday 14 August 2012

RESAAS Windows 8 Application Approved by Microsoft

The engineering team at RESAAS recently started working on a Windows 8 styled application (using the Metro design language) for our real-time Question and Answer feed. I'm happy to say that Microsoft has approved the application and it is now available in the Microsoft Windows 8 Store.

RESAAS also pushed out a press release which found its way onto Yahoo Finance: http://finance.yahoo.com/news/resaas-enterprise-platform-real-estate-160000716.html

A PDF version of the press release can be found on the CSNX where our company's stock is listed: http://www.cnsx.ca/Storage/1472/134962_NRAug92012.pdf

Tuesday 17 July 2012

Interviewed by Jonathan Rozenblit for "Canada Does Windows Azure"

Jonathan Rozenblit who is a Developer Advisor at Microsoft interviewed me for a series he does called "Canada does Windows Azure". We spent 17 min talking about how RESAAS uses Azure, why it decided to use Azure and how our developers ramped up on the platform when we first started.


Here are links to the blog posts Jonathan put on his blogs:

Friday 20 April 2012

Towards Instant Programming Feedback

This hour long conference lecture by Bret Victor is a really fascinating look at how to change the way engineering professionals create what they create. He experiments with some simple yet powerful tools to show what creative tasks could look like if they provided almost instant feedback to their creator. Most of the examples relate to software engineering and the languages used are either in JavaScript or Java.

Update (6/7/2014):

I recently came across Light Table which is a next generation IDE which displays code right next to a browser window. You can then modify the code and it is updated within the browser in real-time.

Microsoft's Case Study on how RESAAS leverages Windows Azure

Microsoft recently put together a case study on how RESAAS has used Windows Azure to scale its platform to meet future demand. As mentioned previously in the slides on this post, it took us approximately 3 months to port over our entire application from a typical IIS web server and SQL Server database based application to a cloud based one.

Thursday 19 April 2012

How did Instagram Scale their Infrastructure?

TechCrunch has a great post written up just after Instagram sold to Facebook for $1 Billion. The post contains a 185 slide deck filled with the trials and tribulations Instagram went through on their way to 30+ million users in less than 2 years.

Tuesday 24 January 2012

Appointed as VP of Engineering at RESAAS

I am thrilled to accept my new positon as the VP of Engineering at RESAAS. I joined the company 8 months ago as the first full-time software engineer and with the help of Tom Rossiter, CTO, we have grown the development team to 8 full-time backend, frontend and quality assurance engineers all in 6 short months. Our team has been able to successfully expand in such a short time-frame by utilizing Agile methodologies, test automation, continuous integration and database change management tools along with the dedication of some amazingly talented engineers who work very hard to continuously improve the processes, the tooling and most importantly the product RESAAS is offering.

The enterprise social network that our team is building has evolved considerably over that time period from a single .NET and SQL Server based system hosted in a data center to a scalable platform hosted in the cloud (Windows Azure). Our core platform has been able to leverage Worker roles, Web roles, SQL Azure and Azure Table and Blob Storage along with various other 3rd party services to connect real estate professionals and their clients together in real-time via their laptop or chosen mobile device.

I am looking forward to working hard and developing the full potential of both our engineering team and the RESAAS core platform in the coming months.

Note that RESAAS is a public company currently listed on the CNSX under the symbol: RSS. Here is the press release on the CNSX of my appointment as Vice President of Engineering at RESAAS. 

Sunday 22 January 2012

My Social Graph: Linkedin Professional Network

I recently tried out Linkedin Lab's InMaps product which visualizes your professional social graph (see image below). Apart from it creating a great visualization of my own professional data, I found it quite interesting how it seems to capture pivotal sets of relationships that have shaped my professional life and essentially weights them according to their inter-connectedness.

The largest mass of blue connections on the right are a result of a former company I worked for called Blast Radius. They are a digital agency so I was exposed to many short-term or contract employees throughout all the projects I worked on. Blast Radius is a global company so the clump of green connections in the top right are mostly the European based employees and contractors that I worked with.

Smaller clumps of connections on the bottom and bottom-left are due to the startup company I currently work for, RESAAS, classmates during my university days and various people I met while working for Toshiba Medical in the United Kingdom. Finally the smallest clumps scattered around are due to friends, family and football team members whom I have connected with over the years.

So what value can be extracted from this InMaps visualization?
  1. Information that I create or update on Linkedin clearly has a bias towards being seen and/or acted upon by current and former Blast Radius employees in Canada, the United States and across Europe. Therefore when I post something or update my profile I can take this into account as I now have a better understanding of who my audience actually is.
  2. I have heard it said before that you never have more access to connect with people than when you are in university. That may still be true but given that Linkedin was in its infancy around the time I was in university and the fact that I have taken the opportunity to connect with many people in each professional setting I have worked in, my current professional social graph indicates that my current university connections take on an increasingly smaller part of my overall ability to connect with people worldwide.
  3. The visualization says nothing about the individual importance of each connection or group of connections that make up my professional social graph. However, by using my own judgement about the people I am closest too and/or have the ability to influence the most, I can roughly identify how to spend my time wisely to either obtain the information I'm looking for or to maximise the reach of information that I would like people to know or act upon.


Thursday 19 January 2012

Launch of Career in Canada Event @ UBC

I will be speaking again at the Univeristy of British Columbia's Launch a Career in Canada event on February 7, 2012 at 6:00pm.

I spoke at it last year and it was a great opportunity to meet many motivated university students looking to get a foothold in a career after they graduate.

Thursday 12 January 2012

Presenting at the Vancouver Windows Azure Meetup Group hosted by Microsoft

I will be giving a talk about SQL Azure and Data-tier Applications (DACPAC) at the Vancouver Windows Azure Meetup Group this coming Wednesday, January 18th, 2012. There will also be a guest speaker who will be a Sr. Azure Architect from Redmond, Washington.

The meetup will be hosted by Microsoft in their Downtown Vancouver offices and if you would like to attend please RSVP via this meetup group.

I will also be attaching the presentation slides to this post after the talk.

-----------------

The talk went really well and there were lots of great questions from the audience. Below are the presentation slides I used during the Windows Azure Meetup talk (hosted on SlideShare):


Wednesday 11 January 2012

Insights from LeWeb'11


I attended LeWeb in Paris, France on Dec 7-9th 2011. LeWeb is the premier technology conference in Europe which attracts influential businesses, up and coming entrepreneurs, passionate speakers and bold ideas while providing an opportunity to network with everyone involved. LeWeb'11 was themed around what they called SOLOMO (Social-Local-Mobile) and they had a wealth of influential speakers on the subject.

Below I have captured some of the insights and notable content from LeWeb'11:

:: Berlin


Bright and young entrepreneurs are flocking to Berlin. London has always been seen as a cultivator of entrepreneurs and their startups within Europe but Berlin is taking off. Its tech scene is trying to emerge as a thriving economy: entrepreneurs are setting up shop along with talented young professionals seeking employment. From SoundCloud to Gibsy to Wooga.

:: Tackling Heavily Regulated and Entrenched Markets


:: Mobile


In the history of computing, mobile has had the fastest adoption rate. There are currently 700 million smart phones being used and the projections are that this will treble in 3 years. Product companies building web-based applications should therefore think seriously about where their user base currently is and where it will be in 3 years. Given that the general trend is heading towards mobile, if companies are not heavily focused on it then they should at least know why not and have a good reason for it.

From a technology standpoint mobile requires different skill sets within an engineering team. Depending on whether the chosen development architecture is cross-platform (PhoneGap, Appcelerator, or simply HTML5) or native (Objective-C for iOS, Java for Android, C# .NET for Windows Phone 7), a clear understanding of how these mobile devices, with their many versions and layouts, will interact with your web-based application is critical. If you are fortunate enough to have already built or inherited a service-oriented architecture (SOA) based application, you are one step in the right direction to building out your mobile product offerings. Scaling with hundreds, thousands or even millions of mobile applications being used in the wild (each set with their own application versions) presents some very unique challenges that will need to be overcome if the mobile growth projections become reality.

:: Prioritize Design


Many companies still neglect design to their detriment. Engineering teams usually hire in a contract graphic designer or user-interface designer for usability concerns instead of focusing on it in-house. Flipboard CEO, Mike McCue, stressed that the trade-off in software engineering is between a few very well designed features or many mediocre ones.

:: Focus


Continually question what your company's focus is and make sure you are always correcting your course back to that initial purpose. Many companies start out with a clear focus but over time there is a strong tendency for it to either become diluted or for the company to forget/ignore what the original focus was. The key take away was that a company's focus is key to their growth and holding onto that singular focus is very important in a sea of constant noise.

:: Put People at the Center


Many people attribute Facebook's huge success to a pivotal turning point when the tagging of photos became mainstream on their social platform. There were many companies building web-based photo sharing applications (such as Flickr) but Facebook quickly became the largest by far by simply putting people at the center of those photos. Facebook added a key feature which allowed people to tag their friends, family and just about anyone they knew which caused a massive change in people behaviour to begin uploading and tagging pictures on Facebook in record numbers never before seen in an online application. The power of social is the people are at the centre.

:: Dream Bigger


The now well know advice that Steve Jobs gave to an executive trying to reinvent the Disney Store was to "Dream Bigger". Time and time again VCs on stage would urge the crowd to ignore the easy road of building yet another photo sharing app and find a problem or need in the world that is so huge that it scares you to even think about it. Those are the kinds of ideas entrepreneurs should be thinking about and trying to solve. Whatever you are thinking you can allows dream one step bigger.

:: Best Interviews (Videos)