Thursday, 12 January 2012

Presenting at the Vancouver Windows Azure Meetup Group hosted by Microsoft

I will be giving a talk about SQL Azure and Data-tier Applications (DACPAC) at the Vancouver Windows Azure Meetup Group this coming Wednesday, January 18th, 2012. There will also be a guest speaker who will be a Sr. Azure Architect from Redmond, Washington.

The meetup will be hosted by Microsoft in their Downtown Vancouver offices and if you would like to attend please RSVP via this meetup group.

I will also be attaching the presentation slides to this post after the talk.

-----------------

The talk went really well and there were lots of great questions from the audience. Below are the presentation slides I used during the Windows Azure Meetup talk (hosted on SlideShare):


Wednesday, 11 January 2012

Insights from LeWeb'11


I attended LeWeb in Paris, France on Dec 7-9th 2011. LeWeb is the premier technology conference in Europe which attracts influential businesses, up and coming entrepreneurs, passionate speakers and bold ideas while providing an opportunity to network with everyone involved. LeWeb'11 was themed around what they called SOLOMO (Social-Local-Mobile) and they had a wealth of influential speakers on the subject.

Below I have captured some of the insights and notable content from LeWeb'11:

:: Berlin


Bright and young entrepreneurs are flocking to Berlin. London has always been seen as a cultivator of entrepreneurs and their startups within Europe but Berlin is taking off. Its tech scene is trying to emerge as a thriving economy: entrepreneurs are setting up shop along with talented young professionals seeking employment. From SoundCloud to Gibsy to Wooga.

:: Tackling Heavily Regulated and Entrenched Markets


:: Mobile


In the history of computing, mobile has had the fastest adoption rate. There are currently 700 million smart phones being used and the projections are that this will treble in 3 years. Product companies building web-based applications should therefore think seriously about where their user base currently is and where it will be in 3 years. Given that the general trend is heading towards mobile, if companies are not heavily focused on it then they should at least know why not and have a good reason for it.

From a technology standpoint mobile requires different skill sets within an engineering team. Depending on whether the chosen development architecture is cross-platform (PhoneGap, Appcelerator, or simply HTML5) or native (Objective-C for iOS, Java for Android, C# .NET for Windows Phone 7), a clear understanding of how these mobile devices, with their many versions and layouts, will interact with your web-based application is critical. If you are fortunate enough to have already built or inherited a service-oriented architecture (SOA) based application, you are one step in the right direction to building out your mobile product offerings. Scaling with hundreds, thousands or even millions of mobile applications being used in the wild (each set with their own application versions) presents some very unique challenges that will need to be overcome if the mobile growth projections become reality.

:: Prioritize Design


Many companies still neglect design to their detriment. Engineering teams usually hire in a contract graphic designer or user-interface designer for usability concerns instead of focusing on it in-house. Flipboard CEO, Mike McCue, stressed that the trade-off in software engineering is between a few very well designed features or many mediocre ones.

:: Focus


Continually question what your company's focus is and make sure you are always correcting your course back to that initial purpose. Many companies start out with a clear focus but over time there is a strong tendency for it to either become diluted or for the company to forget/ignore what the original focus was. The key take away was that a company's focus is key to their growth and holding onto that singular focus is very important in a sea of constant noise.

:: Put People at the Center


Many people attribute Facebook's huge success to a pivotal turning point when the tagging of photos became mainstream on their social platform. There were many companies building web-based photo sharing applications (such as Flickr) but Facebook quickly became the largest by far by simply putting people at the center of those photos. Facebook added a key feature which allowed people to tag their friends, family and just about anyone they knew which caused a massive change in people behaviour to begin uploading and tagging pictures on Facebook in record numbers never before seen in an online application. The power of social is the people are at the centre.

:: Dream Bigger


The now well know advice that Steve Jobs gave to an executive trying to reinvent the Disney Store was to "Dream Bigger". Time and time again VCs on stage would urge the crowd to ignore the easy road of building yet another photo sharing app and find a problem or need in the world that is so huge that it scares you to even think about it. Those are the kinds of ideas entrepreneurs should be thinking about and trying to solve. Whatever you are thinking you can allows dream one step bigger.

:: Best Interviews (Videos)

Tuesday, 27 December 2011

Deploying Packages to the Azure Compute Emulator for Automated Testing across Local Environments and a Continuous Integration (CI) Server

:: Introduction

When developing on the Microsoft Windows Azure Platform, a storage and compute emulator is available for running applications via the Windows Azure Tools (as of writing this post I am using v1.6). An Azure application package can be deployed to the emulator (running locally) instead of to an actual Azure compute node in the Azure cloud. The benefits of this are two-fold:

  • Deployments to the emulator are dramatically faster (it takes seconds to deploy to the emulator as opposed to minutes to deploy to the Azure cloud)
  • There are no additional costs associated with deploying to and/or using the emulator to run an application package since the emulator is a locally installed application (deploying to the Azure cloud would incur additional compute and bandwidth charges).

Therefore when I needed to create some automated tests to run against various WCF web methods that were apart of my Azure web role, I first needed to deploy my web role (within the Azure package) to the Azure emulator. Note that I wanted to be able to run the automated tests both locally and on our development team's continuous (CI) server, therefore the solution I chose needed to accomodate both environments.

:: Force the Azure Project to Package during Compilation

Before an Azure package can be deployed to the emulator it must first be packaged. Since v1.4 of the Windows Azure Tools, the csx folder and its contents are no longer generated during compilation (See these release notes for v1.4 and search for the breaking change: "PackageForComputeEmulator"). So in order to force the generation of the csx folder we need to specify that the package should be generated for the compute emulator within the Azure project file (*.ccproj). I embedded the this within an existing property group containing some other items:

 <PropertyGroup>  
  <VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">10.0</VisualStudioVersion>  
  <CloudExtensionsDir Condition=" '$(CloudExtensionsDir)' == '' ">$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v$(VisualStudioVersion)\Windows Azure Tools\1.6\</CloudExtensionsDir>  
  <!-- This flag generates the 'csx' package folder when this project is compiled (it does not require a separate package command) -->  
  <PackageForComputeEmulator>true</PackageForComputeEmulator>  
 </PropertyGroup>  

:: Deploying to the Azure Compute Emulator

Now that we have the csx package folder generated after each compilation run we can use the CSRun command-line tool to deploy the folder to the emulator at the start of each test suite run. I was able to accomplish this as cleanly as possible by embedding the package deployment within our testing framework's assembly initialize method which looks like the following:

 public static void Initialize(TestContext context)  
 {  
   var emulator = ConfigurationSettings.Settings["AzureEmulator"];  
   var computeArgs = ConfigurationSettings.Settings["AzureComputeArguments"];
 
   using (var process = new Process())  
   {  
     process.StartInfo = new ProcessStartInfo(emulator, computeArgs);  
     process.Start();  
     process.WaitForExit();
 
     Assert.AreEqual<int>(0, process.ExitCode, "Starting the Azure compute emulator and loading it with a package failed");             
   }  
 }  

And the 2 configuration settings I used above look like the following:

 <appSettings>  
  <add key="AzureEmulator" value="C:\Program Files\Windows Azure Emulator\emulator\csrun.exe" />  
  <add key="AzureComputeArguments" value="..\..\..\..\source\Project.Azure\csx\Release ..\..\..\..\source\Project.Azure\bin\Release\ServiceConfiguration.cscfg" />  
 </appSettings>  

:: Cleaning-up the Azure Compute Emulator

Once the test suite run is complete, clean-up should take place in which the previously deployed package is removed from the compute emulator and the compute emulator is shutdown. Both of these tasks require a slight work around due to the lack of options available for the CSRun command-line tool.

A single deployed package can be removed from the compute emulator if its deployment identifier is available (see the "/remove:<DeploymentId>" option on this page). However keeping track of the deployment identifier would require some additional logic that is not needed. This logic could be built but instead I elected to simply remove ALL deployed packages using the "/removeAll" option. I did this because I only deploy one package at a time (future implementations could be more sophisticated and allow for multiple deployments but at the moment I have elected for the simplest approach).

Removing all the deployment packages (as mentioned above) does just that, it does not shutdown the Azure compute emulator. Unfortunately the CSRun command-line tool does not provide a shutdown option for the compute emulator (like ti does for the storage emulator) so I have elected to kill the process manually after the test run.

The following code in the testing framework's assembly cleanup method does what I have described above:

 public static void Cleanup()
 {
   var emulator = ConfigurationSettings.Settings["AzureEmulator"];

   using (var process = new Process())
   {
     process.StartInfo = new ProcessStartInfo(emulator, "/removeAll");
     process.Start();

     process.WaitForExit();

     Assert.AreEqual<int>(0, process.ExitCode, "Removing all the packages from the Azure compute emulator failed");
   }

   foreach (var process in Process.GetProcessesByName("DFService"))
   {
     process.Kill();
     process.WaitForExit();
   }  
 }

:: Conclusion

Once the configuration and code changes mentioned above have been included in the Azure project and the testing framework the Azure package will be deployed into the Azure compute emulator at the beginning of each and every test suite run. Subsequently at the end of the test suite run the Azure package will be "un-deployed"and the Azure compute emulator will be forcibly shutdown. Note that this works both locally for developers running automated tests as well as on our development team's continuous integration (CI) server that is using NAnt and MSBuild to compile and test the application.

Tuesday, 6 December 2011

Attending LeWeb'11 in Paris


I'll be fortunate enough to attend LeWeb'11 in Paris this week.

LeWeb's theme this year is Social, Local & Mobile (SOLOMO). There are some great guest speakers and workshops over the 3 days and I'm especially interested to see how Karl Largerfeld's opening address to LeWeb attendees ties in with the SOLOMO theme given that he is the fashion designer at CHANEL.

I'll also be posting a review of LeWeb'11 a few weeks after the event so stay tuned.

-------------------------------

:: Update


The insights gained from LeWeb'11 can be found on this post.

Thursday, 17 November 2011

Data-tier Applications, DAC Framework and SQL Azure Integration with a Continuous Integration (CI) Process

:: Introduction

Leveraging the efficiency and cost savings of the "cloud" within an enterprise requires the use of specific tools applicable to the PAAS provider used. When it comes to working with SQL Azure (Microsoft's cloud based relational database engine), I have found that data-tier applications along with some supporting deployment infrastructure (PowerShell Scripts) work particularly well to synchronize both the database changes across a development team and to deploy those changes to other testing environments as they make their way to a production system.

:: Data-tier Applications

Data-tier applications are a project type within Visual Studio 2010 that allows development teams to manage database changes across ALL of their environments whether in the cloud or on-premiss. Developers work with individual SQL scripts within the project which define anything from table schema's to indexes to stored procedures. Once a change has been made to a data-tier application's scripts, the entire project can be compiled and a DAC Package (called DACPAC from here on out) is then created that can be deployed to a SQL Server 2008 R2 or SQL Azure database instance.

For details on how to create and setup a data-tier application for the first time see this link. 

:: DACPAC Files

A DACPAC is simply a file with a ".dacpac" extension that contains all the schema, stored procedures, users and policies required to setup a new database or upgrade an existing one. Note that DACPACs should not be confused with BACPACs. BACPACs are a newer format put forward by Microsoft and the file format can contain everything a DACPAC can but can also include data. 

There is also a version number (with a format of x.x.x.x) embedded within the DACPAC file which can be used to version the DACPAC. Unfortunately this version number is not updated automatically by project when the data-tier application is modified. Therefore a developer will need to increment it manually before deployment. To access and modify this version number right-click on the data-tier application project and find it under the "Project Settings" tab.

:: Deploying Data-tier Applications

Before a data-tier application can be continuously upgraded with a newer version (the typical workflow of any development team), it must first be deployed. The deployment essentially creates the initial database, sets it up as a data-tier application within the database engine (either SQL Server 2008 R2 or SQL Azure) and then goes about deploying all the schema, store procedures, users... etc that are contained within the DACPAC.

The DACPAC can be easily deployed from within Visual Studio by right-clicking the data-tier application project and selecting deploy (note that the connection string within the data-tier application project will need to be setup before doing this). Alternatively you can use SQL Server Management Studio (SSMS) or some PowerShell commands: How to: Deploy a Data-tier Application

Note that the above deployment is really just a one time operation. Once the initial setup has taken place and the data-tier application is deployed, you'll only need to upgrade the data-tier application from then on (unless you delete the database and start again which is handy for certain developers who might break things and need to start afresh).

:: Upgrading Data-tier Applications

There are a variety of mechanisms available to deploy a DACPAC file to a SQL Server 2008 R2 or SQL Azure database instance. Unfortunately the documentation available for these mechanisms is fairly minimal and therefore it makes it difficult to assess what mechanisms to use for each deployment scenario you may have within your enterprise. Here are the ones I have come across, the way to invoke them and the scenario(s) I think they are applicable to:
  • Within Visual Studio
    • Invoke: Within Visual Studio right-click on the data-tier application project and select "Deploy". Note that Visual Studio figures out
    • Details: Depending on the current configuration (e.g. Debug/Release) the appropriately set connection string with be used when the project is deployed. Therefore the connection string can be changed to access a local SQL Server 2008 R2 database instance or a SQL Azure database instance
    • Scenario: This deployment scenario is most applicable to developers working locally during  a typical development workflow of repeatedly modifying code, compiling and deploying it and then debugging the result. I highly recommend that developer's use a local instance of SQL Server 2008 R2 for two reasons:
      • No data transfer costs are accrued as ALL data transfer occurs on a developer's local machine and not back and forth between SQL Azure and their local machine.
      • DACPAC deployments are much faster and more reliable when performed locally since deploying a DACPAC to SQL Azure can take minutes/hours depending on the connection and the database connections can be dropped mid-deployment (this won't cause an error but the deployment will need to be restarted)
    • This 20min video, by a program manager on the Microsoft SQL Server Developer Manageability team, explains a range of information about data-tier applications and their use within Visual Studio.
  • Within SQL Server Management Studio (SSMS)
    • Invoke: Within SSMS open the "Management" folder and then the "Data-tier Applications" node. Right-click the data-tier application you would like to upgrade and select "Upgrade Data-tier Application..."
    • Details: Just follow the on-screen instructions making sure to save the SQL Script that the DACPAC uses to apply the changes to the database instance. On some occasions the upgrade wizard has failed but because I saved the SQL Script I could execute it myself within SSMS.
    • Scenario: This is the easiest way to get started when deploying a DACPAC to SQL Azure since SSMS provides a easy to follow wizard. Note however that I have seen this entire process take 1-2 hours to upgrade a database (40 tables + 50 stored procedures). Being able to automate this step is very helpful which the next section on PowerShell scripting will explain.
  • Using a DAC Framework enabled PowerShell Script
    • Invoke: You'll need to create your own PowerShell script to manage the upgrade process of one or more of your data-tier applications residing on SQL Server 2008 R2 or within SQL Azure. Note that if you are unfamiliar with PowerShell and the DAC Framework creating a robust and reliable script will take significant development effort.
    • Details: I created my own parameterized PowerShell script that accepts a DACPAC file and some credentials to a SQL Azure database, subsequently connects to it and then upgrades the existing data-tier application. Two sources of information that I used when creating my PowerShell script were the following:
    • Scenario: This is the most appropriate method for building an automated process for managing data-tier applications that can be used by a continuous integration (CI) system.  
    • Note that side-by-side upgrades are the older way to upgrade a data-tier application and were used with the DAC Framework v1.0. There is a newer upgrade process called in-place upgrades and is supported by the DAC Framework v1.1. I decided to use the in-place upgrade since it doesn't create copies of the "pre-upgrade" database. In order to perform an in-place upgrade using PowerShell make sure to use the DacStore.IncrementalUpgrade Method and not the DacStore.Upgrade Method which is now obsolete. Also note that the DacUpgradeOptions passed into the DacStore.IncrementalUpgrade Method control things like whether or not to ignore data loss, ignore drift, skip policy validation and/or perform rollbacks on failure. Make sure to select these options properly for each environment that you're upgrading (i.e. you may not care about drift on your local or CI databases but in your production database database drift should be a cause for concern).
:: Data-tier Application Upgrade Architecture across Environments

The following diagram shows how I used a data-tier application's DACPAC output to upgrade various SQL Server 2008 R2 and SQL Azure database instances across various environments (from local development through to production).


Note that only the changes to the SQL scripts within the data-tier application in Visual Studio are committed (not the DACPAC output that is built locally). The DACPAC is rebuilt on CI using NAnt and MSBuild and is used to immediately upgrade the CI's SQL Azure database instance. The same DACPAC is then archived and used in subsequent deployments for Staging, UAT and Production. The reason behind this build architecture of "build once and deploy to all subsequent environments" is outlined in the book called Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation.

Also note that the "Upgrade" step in both the CI and Staging/UAT/Production environments is performed via the parameterized PowerShell script mentioned in a section above.