Showing posts with label Azure. Show all posts
Showing posts with label Azure. Show all posts

Wednesday, 27 November 2024

Microsoft Azure's case study of Orbital Witness

I've been working with Microsoft for the last couple months on a case study they've written. It's about how Orbital Witness is revolutionising the legal tech sector and delivering efficiencies with Azure OpenAI.

In the case study we touch on the following concepts:

  • Pivoting early to GPT-4 and away from labelled data and more classical machine learning models
  • Building and productionising an AI Agent in late 2023 before AI Agents were a big thing
  • Embedding domain expertise into a product via prompt engineering
  • How LLM development requires a different mindset centred around iterating with human language
  • How customers are adapting to a generative AI product that behaves differently
  • A proprietary OCR system that enriches textual data

Here's that case study on Microsoft's website: https://customers.microsoft.com/en-gb/story/1827391161519296074-orbitalwitness-azure-professional-services-en-united-kingdom

Alternatively here's a PDF of the case study:

Thursday, 10 January 2013

Avoiding Downtime For Azure Web/Worker Role Instances when the Azure Infrastructure Upgrades the OS

I discovered a problem the other day where a typical web application that was hosted on 2 Azure web roles experienced downtime for approximately 6 minutes. I was initially alerted to the problem by PingDom which informed me that every single page across my web application went offline. The screenshot below is for one of these pages and it indicates 2 downtime events. The first of which was the actual problem and the second was something else:


PingDom's root cause analysis of the downtime simply indicated a "Timeout (> 30s)" from multiple locations around the globe. Given that every page monitored by PingDom indicated the same problem I was pretty sure the entire site went down. I quickly logged into the Azure Management Portal during the downtime event to observe the status of my web role instances and I noticed that one of them (actually the 2nd of two instances) was currently rebooting. I immediately got the idea that an Azure OS update must have been initiated and that I was observing the second of my web role instances being rebooted after an update. Note that Azure is a PAAS service so it automatically handles OS and infrastructure upgrades. This makes it easier to focus on whatever core application development you are doing (instead of administrating machines) but it does come at a small unexpected price (unintended consequences) which I will explain further.

I wanted to confirm my hypothesis that the OS upgrade had occurred around the same time. Thus I got in touch with Azure support to find out if in fact an OS upgrade had been initiated by the Azure infrastructure on December 20th around 15:20 PST. This is the reply I got:
Thank you for your patience. The behavior your perceived for the update is correct. One of the instances was brought online after an update and showed when the role was set to “started” it moved to the other machine in the update domain to update the node.

The behavior is by design, we wait for the machine to display the result as role started for the machine in order to start updating the other instance.

The ideal will be to try to lower the startup time for the application. [Unfortunately] this will happen every month for the machines since we just count on the role status to update the role instance in the other  domain.
I was also sent the following link by Azure support which talks about managing multi-instance Windows Azure applications.

The web application I have running does take a bit of time to boot-up due to the JIT compilation along with New Relic's .NET agent profile hook-in to IIS, but this usually takes several seconds not minutes to complete. What seems to be going on is that although the 2 web role instances are in different upgrade domains (upgrade domain 1 and 2 respectively), which causes any updates to happen in a non-overlapping schedule, in reality the updates can occur immediately one after the other which makes sense from an infrastructure perspective. And because the Azure infrastructure relies on the status of the actual role instance itself and NOT your own application's status, its entirely possible that when a web role instance, that was just updated (in upgrade domain 1), appears as Ready to the Azure infrastructure, the web application that runs on the instance might still be initializing. And if its still initializing and the second web role instance (in upgrade domain 2) is rebooted due to the OS upgrade, there is no longer a live web role responding to web requests.

There really was only 3 ways around this:

  1. Turn off automatic upgrades of the OS (but then who wants to do that manually given that web roles are a PAAS service after all).
  2. Figure out exactly what and why is causing the web role to come alive more slowly than expected and then spend the engineering resources to reduce this time substantially. Given that we'd never be able to drive that time down to zero there would always be a small but noticeable wait time (probably in seconds rather than minutes).
  3. Have 3 web roles instead of 2. This way OS upgrades that are rolled across upgrade domains will only ever affect 2 of the web roles at any given time (with a minor overlap which in my case was 1-2 minutes). This does cost more money to have an additional web role but its a really easy fix and depending on the size of role instance it might be the cheapest and easiest solution by far.
I chose option 3. In a startup time is usually more precious than money, so if something can be fixed easily with money instead of time... that's generally a good root to take.

------

Update: Since switching to using more than 2 web roles I have never seen this downtime issue happen again.

Saturday, 17 November 2012

RESAAS Wins Prestigious Microsoft Award

Yesterday my company, RESAAS, was announced as the winner of the 2012 Microsoft Impact Award in the category of Windows Azure Platform ISV Partner of the Year.

Our press release of can be found on Bloomberg and Microsoft has a few more details about our category and who we were up against on the Microsoft Partner Network.

Tuesday, 17 July 2012

Interviewed by Jonathan Rozenblit for "Canada Does Windows Azure"

Jonathan Rozenblit who is a Developer Advisor at Microsoft interviewed me for a series he does called "Canada does Windows Azure". We spent 17 min talking about how RESAAS uses Azure, why it decided to use Azure and how our developers ramped up on the platform when we first started.


Here are links to the blog posts Jonathan put on his blogs:

Tuesday, 24 January 2012

Appointed as VP of Engineering at RESAAS

I am thrilled to accept my new positon as the VP of Engineering at RESAAS. I joined the company 8 months ago as the first full-time software engineer and with the help of Tom Rossiter, CTO, we have grown the development team to 8 full-time backend, frontend and quality assurance engineers all in 6 short months. Our team has been able to successfully expand in such a short time-frame by utilizing Agile methodologies, test automation, continuous integration and database change management tools along with the dedication of some amazingly talented engineers who work very hard to continuously improve the processes, the tooling and most importantly the product RESAAS is offering.

The enterprise social network that our team is building has evolved considerably over that time period from a single .NET and SQL Server based system hosted in a data center to a scalable platform hosted in the cloud (Windows Azure). Our core platform has been able to leverage Worker roles, Web roles, SQL Azure and Azure Table and Blob Storage along with various other 3rd party services to connect real estate professionals and their clients together in real-time via their laptop or chosen mobile device.

I am looking forward to working hard and developing the full potential of both our engineering team and the RESAAS core platform in the coming months.

Note that RESAAS is a public company currently listed on the CNSX under the symbol: RSS. Here is the press release on the CNSX of my appointment as Vice President of Engineering at RESAAS. 

Thursday, 12 January 2012

Presenting at the Vancouver Windows Azure Meetup Group hosted by Microsoft

I will be giving a talk about SQL Azure and Data-tier Applications (DACPAC) at the Vancouver Windows Azure Meetup Group this coming Wednesday, January 18th, 2012. There will also be a guest speaker who will be a Sr. Azure Architect from Redmond, Washington.

The meetup will be hosted by Microsoft in their Downtown Vancouver offices and if you would like to attend please RSVP via this meetup group.

I will also be attaching the presentation slides to this post after the talk.

-----------------

The talk went really well and there were lots of great questions from the audience. Below are the presentation slides I used during the Windows Azure Meetup talk (hosted on SlideShare):


Tuesday, 27 December 2011

Deploying Packages to the Azure Compute Emulator for Automated Testing across Local Environments and a Continuous Integration (CI) Server

:: Introduction

When developing on the Microsoft Windows Azure Platform, a storage and compute emulator is available for running applications via the Windows Azure Tools (as of writing this post I am using v1.6). An Azure application package can be deployed to the emulator (running locally) instead of to an actual Azure compute node in the Azure cloud. The benefits of this are two-fold:

  • Deployments to the emulator are dramatically faster (it takes seconds to deploy to the emulator as opposed to minutes to deploy to the Azure cloud)
  • There are no additional costs associated with deploying to and/or using the emulator to run an application package since the emulator is a locally installed application (deploying to the Azure cloud would incur additional compute and bandwidth charges).

Therefore when I needed to create some automated tests to run against various WCF web methods that were apart of my Azure web role, I first needed to deploy my web role (within the Azure package) to the Azure emulator. Note that I wanted to be able to run the automated tests both locally and on our development team's continuous (CI) server, therefore the solution I chose needed to accomodate both environments.

:: Force the Azure Project to Package during Compilation

Before an Azure package can be deployed to the emulator it must first be packaged. Since v1.4 of the Windows Azure Tools, the csx folder and its contents are no longer generated during compilation (See these release notes for v1.4 and search for the breaking change: "PackageForComputeEmulator"). So in order to force the generation of the csx folder we need to specify that the package should be generated for the compute emulator within the Azure project file (*.ccproj). I embedded the this within an existing property group containing some other items:

 <PropertyGroup>  
  <VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">10.0</VisualStudioVersion>  
  <CloudExtensionsDir Condition=" '$(CloudExtensionsDir)' == '' ">$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v$(VisualStudioVersion)\Windows Azure Tools\1.6\</CloudExtensionsDir>  
  <!-- This flag generates the 'csx' package folder when this project is compiled (it does not require a separate package command) -->  
  <PackageForComputeEmulator>true</PackageForComputeEmulator>  
 </PropertyGroup>  

:: Deploying to the Azure Compute Emulator

Now that we have the csx package folder generated after each compilation run we can use the CSRun command-line tool to deploy the folder to the emulator at the start of each test suite run. I was able to accomplish this as cleanly as possible by embedding the package deployment within our testing framework's assembly initialize method which looks like the following:

 public static void Initialize(TestContext context)  
 {  
   var emulator = ConfigurationSettings.Settings["AzureEmulator"];  
   var computeArgs = ConfigurationSettings.Settings["AzureComputeArguments"];
 
   using (var process = new Process())  
   {  
     process.StartInfo = new ProcessStartInfo(emulator, computeArgs);  
     process.Start();  
     process.WaitForExit();
 
     Assert.AreEqual<int>(0, process.ExitCode, "Starting the Azure compute emulator and loading it with a package failed");             
   }  
 }  

And the 2 configuration settings I used above look like the following:

 <appSettings>  
  <add key="AzureEmulator" value="C:\Program Files\Windows Azure Emulator\emulator\csrun.exe" />  
  <add key="AzureComputeArguments" value="..\..\..\..\source\Project.Azure\csx\Release ..\..\..\..\source\Project.Azure\bin\Release\ServiceConfiguration.cscfg" />  
 </appSettings>  

:: Cleaning-up the Azure Compute Emulator

Once the test suite run is complete, clean-up should take place in which the previously deployed package is removed from the compute emulator and the compute emulator is shutdown. Both of these tasks require a slight work around due to the lack of options available for the CSRun command-line tool.

A single deployed package can be removed from the compute emulator if its deployment identifier is available (see the "/remove:<DeploymentId>" option on this page). However keeping track of the deployment identifier would require some additional logic that is not needed. This logic could be built but instead I elected to simply remove ALL deployed packages using the "/removeAll" option. I did this because I only deploy one package at a time (future implementations could be more sophisticated and allow for multiple deployments but at the moment I have elected for the simplest approach).

Removing all the deployment packages (as mentioned above) does just that, it does not shutdown the Azure compute emulator. Unfortunately the CSRun command-line tool does not provide a shutdown option for the compute emulator (like ti does for the storage emulator) so I have elected to kill the process manually after the test run.

The following code in the testing framework's assembly cleanup method does what I have described above:

 public static void Cleanup()
 {
   var emulator = ConfigurationSettings.Settings["AzureEmulator"];

   using (var process = new Process())
   {
     process.StartInfo = new ProcessStartInfo(emulator, "/removeAll");
     process.Start();

     process.WaitForExit();

     Assert.AreEqual<int>(0, process.ExitCode, "Removing all the packages from the Azure compute emulator failed");
   }

   foreach (var process in Process.GetProcessesByName("DFService"))
   {
     process.Kill();
     process.WaitForExit();
   }  
 }

:: Conclusion

Once the configuration and code changes mentioned above have been included in the Azure project and the testing framework the Azure package will be deployed into the Azure compute emulator at the beginning of each and every test suite run. Subsequently at the end of the test suite run the Azure package will be "un-deployed"and the Azure compute emulator will be forcibly shutdown. Note that this works both locally for developers running automated tests as well as on our development team's continuous integration (CI) server that is using NAnt and MSBuild to compile and test the application.