Tuesday, December 27, 2011

Deploying Packages to the Azure Compute Emulator for Automated Testing across Local Environments and a Continuous Integration (CI) Server

:: Introduction

When developing on the Microsoft Windows Azure Platform, a storage and compute emulator is available for running applications via the Windows Azure Tools (as of writing this post I am using v1.6). An Azure application package can be deployed to the emulator (running locally) instead of to an actual Azure compute node in the Azure cloud. The benefits of this are two-fold:

  • Deployments to the emulator are dramatically faster (it takes seconds to deploy to the emulator as opposed to minutes to deploy to the Azure cloud)
  • There are no additional costs associated with deploying to and/or using the emulator to run an application package since the emulator is a locally installed application (deploying to the Azure cloud would incur additional compute and bandwidth charges).

Therefore when I needed to create some automated tests to run against various WCF web methods that were apart of my Azure web role, I first needed to deploy my web role (within the Azure package) to the Azure emulator. Note that I wanted to be able to run the automated tests both locally and on our development team's continuous (CI) server, therefore the solution I chose needed to accomodate both environments.

:: Force the Azure Project to Package during Compilation

Before an Azure package can be deployed to the emulator it must first be packaged. Since v1.4 of the Windows Azure Tools, the csx folder and its contents are no longer generated during compilation (See these release notes for v1.4 and search for the breaking change: "PackageForComputeEmulator"). So in order to force the generation of the csx folder we need to specify that the package should be generated for the compute emulator within the Azure project file (*.ccproj). I embedded the this within an existing property group containing some other items:

 <PropertyGroup>  
  <VisualStudioVersion Condition="'$(VisualStudioVersion)' == ''">10.0</VisualStudioVersion>  
  <CloudExtensionsDir Condition=" '$(CloudExtensionsDir)' == '' ">$(MSBuildExtensionsPath)\Microsoft\VisualStudio\v$(VisualStudioVersion)\Windows Azure Tools\1.6\</CloudExtensionsDir>  
  <!-- This flag generates the 'csx' package folder when this project is compiled (it does not require a separate package command) -->  
  <PackageForComputeEmulator>true</PackageForComputeEmulator>  
 </PropertyGroup>  

:: Deploying to the Azure Compute Emulator

Now that we have the csx package folder generated after each compilation run we can use the CSRun command-line tool to deploy the folder to the emulator at the start of each test suite run. I was able to accomplish this as cleanly as possible by embedding the package deployment within our testing framework's assembly initialize method which looks like the following:

 public static void Initialize(TestContext context)  
 {  
   var emulator = ConfigurationSettings.Settings["AzureEmulator"];  
   var computeArgs = ConfigurationSettings.Settings["AzureComputeArguments"];
 
   using (var process = new Process())  
   {  
     process.StartInfo = new ProcessStartInfo(emulator, computeArgs);  
     process.Start();  
     process.WaitForExit();
 
     Assert.AreEqual<int>(0, process.ExitCode, "Starting the Azure compute emulator and loading it with a package failed");             
   }  
 }  

And the 2 configuration settings I used above look like the following:

 <appSettings>  
  <add key="AzureEmulator" value="C:\Program Files\Windows Azure Emulator\emulator\csrun.exe" />  
  <add key="AzureComputeArguments" value="..\..\..\..\source\Project.Azure\csx\Release ..\..\..\..\source\Project.Azure\bin\Release\ServiceConfiguration.cscfg" />  
 </appSettings>  

:: Cleaning-up the Azure Compute Emulator

Once the test suite run is complete, clean-up should take place in which the previously deployed package is removed from the compute emulator and the compute emulator is shutdown. Both of these tasks require a slight work around due to the lack of options available for the CSRun command-line tool.

A single deployed package can be removed from the compute emulator if its deployment identifier is available (see the "/remove:<DeploymentId>" option on this page). However keeping track of the deployment identifier would require some additional logic that is not needed. This logic could be built but instead I elected to simply remove ALL deployed packages using the "/removeAll" option. I did this because I only deploy one package at a time (future implementations could be more sophisticated and allow for multiple deployments but at the moment I have elected for the simplest approach).

Removing all the deployment packages (as mentioned above) does just that, it does not shutdown the Azure compute emulator. Unfortunately the CSRun command-line tool does not provide a shutdown option for the compute emulator (like ti does for the storage emulator) so I have elected to kill the process manually after the test run.

The following code in the testing framework's assembly cleanup method does what I have described above:

 public static void Cleanup()
 {
   var emulator = ConfigurationSettings.Settings["AzureEmulator"];

   using (var process = new Process())
   {
     process.StartInfo = new ProcessStartInfo(emulator, "/removeAll");
     process.Start();

     process.WaitForExit();

     Assert.AreEqual<int>(0, process.ExitCode, "Removing all the packages from the Azure compute emulator failed");
   }

   foreach (var process in Process.GetProcessesByName("DFService"))
   {
     process.Kill();
     process.WaitForExit();
   }  
 }

:: Conclusion

Once the configuration and code changes mentioned above have been included in the Azure project and the testing framework the Azure package will be deployed into the Azure compute emulator at the beginning of each and every test suite run. Subsequently at the end of the test suite run the Azure package will be "un-deployed"and the Azure compute emulator will be forcibly shutdown. Note that this works both locally for developers running automated tests as well as on our development team's continuous integration (CI) server that is using NAnt and MSBuild to compile and test the application.

Tuesday, December 6, 2011

Attending LeWeb'11 in Paris


I'll be fortunate enough to attend LeWeb'11 in Paris this week.

LeWeb's theme this year is Social, Local & Mobile (SOLOMO). There are some great guest speakers and workshops over the 3 days and I'm especially interested to see how Karl Largerfeld's opening address to LeWeb attendees ties in with the SOLOMO theme given that he is the fashion designer at CHANEL.

I'll also be posting a review of LeWeb'11 a few weeks after the event so stay tuned.

-------------------------------

:: Update


The insights gained from LeWeb'11 can be found on this post.

Thursday, November 17, 2011

Data-tier Applications, DAC Framework and SQL Azure Integration with a Continuous Integration (CI) Process

:: Introduction

Leveraging the efficiency and cost savings of the "cloud" within an enterprise requires the use of specific tools applicable to the PAAS provider used. When it comes to working with SQL Azure (Microsoft's cloud based relational database engine), I have found that data-tier applications along with some supporting deployment infrastructure (PowerShell Scripts) work particularly well to synchronize both the database changes across a development team and to deploy those changes to other testing environments as they make their way to a production system.

:: Data-tier Applications

Data-tier applications are a project type within Visual Studio 2010 that allows development teams to manage database changes across ALL of their environments whether in the cloud or on-premiss. Developers work with individual SQL scripts within the project which define anything from table schema's to indexes to stored procedures. Once a change has been made to a data-tier application's scripts, the entire project can be compiled and a DAC Package (called DACPAC from here on out) is then created that can be deployed to a SQL Server 2008 R2 or SQL Azure database instance.

For details on how to create and setup a data-tier application for the first time see this link

:: DACPAC Files

A DACPAC is simply a file with a ".dacpac" extension that contains all the schema, stored procedures, users and policies required to setup a new database or upgrade an existing one. Note that DACPACs should not be confused with BACPACs. BACPACs are a newer format put forward by Microsoft and the file format can contain everything a DACPAC can but can also include data. 

There is also a version number (with a format of x.x.x.x) embedded within the DACPAC file which can be used to version the DACPAC. Unfortunately this version number is not updated automatically by project when the data-tier application is modified. Therefore a developer will need to increment it manually before deployment. To access and modify this version number right-click on the data-tier application project and find it under the "Project Settings" tab.

:: Deploying Data-tier Applications

Before a data-tier application can be continuously upgraded with a newer version (the typical workflow of any development team), it must first be deployed. The deployment essentially creates the initial database, sets it up as a data-tier application within the database engine (either SQL Server 2008 R2 or SQL Azure) and then goes about deploying all the schema, store procedures, users... etc that are contained within the DACPAC.

The DACPAC can be easily deployed from within Visual Studio by right-clicking the data-tier application project and selecting deploy (note that the connection string within the data-tier application project will need to be setup before doing this). Alternatively you can use SQL Server Management Studio (SSMS) or some PowerShell commands: How to: Deploy a Data-tier Application

Note that the above deployment is really just a one time operation. Once the initial setup has taken place and the data-tier application is deployed, you'll only need to upgrade the data-tier application from then on (unless you delete the database and start again which is handy for certain developers who might break things and need to start afresh).

:: Upgrading Data-tier Applications

There are a variety of mechanisms available to deploy a DACPAC file to a SQL Server 2008 R2 or SQL Azure database instance. Unfortunately the documentation available for these mechanisms is fairly minimal and therefore it makes it difficult to assess what mechanisms to use for each deployment scenario you may have within your enterprise. Here are the ones I have come across, the way to invoke them and the scenario(s) I think they are applicable to:
  • Within Visual Studio
    • Invoke: Within Visual Studio right-click on the data-tier application project and select "Deploy". Note that Visual Studio figures out
    • Details: Depending on the current configuration (e.g. Debug/Release) the appropriately set connection string with be used when the project is deployed. Therefore the connection string can be changed to access a local SQL Server 2008 R2 database instance or a SQL Azure database instance
    • Scenario: This deployment scenario is most applicable to developers working locally during  a typical development workflow of repeatedly modifying code, compiling and deploying it and then debugging the result. I highly recommend that developer's use a local instance of SQL Server 2008 R2 for two reasons:
      • No data transfer costs are accrued as ALL data transfer occurs on a developer's local machine and not back and forth between SQL Azure and their local machine.
      • DACPAC deployments are much faster and more reliable when performed locally since deploying a DACPAC to SQL Azure can take minutes/hours depending on the connection and the database connections can be dropped mid-deployment (this won't cause an error but the deployment will need to be restarted)
    • This 20min video, by a program manager on the Microsoft SQL Server Developer Manageability team, explains a range of information about data-tier applications and their use within Visual Studio.
  • Within SQL Server Management Studio (SSMS)
    • Invoke: Within SSMS open the "Management" folder and then the "Data-tier Applications" node. Right-click the data-tier application you would like to upgrade and select "Upgrade Data-tier Application..."
    • Details: Just follow the on-screen instructions making sure to save the SQL Script that the DACPAC uses to apply the changes to the database instance. On some occasions the upgrade wizard has failed but because I saved the SQL Script I could execute it myself within SSMS.
    • Scenario: This is the easiest way to get started when deploying a DACPAC to SQL Azure since SSMS provides a easy to follow wizard. Note however that I have seen this entire process take 1-2 hours to upgrade a database (40 tables + 50 stored procedures). Being able to automate this step is very helpful which the next section on PowerShell scripting will explain.
  • Using a DAC Framework enabled PowerShell Script
    • Invoke: You'll need to create your own PowerShell script to manage the upgrade process of one or more of your data-tier applications residing on SQL Server 2008 R2 or within SQL Azure. Note that if you are unfamiliar with PowerShell and the DAC Framework creating a robust and reliable script will take significant development effort.
    • Details: I created my own parameterized PowerShell script that accepts a DACPAC file and some credentials to a SQL Azure database, subsequently connects to it and then upgrades the existing data-tier application. Two sources of information that I used when creating my PowerShell script were the following:
    • Scenario: This is the most appropriate method for building an automated process for managing data-tier applications that can be used by a continuous integration (CI) system.  
    • Note that side-by-side upgrades are the older way to upgrade a data-tier application and were used with the DAC Framework v1.0. There is a newer upgrade process called in-place upgrades and is supported by the DAC Framework v1.1. I decided to use the in-place upgrade since it doesn't create copies of the "pre-upgrade" database. In order to perform an in-place upgrade using PowerShell make sure to use the DacStore.IncrementalUpgrade Method and not the DacStore.Upgrade Method which is now obsolete. Also note that the DacUpgradeOptions passed into the DacStore.IncrementalUpgrade Method control things like whether or not to ignore data loss, ignore drift, skip policy validation and/or perform rollbacks on failure. Make sure to select these options properly for each environment that you're upgrading (i.e. you may not care about drift on your local or CI databases but in your production database database drift should be a cause for concern).
:: Data-tier Application Upgrade Architecture across Environments

The following diagram shows how I used a data-tier application's DACPAC output to upgrade various SQL Server 2008 R2 and SQL Azure database instances across various environments (from local development through to production).


Note that only the changes to the SQL scripts within the data-tier application in Visual Studio are committed (not the DACPAC output that is built locally). The DACPAC is rebuilt on CI using NAnt and MSBuild and is used to immediately upgrade the CI's SQL Azure database instance. The same DACPAC is then archived and used in subsequent deployments for Staging, UAT and Production. The reason behind this build architecture of "build once and deploy to all subsequent environments" is outlined in the book called Continuous Delivery: Reliable Software Releases through Build, Test, and Deployment Automation.

Also note that the "Upgrade" step in both the CI and Staging/UAT/Production environments is performed via the parameterized PowerShell script mentioned in a section above.

Tuesday, July 12, 2011

Agile Contracts

:: Questions

Lately I've been curious about how one would structure an agile development team within a consulting organization which typically sells fixed-priced projects.
  • Can an Agile development team really work effectively when the fixed-priced contract, which governs the deliverables produced, is signed upfront before the project has started and therefore before the requirements have been adequately understood by either the consultant or the client?
  • Should a fixed-price contract be avoided entirely or can the incentives somehow be lined up properly such that both the consultant and the client have a large enough stake in producing the best possible outcome together?
:: Resources

I have scoured the web and found the following articles which deal with Agile development and how to structure contracts around it's development methodology:
:: Conclusions

In an ideal world one can try to structure an agile development methodology around a fixed-priced contract in various ways but most of them only work well when the project is running smoothly. As soon problems start occurring most of the contracts start looking like the wrong approach. Eventually when the rubber hits the road and scope starts screeping, money runs out and/or deadlines loom, something has to give and invariable its the agile methodology because fixed-price contracts are less malleable than the underlying development process. This would be all well and good unless the abandonment of core agile practises during an emergency period of the project did not have adverse effects... but unfortunately time and time again I have seen adverse effects push projects even further into trouble.


-------------------------------

:: Update

I recently came across a 44 page PDF titled Agile Contracts Primer

Thursday, June 9, 2011

Configuring an Android Application's WebView Component

I recently started on a new project at work to build out a prototype Android application for one of our clients. I'm using the standard Eclipse IDE, Android SDK and ADT Plugin for Eclipse that the Android team recommends.

During development I came across a couple issues related to the WebView component (which is both a layout and a class) that caused a bit of confusion. In case you're not familiar with it, a WebView component can be embedded into an Activity and allows for the viewing of webpages (essentially it mimics the functionality that exists in the device's built-in browser application). I discovered that the WebView component requires some additional tweaking in order for it to behave in a similar manner to the device's built-in browser. The following are the problems and their solutions that needed to be implemented before the Android prototype worked as required:

:: Enabling Flash Content

When viewing a webpage with embedded Flash video content, a pop-up window would display informing the user that they needed to install the latest Flash plug-in. Even after installing the plug-in via the built-in browser and the Android Market App, the pop-up window still appeared and the Flash content was unable to be played. The phone's built-n browser however was able to play the Flash video content (without any pop-up being displayed).

The solution to this problem was that the WebView component needed to have JavaScript and any plugins explicitly enabled before the Flash video content would play. Here is what I included in my Activity's OnCreate method to explicitly enable both these properties:

WebSettings webSettings = webView.getSettings();
webSettings.setJavaScriptEnabled(true);
webSettings.setPluginsEnabled(true)

Note that the webView variable above was previously declared as a private instance variable of type WebView.

:: Initially Rendering in Zoomed Out Mode

When the WebView component loaded and became visible to the user, it arbitrarily rendered a webpage partially zoomed in. This gave a sub-optimal user experience and so the solution was to initially render the webpage in a fully zoomed out mode so that the user could view the whole webpage and decide for themselves if they wanted to perform a pinch gesture to zoom in on a particular area. Again, here is what I included in my Activity's OnCreate method so that the WebView component rendered any webpage fully zoomed out:

WebSettings webSettings = webView.getSettings();
webSettings.setLoadWithOverviewMode(true);
webSettings.setUseWideViewPort(true);

Again note that the webView variable above was previously declared as a private instance variable of type WebView.

:: Forcing WebView to Load all URLs

For a certain subset of URLs that I was trying to load within my WebView component, my application would quit and the phone's internal web browser would launch and load the webpage successfully. I therefore needed to find a solution that would force all URLs to load within my WebView instead of being launched externally on the phone's internal web browser.

The solution entailed creating a CustomWebViewClient class that extended WebViewClient and then wiring up the CustomWebViewClient to handle the shouldOverrideUrlLoading method. This allowed me to force the WebView within my application to render all URLs I was presenting to the user:

private WebView webView = new WebView();
private CustomWebViewClient webViewClient = new CustomWebViewClient();

protected void OnCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    webView.setWebViewClient(webViewClient);
}

private class CustomWebViewClient extends WebViewClient {
    @Override
    public boolean shouldOverrideUrlLoading(WebView view, String url) {
        view.loadUrl(url);
        return true;
    }
}

-------------------------------

:: Update

I recently found that the Android developer resources have a tutorial on setting up a WebView correctly. It goes into detail for some (but not all) of the items described above.

Wednesday, June 8, 2011

.NET Code Contracts: Invariants Explained

I have recently started using .NET Code Contracts. I love the streamlined approach to implementing pre-conditions and post-conditions that previously required multi-line if statements along with the ability to check coding assumptions statically and even at runtime (release or debug builds). .NET Code Contracts are another valuable tool in a software engineers arsenal to combat the growing complexity of software systems.

The latest MSDN magazine (June, 2011) has an thorough article on invariants which are an element of .NET Code contracts. The article will go into much more detail but essentially invariants allow for the verification of an object's internal state before and after a set of operations. To create an object invariant, a class simply needs to contain a new private method with the ContractInvariantMethod attribute and then specify any conditions within the private method that should hold true before and after any operation on the class or instances of the class. Here is an example of what the object invariant method would look like for a class called News which has internal state for the Title and Body of a news article:

[ContractInvariantMethod]
private void ObjectInvariant()
{
     Contract.Invariant(!String.IsNullOrEmpty(Title));
     Contract.Invariant(!String.IsNullOrEmpty(Body));
}

What this accomplishes for programmers is that if they inadvertently try to alter the internal state on an instance of the News class to something it should not be (such as setting Title = string.empty), the .NET Code Contract static checker will warn the programmer that they are violating the object's invariant.

Thus, programmers can now design and build classes with the help of .NET Code Contracts that enforce correct API usage. This is a very powerful feature that will avoid subtle bugs from ever occurring.

Thursday, May 5, 2011

Technical Debt

:: Definitions

Technical debt is a high-level metaphor used to describe the quality of software source code. It was first introduced by Ward Cunningham back in 1992 and has since take shape in many different forms. Today it can be used informally to describe the state of a software application to a non-technical audience or it can be used more formally for measuring technical aspects related to software source code or specific tasks required to "clean" it up. Given that the term "technical debt" can have a wide array of definitions, the following sources provide some deeper insight about the term: 
A software application is only as good as the team that designs, builds and tests it. Therefore, technical debt has many sources and can be introduced by architects, developers or testers. It usually starts out being just a small issue but if left unmanaged, can grow to become very debilitating for a development team to produce software in a timely manner. Technical debt can be associated at any level of the software application stack whether embedded in the underlying architecture, classes, methods, algorithms, documentation and/or build systems. It is therefore important in software projects to not only know what requirements have yet to be implemented fully but also to know what technical debt lurks within the software so that it can be managed effectively.

:: Measurement

The holy grail for measuring technical debt would be to measure the impact of a single software change in terms of both the value it produces (most likely profitability) and the risk it either increases or mitigates. The ideal software change would maximize overall value while reducing overall risk. As an industry, software development is a very long way from this fine grained approach to measuring technical debt but there are many techniques that can be used (either individually or combined together) to measure the potential burden that a piece of software may have and thus provides some better insight. Like tracking finances you never really know where or how much money is being spent until you keep track of it and the same is true for technical debt... you never really know what software components might need attention or how bad they actually are until you measure them directly.

There are a number of metrics that can be used to measure specific aspects of technical debt across software source code. Unfortunately none of these metrics are able to provide a complete picture of the technical debt for a piece of software but taken together they can provide some key insights into its overall quality. Note that care should be taken to fully understand what a metric is in fact measuring and then to not over-state its importance. I have seen many development teams become religious over certain metrics and subsequently implement policies to stay within some arbitrary bounds. This can often lead to development practises that are detrimental to the actual software being developed. Measuring technical debt is more of an art than a science but quantitative values can still be helpful in the arts.

Here are some of the metrics I have used to measure source code:
  • SLOC (Source Lines of Code): The size of a software application in SLOC isn't a great metric in an of itself, but when used as a comparison mechanism (e.g. the ratio of production code to testing code) it becomes a good indication of relative code quality.
  • Cyclomatic Complexity: A great indication of how complicated the code paths are in a portion of code (usually a method). For methods, it is also a very good indication of how many unit tests are required to effectively traverse and thus test each code path. This is also one of the few metrics statistically correlated with buggy software.
  • Coupling: A number of metrics are defined in this category but of particular interest are fan-in and fan-out metrics which measure the number of calling or called routines for a portion of code.
  • Branch Coverage (Also known as code coverage): Measures the amount of production code that is covered by unit tests. If you think you have tested a method or class sufficiently this is a good metric to observe to verify that thought.
  • Duplicate code: Duplicate code is a particularly interesting metric because the code that is duplicated can in fact be of a high quality. The problems arises when the duplicated code needs to be modified due to a bug. As a result the developer must make sure that each and every occurrence of the bug is fixed otherwise one occurrence of the bug will be fixed but one or possibly many others will remain. A false sense of security that a bug is fixed when it is not can be a disaster in certain contexts.
  • Nested depth level: Some algorithms require nested 'if', 'for' or 'while' statements and an large number of nested statements (also called the depth level) increases the complexity of a portion of code and makes it harder for any one developer to understand it without considerable effort (leading to possibly buggier code).
  • Violations of coding standards: If a coding standard is defined there a usually a number of tools available that either enforce compliance with the standards or simply count the number of infractions.
Apart from simply using metrics to quantify the technical debt of a software application, I have also found the need to capture various software "cleanup" tasks. Anything from dead code to code refactoring to build system improvements can be added to a ticket in your bug tracking system, labelled as "technical debt" and estimated accordingly. By doing this the technical debt within an application can be is tracked over time and observed as it either increases or decreases. It can therefore be managed in a pragmatic way instead of avoided or simply kept unknown.

:: Tooling

There are a whole range of open-source and commercial static analysis software tools available to capture metrics and enforce their compliance:

Tuesday, April 19, 2011

Designing Scalable Systems

Many software based problems with large amounts of persistent data have already, or are currently, out growing the typical design of a single database for all persistence requirements. The need to build scalable software systems is becoming paramount as the amount of data that software applications deal with grows and as we move software applications to the cloud where the addition of more distributed computing power is only an API call away.

Ricky Ho has put together a comprehensive list of general principles and common techniques to use when designing scalable systems. It's a great list to reference while thinking about and designing new components for a software system that is required to scale.

Monday, April 4, 2011

Coding the Architecture

I recently found this website called Coding the Architecture. They have an online book full of resources on how to approach, document and refine software architecture. The information contained on the site is simple, concise and backed by some experienced software developers.

One of their ideas/practises I find particularly compelling is what they call 'C4' otherwise known as context, containers, components and classes. The authors highlight the usefulness of decomposing the architecture of an entire software system into 4 separate architecture diagram types. This separation helps development teams understand their software system(s) more clearly by only showing a specific view of them. By reducing the amount of information you put into a single "architecture" diagram, the more easily it is to digest, understand and then act upon.

They also recently blogged about a similar topic in a post called Deliberate practise, effective sketches.

Sunday, March 20, 2011

Achieving Transactions with a Database and File System

:: The Overarching Problem

Recently I was tasked with solving an interesting software design problem. The existing software's implementation wrote many files to a file system through a web service and the requirement was to add in a mechanism to store metadata for specific files. The files still needed to be written to the file system and were not to be modified in any way.  Other than that, it was up to me to devise a solution for how best to store the file metadata. Here is a diagram to illustrate the basic architecture (the red question mark represents the unknown for how to store the file metadata):



:: Design Approaches

I opted early to use a database to store the file metadata instead of more files in the file system that stored associated metadata (that would very quickly become unmanageable). Given the use of a database, the difficult aspect of this problem was how I would guarantee that both a file gets written to the file system and its associated metadata is stored in the database table. I could not get into a situation where one of the operations succeeded while the other failed... it was all or nothing. While researching various solutions and talking to other developers I came up with these 3 approaches as outlined below.

   1. Microsoft SQL Server FILESTREAM Data Type

I won't go into too much detail about this SQL Server FILESTREAM data type since it is explained thoroughly here (and an example here) but essentially it allows for structured data to be stored in a database while also allowing the storage of unstructured data on an NTFS filesystem. Essentially a single SQL operation can be used be used to store both the file on the file system and the file's metadata in the database. With regards to the transaction around this operation here is what I found from the first article:

FILESTREAM is the only solution that provides transactional consistency of structured and unstructured data as well as integrated management, security, low-cost, and excellent streaming performance. This is accomplished by storing the structured data in the database files and the unstructured BLOB data in the file system, while maintaining transactional consistency between the two stores. More details of the FILESTREAM architecture are given in the “Overview of FILESTREAM” section later in this white paper.

I also came across these two issues that may restrict the ability to use this FILESTREAM data type:
  1. SQL Server maintains a link of sorts from table rows to the FILESTREAM files associated with them. This means that deleting or renaming any FILESTREAM files directly through the file system will result in database corruption.
  2. The FILESTREAM data type is only available on SQL Server 2008
Using the FILESTREAM data type would definitely be a viable solution to explore further but I continued to investigate other alternative solutions.

   2. Modified File Query Architecture

Given that it's not trivial to guarantee that both a file system and SQL operation either succeed or fail together, another approach can be used where each operation can happen independently but only the final operation in the sequence of operations "commits/registers" the overall action (in my case the overall action is that both the file and its metadata are stored successfully or not at all). 

The thought here was to perform the operation which stores the file to the file system first. If this fails I know the overall action has failed. It it succeeds I can move onto the next action in the sequence which is the SQL operation.

For the SQL operation I would be storing the file's metadata in a row within a database table. This row will have an identifier for the file stored on the file system (whether it be its location on the file system or simply its unique filename). At this point if the SQL operation succeeds the overall action succeeds. The tricky part comes if the SQL operation fails, I would then be left with a file on the file system which is unreferenced by any row in the database table (the beginnings of data integrity problems). Given that the files stored on the file system are queried by another process, this can very easily cause a corrupted system where files did not have any associated metadata.

So what to do if the SQL operation fails? The simplest (but naive) solution would be to simply delete the file on the file system if the SQL operation failed for any reason. This has 2 problems:
  1. What if the deletion of the file fails? You could build in some retry logic but if the deletion failed indefinitely, the file would sit lying around on the file system forever as an orphaned file with no associated metadata.
  2. What if the other process queried for the file on the file system before its metadata was inserted into the database? Given that the file is stored first and then its metadata is stored second, it is theoretically possible for the other process to query a file that is on the file system but that does not yet have its metadata stored in the database. This could be solved with sufficient error case handling (if metadata doesn't exist try again until it is) but this would not be a very elegant solution.
To use this approach effectively, the other process's query mechanism would need to be updated such that it queried the database table first (to determine what metadata exists) and then only queried for files on the file system to which there was metadata for. 

The drawbacks of this approach is that the other process needs to be modified (this could be done but the less code changes the better) and there would still be the issue of orphaned files, with no metadata, that exist on the file system. A separate clean-up routine could be built that periodically tries to take care of (i.e. delete) the orphaned files but again, I'd have to write a whole other application to perform clean-up... there has to be a better way.

   3. Transactional NTFS & The .NET TransactionScope Class

I knew that the .NET framework has the ability to wrap SQL operations in a transaction but I knew this didn't work for traditional file system operations. What I wanted to guarantee was that if any of the file operations or SQL operations failed, then none of them would be committed. I needed an atomic transaction for both the file system write and the SQL insert statement.

I posted this question on Stack Overflow and got a ton of responses. It turns out that since Windows Vista was introduced, NTFS has been bundled with a special component called Transactional NTFS (TxF). According to Wikipedia, Transactional NTFS "brings the concept of atomic transactions to the NTFS file system, allowing Windows application developers to write file output routines that are guaranteed either to succeed completely or to fail completely". Thus I now had a mechanism to enable transactions for both SQL and file system operations.

Upon researching more about how I would add this transactional support to file system operations within my application, I came across this Transactional NTFS Screen-cast by Jason Olsen. Not only does he highlight the difference between a regular file system operation and a transacted file system operation, but he also dives deeper into the underlying code that makes the transaction possible. As you will notice from the screen-cast, the underlying code supporting the transactions is very low-level and uses the Win32 API functions. I'd prefer to interact with managed code (at the architectural layer that this is being implemented in) so I found a more convenient managed wrapper that allows me to use the well known .NET TransactionScope class (This is the same class that SQL operations can be wrapped with to form transactions).

Note that using Transactional NTFS requires developing and running the application on Windows Vista or above. Additionally, transacted file operations can be performed on network shares but only if their underlying file systems support Transactional NTFS themselves.

:: Conclusion

After evaluating these 3 approaches I decided that the use of "Transactional NTFS & The .NET TransactionScope Class" would solve the underlying problem most effectively while minimizing the amount of code base changes I would need to make in order to get this feature operational.

Friday, March 11, 2011

Continuous Integration: Tooling, Processes & Implicit Communication

Continuous integration (CI) is a process and set of tools that development teams can use to continually manage the software they develop, test and deploy. In this post I will highlight the various tooling I have used on various .NET projects that harnesses CI, I will discuss some key processes around using CI that make it most effective and I will present some of the more subtle ways CI allows for implicit communication between team members that isn't so obvious when you're starting out with the practice.

Hudson is a really good open-source, Java, web-based application that can be used as a CI build system on Windows or Unix/Linux. What I find particularly compelling about Hudson is how easy it is to install and get started with a new project. First download the Hudson WAR from the homepage. Once you have started it from the command-line using java -jar hudson.war almost everything else is done from its GUI based menus; there is no need to configure any XML or to have an intimate knowledge of its API (CruiseControl is like this).


:: Tooling

Redsolo has a pretty complete post on setting up Hudson for a .NET project and getting the unit-tests and some static analysis (FxCop) to run. I initially configure my own Hudson instances very similarly to his post but over time I have altered it in the following ways:
  • The use of a source control management (SCM) system is crucial when building a CI infrastructure. I use Subversion (SVN) but you could just as easily use Git if you're more the distributed version control type. It is critical for CI to have a central place where all developers can regularly commit their code and configuration changes for a software project. Applications may work in isolation (when worked on independently) but time and time again I have found that the integration of a team's code changes and configuration can be very very challenging. CI promotes integrating early and often and therefore tries, and I believe succeeds, at avoiding the headaches that are associated with integrating at the end of a project (which by the way is very difficult to estimate against as you have no idea what kinds of bugs are lurking when it comes time to fully integrate).
  • I use NAnt to easily specify and execute a range of tasks from project compilation, database management, unit and acceptance test execution, static analysis, documentation generation and deployment. Hudson has a NAnt plugin which allows it to use NAnt as a build step. Thus, when Hudson notices that a developer has checked-in code, it does a full checkout of the subversion tree and executes a specific NAnt target to begin compilation or any other task required for that particular project.
  • I generally use MSTest and not NUnit for unit-testing. Both integrate nicely with Hudson but Hudson only knows how to interpret JUnit and NUnit test results. Because I still want a graph of the test results to show up on the project's main page, I install the MSTest plugin that converts the test report (a TRX file) into a format that Hudson understands natively (which happens to be JUnit XML).
  • I use FxCop, StyleCop and NCover for static analysis of my projects. FxCop and StyleCop are used more for helping the development team adhere to some design and implementation best practises as well as coding standings throughout development whereas NCover is used more for "after the fact" analysis of the code base (e.g. it highlights potential production code our team has not tested adequately). I use the Violations plugin to visually capture the number of FxCop and StyleCop violations within the code base after each new build. I also use a combination of a NAnt target (to generate the NCover report) and the HTML publisher plugin to publish the NCover report to Hudson so it is accessible via a link on the main page.
  • I automatically generate Doxygen documentation from the XML comments associated with the classes and methods within my code base. I use a NAnt target along with the HTML publisher plugin to publish the HTML to Hudson so it is accessible via a link on the main page. This way the development team always has a reference to the most up-to-date API documentation for the code base.
  • I find the Task Scanner plugin particularly helpful to keep track of various keywords in the code base that developers may leave in to remind themselves or others of what to fix or attend to later. Any custom tags such as FIXME, TODO, @deprecated, ??? can all be tracked and are shown visually on the main project page.
  • I use the Claim plugin which allows a specific developer to claim a build break and comment on what the problem/solution might be. The "claim" mechanism is sticky which means that if a developer continually breaks the build they will be continually responsible for fixing it. This plugin is great for helping a distributed team communicate implicitly about who is responsible for build breaks (instead of email or instant message communication between the whole team - less communication spam is always a good thing).
  • I also archive specific build artifacts from each CI build that I may want to analyze in the future. That way if I ever want to go back in time and discover where a particular bug entered the code base (and by whom) I can do so. It is much easier to simply download the executables from the CI build server and run them than to checkout a specific revision from source control, compile it and then run it to verify it does or does not have the bug I'm looking for.
Hudson's website also has some best practices to follow for Build Management, Release Management, Deployment Automation, and Test Orchestration

:: Key Processes

Here are some key processes that I use on teams to harness the benefits of CI:
  • Only one Commit per Build - Finding and solving software bugs that cause a build to break can be like searching for a needle in a haystack. Finding that needle can be made much easier if your software uses logging (such as log4net), a team member has deep insight into certain parts of the application that they know cause certain problems, specific sets of code and configuration changes can be isolated and labelled as the changes that may have caused the bug, or your software application has a testing suite that can pin-point failure areas. Each of these can be harnessed more effectively and precisely if a CI build is executed after each and every developer's commit (i.e. a single CI build does not encompass multiple commits). The reason for this is, if a build failure does occur, it is far easier to analyze a single code change set from one developer and begin the process of finding the bug than to begin analyzing multiple change sets from multiple developers. Whomever's code change actually broke the build at this point is very hard to determine without all developer's working together instead of just the single developer who's code change broke the build. Caveat: The only additional note on this is that it becomes difficult to keep to this requirement as a team grows in size and the build takes longer and longer to execute. For example: If a single build takes 1hr to execute, in a given working day a development team will only be able to run 8-10 CI builds. The more developers that are on a team, the more commits your team will perform each day. At some point your team won't be able to keep to this "one commit per build" policy but it is definitely something to strive for if you can.
  • Start a Build from Scratch - When setting up your CI build make sure to re-checkout the FULL source controlled tree (do not update the local copy of the tree) and fully rebuild (compile) all executables and libraries from their raw source code. Simply doing an update of your local copy of  the tree or rebuilding only the executable/libraries that are needed, allows for the possibility of artifacts, from the previous build, to be used in the current build. The most effective way to eliminate unintended or corrupted build breaks and, more importantly, to truly test how everything integrates from your raw source code, the CI build should start from scratch. Delete everything, start from a freshly created directory and then checkout your source code from source control. Compilation, testing and deployments can then be performed on the raw source code and not any other artifacts still lying around in the directory from a previous build.
  • Multiple Builds - On my projects I usually need multiple builds. I need one build, called CI Build, that automatically builds after a single source control commit by a developer. I then have another build called Nightly Build, which is run each and every night and therefore builds against all the source control commits from the previous day. Then I may have a QA Build that has a different deployment procedure to some QA server for the testers. The QA Build is usually setup to build after a manual step that one of the testers will execute. Lastly, I have a Production Build that has a deployment procedure specifically geared towards deploying the application into a production environment and doing all the appropriate configuration and tests required to validate that it is stable and functional in production. Of course the Production Build is very important to get right and should be protected against unauthorized execution (i.e. password protect its usage or have only a select few individuals with the permissions to execute it).
  • Build Both the Release and Debug Configurations - When compiling .NET projects in Visual Studio you have the ability to compile in Release or Debug (or others if you have defined your own). I would highly recommended having both a CI Release Build and a CI Debug Build. I have found very subtle differences between the two when it comes to compilation, timing for unit-tests, configuration & static analysis results. I've had situations where the debug build passes but the release build does not and it was usually the result of some mis-configuration but occasionally I catch a real nasty timing issue in a unit or acceptance test. Note that I care about both build configurations because developer's routinely develop and test an application which was compiled in debug mode whereas the software actually being released to production is compiled in release mode.
  • Keep the Main Source Control Tree Pristine - Each and every developer works from and modifies the main source control tree. Each modification to the tree affects all other developers so it is imperative that it be kept clean and at the highest quality. The software that a development team produces is what is in the tree, therefore whatever quality the tree has the software has also. This is a slightly more concrete way of thinking about the intangible aspects of software. It is hard to visualize what software really is and what quality it has but it is much easier to visualize the files checked-in to a source control tree and the quality of each of those files. Therefore, keep the files that are checked-in to the source control tree pristine at at the highest quality.
  • Priority Number 1: Fix the Broken Build - When a build breaks during CI it effects the whole team. Developer's should be immediately informed that the source control tree is unstable and they then know NOT to commit any changes or to update their local code versions to the broken/unstable one. Developer's can of course continue to develop their feature(s) or resolve their bug(s) but eventually they're going to want to commit code changes or update their local versions with everyone else's  code changes. With a broken build, developer's are forced to work in isolation which is the very thing CI is trying to avoid. A broken build delays integration and should therefore be avoided at all costs.

:: Implicit Communication

Some projects can be routinely bogged down by communication overhead between team members which makes it difficult to be productive (less communication overhead means more time building and improving software). Question such as: why does this code not compile, who's responsible for this piece of code, what effect will this code change have, what exactly are we testing, what are the exact files I need in order to test this on another machine, are all valid questions developers can ask but if there is an easier way to communicate the answers other than direct developer-to-developer interaction, productivity can be greatly improved. This is particularly useful for development teams that are distributed by both location and time but even small teams located in the same office can see marked improvements by reducing communication overhead by using CI practices.

Having a single, repeatable process which is what CI fosters, allows entire development teams to be on the same page. At any point in time you can verify the state of the code base: is it stable, is it broken or is it in transition (i.e. currently building, testing and deploying). If things are stable then I can concentrate on developing features for my software application. If it's broken then I know something is wrong and the CI build system will most likely help me out by informing me why it's broken or at least try to point me in the right direction (i.e. what commit by what developer broke the build, did a test fail, did deployment fail, is it a configuration issue or a code change... etc). If it's in transition I know that it's potentially unstable and should wait until I know more. Following these basic principles, development teams keep in-sync by knowing at any point in time whether the life blood of their software application (it's source code and configuration) are stable or whether there is a problem that needs fixing.

Extending this CI process with the tools mentioned above: static analysis reports, the ability for specific developers to claim broken builds and comment on them, auto-generated API documentation and code base task scanners, a development team has relevant information at its finger tips that helps them communicate without direct face-to-face or email communication. I call this implicit communication. If, as a developer, I can quickly determine who broke the build, if they have taken responsibility for it and what they have decided to do to fix it, I can quickly go back to the feature I was previously building or I can get in touch with the person responsible for breaking the build and help them by providing a deeper insight about the reason for the failure. Apart from build breaks, I can access the latest API documentation and verify that I understand a method's contract that I am about to use. I can also quickly verify the cyclomatic complexity of a method that I am refactoring and decide whether it has enough test coverage. All of this can be done from my own desk with no direct communication with another developer. I have found that on some projects without this CI infrastructure there is always a "guru" developer who seems to know the most about a project (usually because he's been there the longest) and everyone needs their insight about each and every software related issue. You can imagine what kind of bottle neck that causes and what might happen if the "guru" leaves the project... chaos.

At the end of the day this implicit communication helps me build better software as I have the information I need to design and implement software features and fix bugs. By reducing the communication overhead that CI tools and processes allow, I am more productive at developing, improving and delivering the software customers pay for.

-------------------------------

:: UPDATE

Recently the team that was building Hudson left the project and subsequently forked it into what is now called Jenkins (This was due to some political issues with Oracle over the name Hudson and who had control over it). If you already have a Hudson instance and wish to switch to Jenkins (as the team is mostly over their now) this Jenkins wiki article will help you make the switch.

Sunday, February 20, 2011

Problems Unit-Testing Quartz.NET's IJob.Execute Method

I've been using an open source library at work called Quartz.NET which is an Enterprise Job Scheduler for the .NET platform. Quartz.NET is modeled after the popular Quartz Scheduler for the Java platform and shares many similar elements. I needed a scheduler for a variety of reasons but it is mostly used to fire "jobs" at specified times that perform some work like downloading a file via FTP and storing it to disk.

I've been using Quartz.NET version 1.0.3 which has the following IJob interface and can be used to implement your own custom "jobs" that perform some task:

namespace Quartz
{
     public interface IJob 
     {
          void Execute(JobExecutionContext context);
     }
}

The Execute method looks fairly simple and non-problematic until you realize that it is the single entry point that Quartz.NET has into your own application code (i.e. the scheduler calls the implemented Execute method on your custom jobs when a trigger fires). Problems start to arise if you ever want to thoroughly unit-test your custom jobs as the JobExecutionContext parameter passed in is a concrete object and not an interface. A previous post of mine has already mentioned my theory of software development with bugs so in that light let me explain further.

In order to effectively unit-test you need to isolate the method under test and test ONLY the logic contained within that singular method (not any other class's methods that may be called from the method under test). Misko Hevery of Google has a good presentation on what makes code hard to test and the above Quartz.NET interface code makes it hard to pass in a completely clean mock object as the parameter since the JobExecutionContext parameter is concrete and cannot be completed mocked (unless all its methods are virtual which they are not).

I've been using RhinoMocks to easily mock up my own interfaced objects, set expectations on what methods should be called on those mocked up objects and then asserting that the methods were in fact called (see this post for details). When RhinoMocks is used to mock up an interface or class it essentially wraps a proxy object around the interface, abstract class or concrete class you give it. It works great for interfaces but it can only override interface methods, delegates or virtual methods. Therefore if you try to mock up any class with non-virtual methods and then need to set expectations on those non-virtual methods you're out of luck (See these RhinoMock limitations). Most of the methods on the JobExecutionContext are virtual but there are some key ones related to the next, previous and current fire times which are not virtual and therefore cannot be set by RhinoMocks.

After understanding this problem I posted a question to the Quartz.NET Google Group asking why the JobExecutionContext parameter is not an interface. As it turns out the next version of Quartz.NET will use an interface for this parameter instead of a concrete type.

So there are 3 options that I can think of when using the Quartz.NET IJob interface for your own custom jobs (Some are more geared towards fully unit-testing the logic contained within your custom jobs):
  1. Continue using the Quartz.NET 1.0.3 library and either avoid testing your custom job logic completely by ignoring the next, previous and current fire time properties OR create a concrete JobExecutionContext object within your test code and pass that into your custom job's Execute method. I would avoid the first approach as you're then missing test coverage but the second approach is not ideal as you're not isolating your method logic and therefore a single change to one method may end up breaking multiple disparate unit-tests which take a while to determine whats wrong and fix.
  2. Wait for the next version of the Quartz.NET library which has an interfaced parameter for the IJob.Execute() method. This should be released sometime in 2011.
  3. Modify the Quartz.NET 1.0.3 open source library yourself so that the JobExecutionContext parameter on the IJob.Execute() method is an interface instead of a concrete type. This isn't an ideal solution as you now have to manage your own 3rd party library but if you can't wait for the next Quartz.NET release but still need the convenience of an interfaced JobExecutionContext parameter this is your only real option.

Friday, February 18, 2011

A Theory of Software Development: Bugs

Software development is a difficult task. It requires in-depth knowledge of the technical problem being solved, an acute awareness of how the technical problem fits into the framework of the business it functions within, an adherence to a set of established or evolving processes and finally, the ability to regularity adapt to new requirements, new technologies, new people and many forms of technical shortcomings throughout the lifecycle of project. Essentially software development is the process of solving many technical problems while simultaneously managing complexity and adapting to change.

Throughout a project many challenges are overcome and the ones that are not usually manifest themselves one way or another as bugs within the software application. And one thing is clear during today's modern software development: there will be many software bugs that need to be managed. Managing these bugs will become an routine task that every development team will face over and over again throughout the development lifecycle. So where do these bugs come from? Bugs find their way into software applications for 2 main reasons:
  1. Insufficient Requirements: Bugs that surface because the software is exposed to scenarios that it was never designed to be exposed to. For example, a method is designed to do X when given data Y, however after it is built method X is given unexpected data Z which causes it to return an unexpected result or alternatively throw an exception. This can also be thought of as violating a precondition.
  2. Insufficient Code:  Bugs that surface because developers design or build the software incorrectly. For example, a method should do X (the requirement specifies that it should do X) but it is built by a developer in such a way that it does Y instead of X. This can also be thought of as violating a postcondition.
I find it helps to think about software bugs in this manner because over time you can start to see which bugs are continually cropping up and therefore which ones your team should focus on avoiding. If it's of the Insufficient Requirements variety, I find that something is going wrong in the requirements gathering and dissemination process (either the requirements that are being gathered are incorrect or incomplete or the way they are understood and used by developers are incorrect). If it's of the Insufficient Code variety, I find that there is generally not enough (or any) unit-testing and/or acceptance testing being done. Finding ways to identify these issues and correct them is paramount to continually improving the software your team builds. The rest of this post will focus on the Insufficient Code variety and how a specific type of testing can be used anticipate and mitigate these bugs.

Once the understanding of where bugs come from has been established, the next step is to understand how bugs are classified within the Insufficient Code variety. Misko Hevery at Google has a unified theory of bugs which I find particularly compelling. He classifies bugs into 3 types: logical, wiring and rendering. He goes onto argue that logical bugs are on average the most common type of bug, are notoriously hard to find and are also the hardest to fix. Thus if developers only have so much time and energy to spend on testing, they should focus their testing efforts on uncovering logic bugs more so than wiring or rendering bugs. Of all the types of testing, unit-testing is the best mechanism for uncovering logic bugs. Misko Hevery goes onto say the following about unit-testing:
Unit-tests give you greatest bang for the buck. A unit-test focuses on the most common bugs, hardest to track down and hardest to fix. And a unit-test forces you to write testable code which indirectly helps with wiring bugs. As a result when writing automated tests for your application we want to overwhelmingly focus on unit test. Unit-tests are tests which focus on the logic and focus on one class/method at a time.
A key question that can be asked after reading the above paragraphs is how then do I write testable code that makes it easier to unit-test and therefore uncover logic bugs more easily? This can be accomplished by doing the following:
  1. Methods should have parameters that are interfaces not concrete types (unless the types are primitives such as ints, strings, doubles... etc)
  2. Constructors should not do any significant work besides checking and setting their parameters to internal member variables
  3. Avoid global state at all costs (this includes using singletons and static methods)
  4. Do not use the new operator within classes whose primary focus is logical in nature (i.e. classes whose methods contain ifs, loops and calculations)
    1. Note 1: Factory classes can and should have the new operator as these classes are specifically designed and used for wiring up other classes together
    2. Note 2: Certain data structures like lists and arrays are usually fine to "new" up inside logical classes
  5. Avoid Law of Demeter violations
So here is what the above rules actually look like in source code format for a class called Process which has a Start and Stop method and performs logic functions on some scheduler and data store objects:

public class Process
{
     private IDataStore _dataStore;
     private IScheduler _scheduler;

     public Process(IScheduler scheduler, IDataStore dataStore)
     {
          if(scheduler == null)
          {
               throw new NullParamException("scheduler");
          }
          if(dataStore == null)
          {
               throw new NullParamException("dataStore");
          }
      
          _scheduler = scheduler;
          _dataStore = dataStore;
     }

     public void Start(int delay)
     {
          _scheduler.JobComplete += _dataStore.Save;

          _scheduler.Start(delay);
     }

     public void Stop(bool forceShutdown)
     {
          _scheduler.JobComplete -= _dataStore.Save;

          _scheduler.ShutDown(forceShutdown);
     }
}

This class can now be effectively unit-tested by creating mock IDataStore and IScheduler objects that are passed into the constructor upon creation (see how I do this using RhinoMocks: a dynamic mock object framework). By using this approach, only the method under test is actually being tested and not any other classes. I have found this approach for designing classes and methods to be of particular importance as a project grows larger and matures. When a healthy regression suite (set of unit-tests) has been created for an application without taking into account this approach (i.e. a single unit-test will test multiple classes at the same time) and then a change request occurs, I find that many disparate unit-tests begin to fail after making code changes. It is therefore essential to isolate each unit-test so that it tests only the logic contained within a single classes' method and no other methods. Adhering to this principle from the beginning of development will save countless hours of refactoring seemingly unconnected unit-tests that fail after a single code change.

A final note about wiring logical classes together: By using the above approach, one or more application builder classes (factory classes) will need to be built that are responsible for wiring together the application. These builder classes create ("new" up) all logic classes and use inversion of control (IoC) to pass other logic classes to each other through their constructors. This essentially wires up the entire object graph of the application. These builder classes can then be tested but the resultant tests are more integration/system tests as they verify that the application’s object graph has been setup correctly and are therefore an attempt at uncovering any possible wiring bugs that may exist.