5 .NET Performance Issues Where Dev and Ops Play the Blame Game

I spend a large portion of my time working with customers to review performance issues in production environments. I was asked by some colleagues to share some insight into some of the more interesting findings.

Before launching into a list of offenders, I think it’s worth explaining how often I hear the phrase “We need to prove it’s not our tin but their code” and how this relates to the discoveries here.  Typically, when a developer sits down to write some code, they do it in isolation, they’ll apply theory and best practise and throw in a bit of experience to come up with the best (hopefully) code they can produce for the task.

What happens, too often, is when the code reaches a production environment it’s executed in a way not predicted by the developer, architect or designer. Quite often the dynamics of a live environment varies drastically from the test and pre-production environments in terms of data and patterns of use. Just recently a client told me how their eCommerce site goes from a normal state of “browsing products” to a Black Friday or major event that’s “buy-only”, their application behaved very different under each circumstance.

So all that aside, let’s take a look of some of my favourite top 5 from this year…

1. Object Relational Mapping (ORM)

ORM is a broad term, but I apply it to anything that abstracts a developer from writing raw SQL – this includes LINQ, Entity Framework and NHibernate.

In a nutshell, something like Entity Framework allows a developer to create a neat design of their object and use syntax within .NET to query and manipulate the objects, which then results in queries and inserts into an underlying datastore. Figure 1 shows the type of view and code a developer would use.

Figure 1 – Entity Framework and the underlying code

Very few developers get a chance to review what goes on under the hood. I can normally tell when an Entity Framework is being used because it generates a lot of square brackets!

In fact, the best Entity Framework generated query I’ve encountered when pasted into Word covered 21 pages! To quote my customer at the time, “No sane human being would write that”. Most of the SQL was to cater for NULL cases.

In addition to creating lengthy SQL calls, ORMs can also create a lot of SQL calls — especially in production environments where data volumes can vary drastically from those in development environments and developers didn’t foresee the potential behaviour of their code. A client sent me this screenshot of their application, which spent close to 25 seconds executing about 2,000 queries into their database, this was all Entity Framework developed code.

BUT – before going on I need to point out – I like Entity Framework. It speeds up development time and improves collaboration between teams – just be careful with it and ensure you monitor what it does in the backend.

2. Data Volumes in Queries

Another client recently sent me the following screenshot to enquire what a SNIReadSyncOverAsync call was? I’ve seen this a few times; essentially this is the code that runs in .NET after running a query to get the data you’ve just queried.

I put together the following as a neat example of how this works, (No EF here I’m afraid).

The query we’re executing is to query all 1.6 million postcodes in the UK! The resulting underlying code shows our query is fast. The DBA’s done a great job, but retrieving and reading this data is is the time-consuming part and that’s what SNIReadSyncOverAsync is. You’ll see this a lot in scenarios where you’re retrieving a lot of data or you’re reading a cursor over a slow network connection.

3. Web Service and WCF calls

In a similar manner to our ORMs making multiple database calls, I’ve seen production code perform in a similar manner with Web Service and WCF calls. In one case, a front-end UI component was responsible for crunching 8 seconds worth of time performing close to 300 backend web service calls simply to render a grid control (just think of all that redundant SOAP information passed to and fro):

The same happens in WCF when you don’t consider how many calls your code may need to make in a production environment:

Another trap to be aware of in WCF is the Throttling settings. Early versions of WCF had very low Throttling settings which meant that in high volume sites you suddenly hit the buffers. Take a quick review of what your WCF applications are using, such as:

<serviceThrottling maxConcurrentCalls=”500″

maxConcurrentInstances =”100″

maxConcurrentSessions =”200″/>

Older implementations of the framework also hit issues with the maximum number of threads available, which will result in seeing a lot of “WaitOneNative” as your code in stuck in a “Holding Pattern”:

4. File and Disk IO

If you’re doing a lot of file manipulation, consider using some of the Win32 API calls instead of managed code. I was tasked a few years back writing a quick utility to recursively work through files scattered amongst ~100,000 directories and update some text in them. When we ran the tests with my initial code, it took close to a day to execute! By swapping out to Win32 APIs, I shortened the time to around 30 mins!

5. Task.WaitAll and The Task Parallel Library

I’ll cover this in another post, but I’m starting to see a huge adoption in the use of the Task Parallel Library (TPL) in applications such as Insurance Aggregators, sites that compile as much information from multiple insurers for quoting insurance premiums. The same applies to a number of travel sites retrieving flight or hotel details from multiple providers.

What happens from a DevOps perspective is that the production code you’re managing gets much harder to understand and maintain. Even visualising the complexity of what’s going on under the hood becomes far more challenging than non-multithreading applications. In short, you want to reduce the amount of time your application spends waiting on tasks to complete, therefore keep an eye on how much time your application spends waiting on tasks.

The Enter and ReliableEnter methods of the Monitor object are similar examples of code hanging around before it can continue. In these cases, code is “locking and blocking”, i.e. a Thread is executing on an object and locking out other threads. (refer to https://msdn.microsoft.com/en-us/library/hf5de04k(v=vs.110).aspx for more information on this one).

Interested in gaining end-to-end visibility over your complex application? Check out a FREE trial of AppDynamics today!

Three Clicks to .NET Performance Resolution

Root cause diagnosis remains one of the most difficult issues with modern distributed application environments. A performance issue such as slow end-user response time or slow system to system response time could reside in hundreds, even thousands of places within your environment. How can you narrow down the search and discover the root cause?

When issues lie outside of your application and reside in the cloud, Microsoft OS, any third-party application, or a myriad of other places — the search and diagnosis becomes a daunting task. These issues would be nearly impossible to detect without Machine Snapshots — which capture the state of a server at a specific moment in time — and AppDynamics since nothing in your code or backend would show an issue, you would just have an inexplicably-slow response time.

Making matters worse, you can’t replicate the problem.

Two approaches to root cause discovery

Start with the user experience and transaction response time

Using Machine Snapshots, you can see which processes are contributing to the high response time spike and investigate what’s happening on each node. These snapshots give you an x-ray vision vision of how your environment is performing at any given time and allow you to drill down with code-level visibility.

If necessary, drill down to the infrastructure level

Attempting to first diagnose at the infrastructure level is an archaic method which will take a countless amount of time. Previously, IT professionals would start at the infrastructure level and work inwards, becoming more granular. With SOA and modern environments this method is inefficient because there can we thousands of nodes on the infrastructure level. However, Machine Snapshots allow you to drill down to the process level and make performance decisions to tune the overall performance.

The test

I decided to implement my hypothesis inside a test environment to show how Machine Snapshots can immensely help with diagnosis in a real-life situation.

In this test scenario, we’ve created a lot of environment problems such as server health, response time spike, throughput dropped, and an increase in error rates. All issues were affected at the same time.

Screen Shot 2014-12-09 at 9.09.39 AM

When you drill down to the error details you see all transactions are affected and failing due to a connection timeout to the database as it runs on the same server.
Screen Shot 2014-12-09 at 9.09.48 AM

With every Snapshot, our Machine Agents collect all metrics including memory and CPU usage. In this instance, you can see the hardware metrics showing extremely high CPU usage.

Screen Shot 2014-12-09 at 9.09.56 AM

We’d like to drill down into the machine and find out what’s going on, globally, to cause the high CPU usage. Using Machine Snapshots, we can get granular analysis into every process to see which is the root cause.

Screen Shot 2014-12-09 at 9.10.06 AM

As you can see, several processes are using 100% CPU at the same time. The backup processes are killing the server, causing the overall performance issue.

Screen Shot 2014-12-09 at 9.10.15 AM

The chart below shows when CPU gets to 100% your customers are abandoning their purchases, hurting your bottom line. As an Ops or DevOps professional, this is the only metric that will matter to your suited business counterpart, however, it’s important to note in this instance as response time rises your revenue drops.

Your business no longer runs on software, your business is software.

Screen Shot 2014-12-09 at 9.10.23 AM

Finally, the root cause is discovered — it ends up being a backup utility causing the entire application to run slowly. With AppDynamics and Machine Snapshots you’re able to monitor your environment beyond your typical applications. As one of my customers said, “it’s like having a task manager on the remote server at the moment when the CPU or memory usage are high.”

Screen Shot 2014-12-09 at 9.10.40 AM

With Machine Snapshots, a previously tedious and painstaking task becomes a little bit easier to manager. Save time, save money, and get back to doing your job.

Test drive a FREE trial of AppDynamics and see the power of Machine Snapshots today!

What’s new in the Summer ‘14 release for Java & .NET Developers

As part of AppDynamics’ Summer ‘14 release, we added support for a plethora of new features in the Java & .NET product modules.  What does this news mean for Java & .Net developers?  Let’s take a closer look at how our best in class feature set enables developers to deliver better performing, higher quality apps.

Machine snapshots for .NET

This industry-first feature allows .NET developers to drill into the physical health of a machine that hosts a .NET application. Correlate application performance to exactly what was happening on the server at the time of the incident.

View health metrics like CPU and memory consumption per process on the machine:

Screen Shot 2014-08-12 at 3.55.52 PM

Monitor the state of IIS app pools queues:

Screen Shot 2014-08-12 at 3.56.03 PM

Correlate physical machine health with business transaction performance using snapshots:

Screen Shot 2014-08-12 at 3.56.13 PM

For developers, understanding exactly what occurred on the machine when a particular exception or error was thrown, and automatically correlating this data with performance metrics is invaluable information for finding and fixing bugs and production issues.

CLR crashes for .NET

In another industry-first offering for .NET developers, AppDynamics now automatically detects and tracks CLR crashes in IIS and provides crash details to find the root-cause of .NET application crashes even faster than before.

Screen Shot 2014-08-12 at 3.56.21 PM

With deep-dive visibility into CLR crashes, .NET developers now have another datapoint they can analyze in real-time while investigating the root-cause of application crashes and performance bottlenecks.

Service endpoints for .NET

Service endpoints are all about providing flexibility in how they visualize different services within the UI.  By allowing users to create a business transaction for any service, .NET developers can now create a custom endpoint that they are particularly interested in.  This is most helpful in large, complex environments that are common within our enterprise customers.  A development team may only work on a particular part of the larger application that only deals with one or two services.  By allowing dev teams to create a business transaction for just those services, they can get granular visibility into how their part of the app is performing.

Step 1: Choose a tier and add a rule:

Screen Shot 2014-08-12 at 3.56.29 PM

Step 2: Select entry point type:

Screen Shot 2014-08-12 at 3.56.40 PM

Step 3: Define the service endpoint and create the definition:

Screen Shot 2014-08-12 at 3.56.46 PM

Service endpoints give developers the flexible view they need to track the granular metrics for the services they care about most.

Object instance tracking

AppDynamics now has object instance tracking support for .NET so customers can profile memory utilization of .NET objects.

Screen Shot 2014-08-12 at 3.56.57 PM

This in-depth memory profiling allows developers to find and fix memory leaks to help prevent .NET instances from crashing.

With added support for Java 8 and WebSockets, AppDynamics has extended support for our industry-leading Java application performance management product module.

Java 8 support

With this release we’ve increased compatibility with the latest generation of Java & Scala tools and frameworks.  By providing support for Java 8, AppDynamics now supports new features like lambda expressions and parallel operations.

WebSockets

AppDynamics has added support for WebSockets, providing new visibility into long-lived bi-directional WebSockets transactions.  WebSockets are oftentimes used in critical trading, messaging, and social media platforms and other applications that rely on real-time communication.

Test drive these new features yourself by signing up for a free trial today.

Diving Into What’s New in Java & .NET Monitoring

In the AppDynamics Spring 2014 release we added quite a few features to our Java and .NET APM solutions. With the addition of the service endpoints, an improved JMX console, JVM crash detection and crash reports, additional support for many popular frameworks, and async support we have the best APM solution in the marketplace for Java and .NET applications.

Added support for frameworks:

  • TypeSafe Play/Akka

  • Google Web Toolkit

  • JAX-RS 2.0

  • Apache Synapse

  • Apple WebObjects

Service Endpoints

With the addition of service endpoints customers with large SOA environments can define specific service points to track metrics and get associated business transaction information. Service endpoints helps service owners monitor and troubleshoot their own specific services within a large set of services:

JMX Console

The JMX console has been greatly improved to add the ability to manage complex attributes and provide executing mBean methods and updating mBean attributes:

JVM Crash Detector

The JVM crash detector has been improved to provide crash reports with dump files that allow tracing the root cause of JVM crashes:

Async Support

We  added improved support for asynchronous calls and added a waterfall timeline for better clarity in where time is spent during requests:

  

AppDynamics for .NET applications has been greatly improved by adding better integration and support for Windows AzureASP.Net MVC 5, improved Windows Communication Foundation support, and RabbitMQ support:

 

 

Take five minutes to get complete visibility into the performance of your production applications with AppDynamics today.

Instrumenting .NET applications with AppDynamics using NuGet

Introduction

One of the coolest things to come out of the .NET stable at AppD this week was the NuGet package for Azure Cloud Services. NuGet makes it a breeze to deploy our .NET agent along with your web and worker roles from inside Visual Studio. For those unfamiliar with NuGet, more information can be found here.

Our NuGet package ensures that the .NET agent is deployed at the same time when the role is published to the cloud. After adding it to the project you’ll never have to worry about deploying the agent when you swap your hosting environment from staging to production in Azure or when Azure changes the machine from under your instance. For the remainder of the post I’ll use a web role to demonstrate how to quickly install our NuGet package, changes it makes to your solution and how to edit the configuration by hand if needed. Even though I’ll use a web role, things work exactly the same way for a worker role.

Installation

So, without further ado, let’s take a look at how to quickly instrument .NET code in Azure using AppD’s NuGet package for Windows Azure Cloud Services. NuGet packages can be added via the command line or the GUI. In order to use the command line, we need to bring up the package manager console in Visual Studio as shown below

PackageManager

In the console, type ‘install-package AppDynamics.WindowsAzure.CloudServices’ to install the package. This will bring up the following UI where you can enter the information needed by the agent to talk to the controller and upload metrics. You should have received this information in the welcome email from AppDynamics.

Azure

The ‘Application Name’ is the name of the application in the controller under which the metrics reported by this agent will be stored. When ‘Test Connection’ is checked we will check the information entered by trying to connect to the controller. An error message will be displayed if the test connection is unsuccessful. That’s it, enter the information, click apply and we’re done. Easy Peasy. No more adding files one by one or modifying scripts by hand. Once deployed, instances of this web role will start reporting metrics as soon as they experience any traffic. Oh, and by the way, if you prefer to use a GUI instead of typing commands on the console, the same thing can be done by right-clicking on the solution in Visual Studio and choosing ‘Manage NuGet Package’.

Anatomy of the package

If you look closely at the solution explorer you’ll notice that a new folder called ‘AppDynamics’ has been created in the solution explorer. On expanding the folder you’ll find the following two files:

  • Installer of the latest and greatest .NET agent.
  • Startup.cmd
The startup script makes sure that the agent gets installed as a part of the deployment process on Azure. Other than adding these files we also change the ServiceDefinition.csdef file to add a startup task as shown below.

Screen Shot 2013-11-27 at 8.11.27 PM

In case, you need to change the controller information you entered in the GUI while installing the package, it can be done by editing the startup section of the csdef file shown above. Application name, controller URL, port, account key etc. can all be changed. On re-deploying the role to Azure, these new values will come into effect.

Next Steps

Microsoft Developer Evangelist, Bruno Terkaly blogged about monitoring the performance of multi-tiered Windows Azure based web applications. Find out more on Microsoft Developer Network.

Find out more in our step-by-step guide on instrumenting .NET applications using AppDynamics Pro. Take five minutes to get complete visibility into the performance of your production applications with AppDynamics Pro today.

As always, please feel free to comment if you think I have missed something or if you have a request for content in an upcoming post.

AppDynamics Pro on the Windows Azure Store

Over a year ago, AppDynamics announced a partnership with Microsoft and launched AppDynamics Lite on the Windows Azure Store. With AppDynamics Lite, Windows Azure users were able to easily monitor their applications at the code level, allowing them to identify and diagnose performance bottlenecks in real time. Today we’re happy to announce that AppDynamics Pro is now available as an addon in the Windows Azure store, which makes it easier for developers to get complete visibility into their mission-critical applications running on Windows Azure.

  • Easier/simplified buying experience in Windows Azure Store
  • Tiered pricing based on number of agents and VM size
  • Easy deployment from Visual Studio with NuGet
  • Out-of-the-box support for more Windows Azure services

“AppDynamics is one of only a handful of application monitoring solutions that works on Windows Azure, and the only one that provides the level of visibility required in our distributed and complex application environments,” said James Graham, project manager at MacMillan Publishers. “The AppDynamics monitoring solution provides insight into how our .NET applications perform at a code level, which is invaluable in the creation of a dynamic, fulfilling user experience for our students.”

Easy buying experience

Purchasing the AppDynamics Pro add-on in the Windows Azure Store only takes a couple of minutes. In the Azure portal click NEW at the bottom left of the screen and then select STORE. Search for AppDynamics, choose your plan, add-on name and region.

4-choose-appdynamics

Tiered pricing

AppDynamics Pro for Windows Azure features new tiered pricing based on the size of your VM (extra small, small or medium, large, or extra large) and the number of agents required (1, 5 or 10). This new pricing allows organizations with smaller applications to pay less to store their monitoring data than those with larger, more heavily trafficked apps. The cost is added to your monthly Windows Azure bill, and you can cancel or change your plan at any time.

AppDynamics on Windows Azure Pricing

Deploying with NuGet

Use the AppDynamics NuGet package to deploy AppDynamics Pro with your solution from Visual Studio. For detailed instructions check out the how-to guide.

2-vs-package-search

Monitoring with AppDynamics

  • Monitor the health of Windows Azure applications
  • Troubleshoot performance problems in real time
  • Rapidly diagnose root cause of performance problems
  • Dynamically scale up and scale down their Windows Azure application based on performance metrics

AppDynamics .Net App

Additional platform support

AppDynamics Pro automatically detects and monitors most Azure services out-of-the-box, including web and worker roles, SQL, Azure Blob, Azure Queue and Windows Azure Service Bus. In addition, AppDynamics Pro now supports MVC 4. Find out more in our getting started guide for Windows Azure.

Get started monitoring your Windows Azure app by adding the AppDynamics Pro add-on in the Windows Azure Store.