Okay, here is the scenario:

Occasionally, the web application would respond slowly, an order of magnitude slower than usual. This is still on the developer’s machine, not on production, by the way. Once it started working slowly, it will continue to do so until the machine was rebooted, and then it will work fast, for a time.

This was a cause for great concern, of course. So we set out to investigate. It looked like the moment the machine is getting busy, the slowdown starts. Reducing the load on the machine would speed up the application almost immediately. So a reboot was not necessary.

Just to clarify, this meant that while VS.Net was open, the application crawled, and when it was closed, it flew. My suggestion, moving all the work to notepad and NAnt on the command line was accepted with… less than enthusiastic reaction.

I put it down to machine load and swapping, and didn’t gave it much thought until today. The probable reason for the slowdown hit me while I was driving to work, we are using NHibernate second level caching (the Sys Cache implementation, which is using the ASP.Net Cache). Because ASP.Net cache is really smart, it start releasing things from the cache on low memory conditions.

After working for several hours, VS.Net brought the machine to a low memory condition, and the ASP.Net cache simply evicted everything from the cache. This mean that on every request, the data was loaded from the database, instead of being pulled from the cache.

The moment we closed VS.Net, a lot of memory was freed, and the cache became much less aggressive about freeing memory, so we suddenly had no database activity, and the application performance was stunning again. We still have the problem, and the resolution is still to close VS.Net when you want good performance from the application, but at least we have a (good) reason why this is happening.

The real solution is to upgrade the machine, of course. I have my eyes on this monster, at the moment.