Here’s another blog post about the program I’m currently working on at my job. This is the same program I blogged about yesterday and a couple of weeks ago.
Today, I was trying to fix a performance issue. The app originally ran really fast. It just had to make a few API calls, filter and combine some data, then spit it back out in JSON format. It took less than a minute to run. But then, I was asked to add a new data element to one of the files. An element that I could only get by calling a new web service method repeatedly, once per order, for about 7000 orders. It shouldn’t be an expensive call, but the end result was that my 1 minute runtime was now up to 10 minutes.
The first thing I tried doing was adding some concurrency to those 7000 new API calls. I did that using the first technique described in this article, implementing a ConcurrentQueue. I wasn’t really optimistic that it would help much, but I thought it was worth a try. It didn’t really help at all. The program still took about 10 minutes to run. So I undid that change.
The next thing I did was to look and see if I was repeating any of the API calls. While I was processing 7000 records, there were some cases where the same sales order number was found on multiple records, so I was making extra unnecessary API calls. So I implemented a simple cache with a dictionary, saving the API call results and pulling them from cache when possible. That didn’t help much either. About 90% of the calls were still necessary, so I only got down from 10 minutes to 9 minutes. But that was at least worth doing, so I left that code in place.
Then, finally, it occurred to me to look at how I was calling the API. This new API call was part of the WCF SOAP service that I’ve mentioned previously. Well, the way I wrote my wrapper code for the API, I was creating a new call context and service client for every call. I didn’t think that would be a huge issue, but I went ahead and refactored things so all the calls used the same call context and client. Well, that got the execution time back down to one minute. So really all of that extra time was spent in whatever overhead there is in spinning up the WCF client object (and I guess tearing it down when it goes out of scope).
That was really unexpected. I hadn’t thought about it much, but I assumed the code behind the instantiation of the service client was just setting up a structure in memory. I guess that maybe it’s also establishing communication with the server? Theoretically, I could dig into it, but I don’t really have the time for that.
The moral of this story is that, when performance tuning, some of the stuff that you think will help, won’t, and some of the stuff that seems dubious, might actually make a huge difference!