Category: programming

The end of a weird week

As I alluded to on Tuesday, we had a security incident at work over the July Fourth weekend, so we’ve been spending all week cleaning up after that. There’s a statement about it on my company’s blog. I won’t link to it here, since I don’t necessarily want this post to show up in searches or referral logs related to it. (I’m not going to post anything I shouldn’t be posting, and I don’t know enough about the details to say much anyway, but just in case…)

On Tuesday, we were told to just stay home and chill, basically. They didn’t give us any real info on what happened, so that was basically just a “snow day,” as I mentioned in my last post. On Wednesday, they had made enough progress with the initial mitigation that we could start working on our disaster recovery. So, since then, it’s been a lot of “hurry up and wait” work, where servers are getting restored or rebuilt, scanned, released, and then we can do final setup and testing. I’ve been in the office Wednesday and Thursday, but I’m hoping I can get away with working from home today.

There are a lot of people working very hard on this. For me, I’m primarily a programmer, with only limited admin responsibilities. There are only two servers that I’m “officially” responsible for (out of hundreds total across the company), so mostly I’m just waiting on other stuff, answering questions, and helping out where I can. I feel a little guilty, knowing that some people are sleeping in their offices while I’m going home and sleeping in my own bed, but of course there’s not much I can do to help those folks, other than to stay out of their way.

I had a bunch of other stuff to say about this, but I think I’ll just say that I think we’re doing a pretty good job of handling this thing. Everyone has been calm and professional through it all, at least from my limited vantage point. There’s still a lot of work to do, but we’re making steady progress.

Purely by coincidence, I read an article in Communications of the ACM last week that talked about the best way to handle outages in IT. The article is more about unintentional errors that cause outages, rather than security incidents, but there’s a lot in common. I like the idea that “DevOps celebrates mistakes.” I hope that, when we’re back up & running, we’ll get a good postmortem on this that we can learn from.

Visual Studio extensions and tweaks

I’ve been spending some time at work recently messing around with my Visual Studio setup. I’ve been fine with my current setup for awhile, but I started getting restless recently. I guess it started when I started reading Clean Code, and watching the associated videos. That got me thinking about automated refactorings, which got me looking at JetBrains Rider and Resharper. And looking at Resharper reminded me of the existence of CodeRush.

I’ve occasionally thought about trying out something like Resharper or CodeRush, but I never got around to it. There are a number of reasons for that, mostly around the cost and the possible performance penalty. But I noticed recently that CodeRush is now available for free, so I figured I’d give it a shot. (And I think Roslyn made it easier for extensions like CodeRush to work without a big performance penalty.)

My normal VS setup, which I’ve stuck with for a while now, is pretty basic, with Mads Kristensen’s Web Essentials, and DPack Rx. I use DPack primarily for the numbered bookmarks, and Web Essentials for a number of random things. CodeRush includes numbered bookmarks, so I thought I’d try removing DPack, installing CodeRush, and seeing how that worked out. So far, I’ve found that CodeRush’s numbered bookmarks don’t work quite as well as DPack’s. There are a number of other interesting features in CodeRush, but I’m not sure if any of them are compelling enough for me to keep CodeRush installed.

I also briefly considered uninstalling Web Essentials, and then just reinstalling the specific extensions from that collection that I’m actually using. But I couldn’t quite talk myself into that, so I’ve still got the whole collection installed.

On a related subject, I recently listened to an episode of .NET Rocks with Mads Kristensen on VS 2022 extensions. And another one with Mark Miller (author of CodeRush). I’m not currently subscribed to .NET Rocks in my podcast client, but I do check in on it occasionally to see if there’s anything interesting.

Getting back to the general subject of VS extensions and setup, I also revisited my work setup and tried to decide if I could switch from VS 2019 to VS 2022. I do my VS development at work on a VM that’s running Windows Server 2012 R2. That probably seems weird, but it’s necessary for Dynamics AX development. And VS 2022 isn’t supported on Server 2012, though I could probably get it to work. So I’ve been going back and forth on whether or not I should try to install it on my current VM, or maybe ask my boss for a new Windows 10 VM to use for VS 2022. I’ve decided to stick with VS 2019 for now, but I may need to ask for a new VM at some point. In part because I have a .NET 5 project in VS 2019, and .NET 5 is no longer supported. And .NET 6 isn’t supported in VS 2019. So it’s all kind of complicated. Asking for a new VM shouldn’t be a big deal, but I’m a little nervous about it, since I just got a new boss, and I’m now in a different sub-division of the IT department, and the rules about this stuff might be a little different than they were under the old boss, so I want to feel things out a bit before I start asking for stuff.

I did install VS 2022 on my personal laptop back in May. I do like it, and would love to be able to switch over to it. (Unlike some previous versions of VS, they don’t seem to have made any really bad UI decisions that make me want to stick with the older version…)

I also spent a little time messing with the default font in VS on Friday. I’ve stuck with Consolas for quite some time now. But  I was watching a LinkedIn Learning video on Friday where the teacher’s setup was using Cascadia Code, and it looked kind of nice. I’d read about Cascadia Code when it came out, but I never got around to trying it. So I switched over to it for a little while, but then decided to switch back to Consolas. The whole code ligature thing is interesting, but Consolas just seems to work better for me.

All of this fiddling around made me think about the balance between sharpening the saw and… pointless procrastination. (It’s bothering me that I know that there’s a clever metaphor similar to “sharpening the saw” that basically means “pointless procrastination”, but I can’t remember what it is.)

Well, I’ve now killed a bunch of time on a hot Saturday morning, drinking iced coffee, waiting on my grocery delivery, and writing a rambling blog post. My plan for the rest of the day revolves mostly around watching the last two episodes of Stranger Things on Netflix. Life is good, I guess.

Lazy initialization

My experience working my way through the Clean Code book and videos has gotten me interested in refactoring some of my old code. I have a large API project that I maintain at work, and it’s gotten a little out of hand over the last few years. In particular, there’s one class that got so large that I broke it up into multiple partial class files some time ago. Overall, it would have been around 5000 lines of code if I hadn’t broken it up. Of course, partial class files don’t really change anything; it’s still one big class, technically.

So I started looking for ways to break it down into actual separate classes. What I wound up doing was to leave stuff that was needed across all the code in the old class, then moving all the specific stuff from the partial classes into new “child” classes. I created a new base class for these child classes, then added code to the original (now “parent”) class to contain all of the children as singletons, instantiated as needed.

The code to do that looked a lot like the code in this Stack Overflow question. That was fine, but once I got all that done and working, I started looking around for ways to streamline or simplify the declarations. I tried out a few things, and decided on using the Lazy<T> class. I’m sure I must have stumbled across it before, but I’d forgotten about it and had never actually used it. It took me a little while to figure out exactly how I should use it, in my particular use case. I don’t have anything super-interesting to say about that, but I thought I’d just post a few links that I found helpful:

Overall, I think the code is a little better for having been refactored this way, though I’m still not quite satisfied that I’ve cleaned things up as much as I should.

Comparing objects in .NET and general thoughts on testing

I had a little task today at work, where I needed to replace the way I created a JSON object that was being returned from an API. Long story, but the challenge at the end was to make sure I didn’t screw it up, so I wanted an easy way to compare the “before” and “after” JSON. Now, I was adding a feature here, so the before and after were going to be different, but I wanted to have an easy way to check that they were different only where I expected them to be, and there weren’t any unexpected side-effects.

This let me down a rabbit hole, looking at things like this ObjectsComparer package on GitHub. I also stumbled across this JSON extension for FluentAssertions. In the end, I just wrote a script in LINQPad to dump the before & after JSON to disk, then compared it with Beyond Compare. I keep thinking that I need to put together a better testing framework for my API, so I can better automate this kind of stuff.

I don’t have a unit test project for this API, or any other really structured tests. I have a test client that can run a suite of “safe” tests that don’t alter data, and checks a few things on the results. But it’s not a very complete test set. And I have another client that runs a test cycle that does alter data, and hence needs to be run carefully, and can’t be run too frequently. (And that one isn’t very complete or thorough either.) And I have some LINQPad tests that I run mostly as smoke tests.

I’m currently working my way through the TDD section of the Clean Code series that I’ve been watching/reading. I’ve always liked the idea of TDD, but I’ve rarely worked on a system where it seemed practical. (I’m not saying that it’s not practical, just that I haven’t figured out how to effectively apply it to any of my typical work.) Regardless of TDD, I ought to be able to put together some better and more automated tests for some of my work.

Clean Code, Visual Studio, Windows 11, and a bit more on ebooks

I wanted to add some more notes about the whole Kindle, EPUB, MOBI, etc. thing that I’ve been blogging about in my last few posts. As I mentioned a few posts back, I’ve been working my way through the Clean Code learning path in O’Reilly. Since I’m probably going to lose access to O’Reilly before I’m done with that, I thought I’d buy a copy of the Clean Code book, so I could finish reading it at my leisure, and to have it for later reference. With tech books, I usually like buying a DRM-free ebook directly from the publisher, when possible. In this case, I initially had some trouble finding that, but eventually figured out that it’s purchasable through the InformIT site. I bought the Robert C. Martin Clean Code Collection ebook, which contains both Clean Code and The Clean Coder. I had a discount code, so it was about $40 total.

I copied the EPUB, MOBI, and PDF files for the book to OneDrive. There are a number of ways I can read an ebook on my iPad, if it’s DRM-free and available in multiple formats. For this one, I wound up sending it from OneDrive into my Kindle app, as a MOBI file. That method still doesn’t support EPUBs. And it will copy the file up into my Kindle library, which is nice. I don’t think I’d ever copied anything into the Kindle library that way before, but it worked fine. I also copied the PDF into GoodReader. The code listings in the MOBI version look a little weird, which is a common problem with tech ebooks, so it’s nice to have the PDF handy as an alternative.

Clean Code has some example refactorings that are fairly interesting. They’re all in Java, but I thought it might be interesting to take the original Java code for one of the examples, convert it to C#, get it working, then work through some refactoring that’s similar to what Uncle Bob does in the book/videos. I found some of the code for the examples in GitHub under the Clean Code Kata user account. (I’m not sure if that’s an “official” account for Bob Martin or his organization, but either way, the code is there.)

This idea to convert some of the examples to C# and work through them was prompted in part by a desire to set up a .NET dev environment of some sort on one of my personal machines, and to maybe experiment a bit with some of the more recent .NET stuff, like .NET 6 and VS 2022. I realized that I don’t currently have any dev stuff at all set up on my desktop PC, my MacBook, or my Lenovo laptop. The MacBook is new, so I just haven’t set any dev stuff up yet. The Lenovo was bought in 2020, and I haven’t gotten much use out of it at all. And I’ve been trying to keep the desktop PC free of any heavyweight dev tools, since I just want to keep it clean for personal productivity stuff. After going back and forth on a few possible setups, I decided to install Visual Studio 2022 (Community edition) on my Lenovo laptop. I considered just installing the .NET 6 SDK and Visual Studio Code, which would have been much more lightweight, but I’m used to using the full VS product, and I can’t see a reason not to use it. And the Lenovo is the best place to install it, since I can wipe out that machine entirely and start fresh if things get too messed up. I’ve also recently upgraded that laptop to Windows 11, so this was also an opportunity to (finally) give that a try.

Of course, I’ve had other things to do this weekend too, so I’ve only gotten as far as installing Visual Studio and git, and tweaking some settings. Maybe I’ll actually do some programming next weekend. (Or maybe I’ll get distracted by something else, and the whole thing will fall by the wayside.)

Scrum and tech learning follow-up

This post is a follow-up to my previous post on scrum and tech learning. Since then, I’ve finished reading The Elements of Scrum, via my (work) Percipio account. I’ve also downloaded the audiobook version of Scrum: The Art of Doing Twice the Work in Half the Time, from my (ACM) Skillsoft account. I normally wouldn’t think about consuming tech books in audio format, and most of them wouldn’t work that way. But maybe a scrum book would be comprehensible as audio, so I’ll give it a try.

There’s another scrum book I was considering, Scrum: A Pocket Guide, which I’ve found is also available in audio format, via the author’s YouTube channel. So, in this case, you can actually watch the author sitting at his desk, reading his own book. I don’t think I actually want to do that, in this case, but it was nice of him to do that and put it out there for free.

Meanwhile, I logged in to my ACM O’Reilly account today, to see if there was anything I really wanted to watch or read before the account gets shut down. I decided to take a look at Bob Martin’s Clean Code video series. I watched the first hour-long video. There’s about fifty hours of video in the series, all told. In O’Reilly, it’s organized as a “Learning Path” with the videos interspersed with chapters from his Clean Code book. I’ve been meaning to read that book for a while. It was published in 2008, so it’s probably a bit out of date at this point, at least in terms of some of the specifics. For the videos, you can also find them for sale at the Clean Coders web site. (They’re not cheap.) You can find a few on YouTube. Here’s a link to the first one, which is the one I watched today. (I think the one I watched on O’Reilly is actually a revised version of that.) He definitely tries to keep it interesting, with a bunch of different costumes and backdrops, including several Star Trek ones. It comes off as pretty corny, but I guess it’s better than just watching him read through the material at his desk.

With the changes we’re going through at work, I’ve decided that now is a good time to back up a bit and think about what new stuff I need to learn, or old stuff I need to reinforce and/or brush up on. I’ve probably read enough on scrum by now, though I may branch out and read up on some related topics. And the Clean Code stuff is the kind of thing I like to check out occasionally, to remind myself of some of the fundamentals of good coding, and maybe learn a few new things that I hadn’t stumbled across before.

I’ve also been listening to some of the recent podcast episodes from .NET Rocks around the twentieth anniversary of .NET, which happened back in February. (I stopped subscribing to .NET Rocks a while back, so I don’t listen to it every week, but I go in and cherry pick interesting or relevant episodes once in a while.) Listening to folks like Anders Hejlsberg, Scott Guthrie, and Miguel de Icaza reminisce about .NET was fun. And it got me thinking about what new stuff is going on with .NET that I should learn. Maybe Blazor? Or I should figure out what’s new and interesting with .NET 6? Or I should try to get back into F# again? I don’t know. Maybe I should pick up this Apress Microsoft book bundle from Humble. That would keep me out of trouble for a while, right?

Scrum at work, and tech learning subscriptions

My team at work is going through some changes right now. We’re getting shuffled to a different spot in the IT department hierarchy and getting a new boss. And we’re supposed to start doing scrum.

Actually, we were supposed to start doing scrum almost a year ago, and we kinda started doing it, but we didn’t really go all the way with it. So now, I guess, we’re supposed to go all-in. Or almost all-in. Or something like that.

Anyway, I watched some training videos for scrum last year when we were supposed to start using it, so I already have some understanding of it. But now that we’re going to be going further with it, I decided to do some more reading on it and try to learn more about it.

I wanted to read a book on scrum, and since we’re using Azure DevOps, I decided to try this one: Professional Scrum Development with Azure DevOps, from Microsoft Press. I started reading it in March, and finished it a couple of weeks ago. It wasn’t a bad book, and it does cover the Azure DevOps scrum process pretty well, but it was also pretty dry, and I’m not sure that a lot of it is really going to be relevant to me.

I’ve also considered reading Zombie Scrum Survival Guide. It might be a little cynical to assume that our implementation of scrum is going to fit the definition of “zombie scrum,” but I suspect it might, and I’m not sure what the best way would be to engage with that.

And A Scrum Book: The Spirit of the Game might also be a good general scrum book to read. Honestly, it looks a bit daunting though.

On the related general subject of tech books, I’ve recently learned that ACM is going to lose access to O’Reilly Learning (previously known as Safari) in July. That’s quite a disappointment, but not entirely surprising. ACM got access to the full Safari library in 2017. That always seemed a bit weird to me, since an ACM membership costs about $100/year, and a regular individual O’Reilly membership costs $500/year, raising the question of why anybody would buy one through O’Reilly rather than just signing up for ACM. I guess that logic finally caught up with them, so now they’re cutting it off.

O’Reilly has offered ACM members a $100 discount, so that would bring it down to $400/year, but that’s still a lot of money, and the discount is only good for the first year. I generally only read a few tech books each year, and I can generally buy them for $40 or $50 each, so I guess it’s not going to be worth it to sign up for a paid O’Reilly account.

ACM still has SkillSoft, and I think I still have access to Percipio through work, but neither of those has the broad selection of tech books and videos that O’Reilly has. (And I think SkillSoft and Percipio are actually the same thing, under different names, possibly with slightly different libraries?) I also have access to LinkedIn Learning, both through work, and (last time I checked) through the local public library. LinkedIn Learning really just has videos though, no books. And their video courses generally aren’t as in-depth as the stuff I could get through O’Reilly. I discontinued my Pluralsight subscription this year, so I don’t have that anymore, either.

For now, I guess I’ll take a look at Skillsoft/Percipio when I’m looking to read up on a new tech subject, and see what they’ve got. When I really want a specific book that they don’t have, I’ll probably just buy it from Amazon or directly from the publisher. I’ve been trying not to buy physical tech books anymore, and O’Reilly has certainly helped with that. A lot of the tech publishers still offer direct sales of DRM-free ebooks, so I’ll probably go that route when I can, rather than getting the DRM’d versions from Amazon.

Post-Thanksgiving stuff

I survived Thanksgiving, but it wasn’t a great day. The fire alarm in my apartment building went off on Thanksgiving Eve, at 11 PM. So I had to get out of bed and go stand around outside in the cold for about a half-hour, before being allowed back in. I’d gone to bed at 10, and was pretty well asleep at 11 when the alarm went off. And afterwards, I just couldn’t get back to sleep. So I didn’t have a lot of energy on Thanksgiving. About all I did was re-watch some episodes of Doctor Who and nap. I guess that’s a reasonable Thanksgiving, really.

I tried to get back to something closer to normal yesterday, Black Friday, but that was also a pretty low-energy day. I didn’t read anything on Thanksgiving, and read only one comic on Friday. I’m noticing that my resilience just isn’t what it used to be, physically (and sometimes, mentally). One bad day or night can screw me up for a few days afterward. I’m not sure how much of that is just normal for my age, vs. being something I should worry about.

Anyway, what I wanted to write about was really just some Black Friday stuff. I don’t buy a lot of physical stuff on Black Friday, typically, but I keep my eye on some digital deals on software, subscription services, and stuff like that.

  • Last year, around this time, I signed up for a free one-year subscription to Calm. The deal was for one free year, and a second year at half-price. I’ve been using it pretty consistently this year, so I was ready to let it renew for the second year. But Calm runs a Black Friday deal every year, where you can get a lifetime sub at 60% off. So I went ahead and took advantage of that, and paid $160 for a lifetime sub. That’s a fair amount of money, but I’ve stuck with my meditation habit pretty consistently this year, and I think I’ll keep it up for the foreseeable future. That was my big Black Friday purchase.
  • Another thing I did last year was to sign up for Hulu’s Black Friday deal, which was $2/month for their ad-supported tier, for a year. So that was coming to an end. For that, I decided to pause the subscription for a few months are reevaluate it later. There’s some good stuff on Hulu, but I feel like I’ve got too many streaming services going right now, and too much stuff to watch.
  • I also subscribed to Letterboxd Pro last year, on Black Friday, for $12/year. I’ve been using Letterboxd a lot this year, so I let that renew, and it looks like I’ll continue to get the discounted $12/year rate.
  • It’s also about time for me to do my yearly review as to whether or not Pluralsight is worth renewing. I’m currently on a “legacy” plan, which should renew in January at $179. If I cancel my account, then I won’t be able to get that old rate back. Though it seems like their current Black Friday deal would let me subscribe to their “standard” plan for $179, so maybe there’s a little wiggle room there. I haven’t actually used Pluralsight that much this year, so maybe it’s time to give up on that. I’ll have to decide on that before the end of the year.
  • Meanwhile, my Amazon Prime subscription renews on December 1, for the usual $119. I’m always a little unhappy about supporting Amazon to the extent that I do, but honestly, it’d be kind of hard to live without Amazon at this point, and dropping Amazon Prime would not affect Amazon’s fortunes in the slightest. So I’ll just let that one renew too.
  • I generally think about various hardware upgrades around the end of the year. I don’t really have anything pressing this year though. I looked at the Kindle deals at Amazon. I’m happy enough with my current Kindle, but the new Paperwhite is supposed to be really good. But I just don’t need it. I might want a new iPhone next year, but, again, I don’t really feel like I need one just yet, and there aren’t any really good deals on iPhones. So probably no new hardware this year.

So that’s about it. Nothing much exciting, but it kept me out of trouble for an hour. I need to try to get back in the swing of things today and tomorrow, so I can go back to work Monday and have a good productive day. I know there’s going to be a lot of work waiting for me on the first day back from vacation.

moving from ADAL to MSAL

I haven’t written a programming-related post in a while. I just had to rewrite some code that used ADAL to use MSAL, so I thought I’d write up a short post on that.

There’s a bunch of documentation around this on the Microsoft web site, but for the simple case I was interested in, it took some effort to track down.

Here are links to a couple of general articles:

What I needed was to rewrite a small block of code that calls a web API from a console app, with no user intervention. I have an Azure app registration set up to help with that, with a client ID and secret, and all that stuff. I have some links on how to set that up somewhere, but I’ll skip that for now.

The actual code I needed was something like the code here (to initialize MSAL) and here (to get a token). After I get the token, I just add it as a bearer token to the header for the request.

Here’s a bit of “before” and “after” code:

// before:
using Microsoft.IdentityModel.Clients.ActiveDirectory;

// get the parameters from a config file, or somewhere...
string clientId = ConfigurationManager.AppSettings["ClientId"];
string clientSecret = ConfigurationManager.AppSettings["ClientSecret"];
string authority = ConfigurationManager.AppSettings["Authority"];
string svcResourceId = ConfigurationManager.AppSettings["ServiceResourceId"];

AuthenticationContext authContext = null;
ClientCredential clientCredential = null;

authContext = new AuthenticationContext(authority);
clientCredential = new ClientCredential(clientId, clientSecret);
AuthenticationResult result = null;
try
{
	result = await authContext.AcquireTokenAsync(svcResourceId, clientCredential);
}
catch (AdalException ex)
{
	Console.WriteLine(String.Format(
		"An error occurred while acquiring a token\nTime: {0}\nError: {1}\n",
		DateTime.Now.ToString(), ex.ToString()));
	return;
}
//Console.WriteLine("Access Token: {0}", result.AccessToken);

client = new HttpClient();
client.BaseAddress = new Uri(BaseAddr);
client.DefaultRequestHeaders.Accept.Clear();
client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));
// Add the access token to the authorization header of the request.
client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", result.AccessToken);

// after: 
using Microsoft.Identity.Client;

IConfidentialClientApplication app = ConfidentialClientApplicationBuilder.Create(clientId)
	.WithClientSecret(clientSecret)
	.WithAuthority(authority)
	.Build();

AuthenticationResult authResult = null;
try
{
	List<String> scopes = new List<String>() { svcResourceId + "/.default" };
	authResult = await app.AcquireTokenForClient(scopes).ExecuteAsync();
	//string accessToken = authResult.AccessToken;
	//Console.WriteLine($"Access Token: {accessToken}");
}
catch (Exception ex)
{
	Console.WriteLine($"MSAL Error: {ex.Message}");
}

I’m not sure if anyone other than me will ever find this useful, but here it is, just in case.

Ephemeral Port Exhaustion

We’ve been having some trouble with our main web server at work over the last few months. It all boils down to ephemeral port exhaustion, which sounds kind of like a post-COVID side-effect, but is actually something that can happen to a Windows server if you’re opening too many ports and then not releasing them. The post linked above contains some useful troubleshooting information regarding this problem.

I actually think the best explanation of this issue is in a 2008 TechNet article titled Port Exhaustion and You. (That link goes to the original version of the article via archive.org. Here’s a link to it’s current location at Microsoft’s site.)

The basic issue is that you can run out of ports and then anything that relies on opening a new one fails, and you just need to reboot the server. So, not the end of the world, but not good for a production server. We’ve been working around it for awhile. We had it scheduled to reboot once a week, but upped that to twice a week when it seemed like once wasn’t enough. And now it’s gotten to the point where I really think we need to find the underlying issue and correct it.

In our case, the server is running a bunch of web services under IIS. There are more than a dozen separate services, written by various programmers, at various points in time. They’re all (probably) C# programs, but they’re written under various versions of .NET Framework and .NET Core. They’re grouped into three or four app pools.

The first thing that makes sense to look at here is how the individual programs are handling outgoing network connections. Normally, in C#, you’d use HttpClient for that. I wrote a blog post in 2018 about HttpClient and included a link to this article about how to properly use HttpClient without opening a bunch of unnecessary connections. I think I’ve got all of my own code using HttpClient correctly and efficiently, though I’m not sure about everyone else’s.

It can be hard to tell what’s going on behind the scenes, though, if you need to rely on closed-source third-party libraries that also open up HTTP connections. I’ve got a few of those, and I think they’re not causing problems, but I don’t really know.

To try to monitor and track down port exhaustion issues, there are a few tools you can use. A number of the articles I’ve linked above mention “netstat -anob” or some variation of that, and I’ve found that helpful. One issue with that, if you’re running a lot of web services, is that you can’t easily see which service is causing a problem.

My big breakthrough yesterday was realizing that I could use “appcmd list wp” to get a list of the PIDs and app pool names associated with the various IIS worker processes. From that, you can tie the netstat output back to a specific app pool at least. (Of course, if you have ten web services under one app pool, then you’ve still got some more work to do.) See here for some info on appcmd.

Anyway, we still haven’t quite got our problem solved, but we’re getting closer. For now, we’ll still just need to keep an eye on it and use the old IT Crowd solution: “Have you tried turning it off and on again?”