Category: programming

back home and (slightly) broken

I’m back home today, after spending a few days in Redmond, WA for a two-day workshop on the Microsoft Partner Center SDK. This particular API/SDK is esoteric enough that it’s not worth blogging about much, though it’s been taking up a lot of my time over the last year or so. And the actual workshop contents are under NDA anyway. For what it’s worth, it was a good workshop and I learned some new stuff. I also got clarification that something I’ve been trying to do for the last month or two, and completely failing at, is indeed currently impossible. So that, on its own, made the trip worthwhile. It turns out I’m not an idiot who can’t program his way out of a paper bag. (Rather, I’m an idiot who couldn’t realize that he’s inside a concrete bunker and not a paper bag. Maybe I’m stretching that metaphor a little too far…)

I wanted to mention the trip partly to give me an opportunity to mention that I completely missed Free Comic Book Day, since I spent nearly the entire day yesterday traveling. The Beat has a lot of coverage of FCBD; there were some some interesting books available. Maybe I could go over to my local store today, and see if they’ve got anything good left.

I also wanted to mention that I’m not missing the pre-sale for NYCC 2019, since that’s happening today at 10 AM. I had a good time at last year’s con, so I’d like to go again. A four-day pass is almost $200, so it’s not cheap, but heck, if I don’t spend my money on comic conventions, what am I going to spend it on? (Food? Rent? Nah.)

And I also wanted to check in on the subject of how broken I am after traveling to the west coast and back. I started thinking about this stuff after last year’s workshop, and tweaked some stuff in my routine when I went to WonderCon last month. I think I probably need to tweak some more stuff for the next time I have a long trip, but maybe I’m on the right path. One thing I learned after last year’s workshop is that, if I’m traveling to Redmond in the spring, I need to bring my allergy medicine. So I did that this year. I’ve also figured out that my body doesn’t adjust to time zone changes as easily as it used to. So I’m taking melatonin gummies when I travel now. (That helps a bit, but not as much as I’d like.) And I also figured out, after my WonderCon trip, that it was really time for me to give up on the L.L. Bean duffel bag that I’ve been using for luggage the past few years and get one of those ubiquitous carry-on bags with wheels and a telescoping handle. (I definitely pulled/strained/broke something from carrying that duffel around, coming home from WonderCon.) I think I’ve also broken my long aversion to taking a bag on the airplane with me, rather than checking it. Bag check now costs $30 each way, and it seems like everyone else brings a bag on the plane, so I guess I can too. And if that bag has wheels and a handle on it, it’s less of a pain to carry it through the terminal (and on the monorail, etc).

I have a bit of a residual headache this morning, and I didn’t sleep well last night, so the answer to “how broken am I?” is: Not as broken as I could be, but still more broken that I’d like.

I have enough stuff to do today that I should probably stop blogging and start doing stuff. In addition to the NYCC pre-sale, I also need to do grocery shopping, pay some bills, and scan in the receipts from my trip. So that’s it for now.

Azure and baseball and comics

As mentioned in yesterday’s post, I did manage to watch a few of the Global Azure Bootcamp videos, yesterday and this morning. I didn’t really find any videos that directly applied to the projects that I’m currently working on, but I did pick up some good pointers and some useful background information. It was mentioned on Twitter that the videos are only staying up until Monday, so I guess that if I want to watch any more of them, I should do that today.

I also managed to get out and see a bit of the Somerset Patriots season-opening double-header yesterday too. I arrived about halfway through the first game, and went home just before the second game started, though. I intended on staying through at least the first few innings of the second game, but it was getting too cold. (The final score in the second game was 14-2, Patriots, so that would have been fun to watch.)

I did not get out to see Avengers: Endgame yesterday, and it looks like I’m not even going to try today. I checked a 9am showing this morning, and it wasn’t sold out, but there was only one seat available, and it wasn’t a good one. I assume the later showings are going to be sold out. I’m not sure I can sit through a three-hour superhero movie anyway. (I like Warren Ellis’ reference to the movie as “AVENGERS: SATANTANGO or whatever this bladder test is called.” I don’t think I could sit through the actual Sátántangó either.) This may be the kind of thing where I need to wait for it to come out on Blu-ray, so I can use a pause button as needed.

I did manage to finish up a Batman graphic novel this morning, and I may start on another after lunch, so I am getting some comic book reading done this weekend too.

Meanwhile, I should probably also be doing some prep work for my trip to Redmond at the end of the week for the Partner Center workshop. I think I have everything up-to-date on my laptop, and my laundry is done, so there’s not really much more to do, though.

Global Azure Bootcamp and Pragmatic Programming

I’ve been doing a bunch of work related to Azure recently. It’s mostly not around actually using Azure, but rather managing Azure and billing for Azure. I’m in the middle of something right now that’s honestly driving me to distraction and making me want to take a month or two off and maybe traipse around Europe or something. Anyway, today is Global Azure Bootcamp. There’s an event here in NJ, at Microsoft’s office in Iselin, but I was too late to register for it, and it’s full up now.

There’s also a lot of online stuff going on, though. It should all get posted to this YouTube channel. I can see a bunch of stuff up there already, and it’s only 8am Eastern time. (The Auckland event is already over. I guess because it’s midnight there right now, so today is already over. Funny how that works…)

Anyway, I really want to watch a bunch of this stuff, but it’s Saturday, and the weather should be pretty nice, and yesterday’s rained out Somerset Patriots game has been rescheduled to today, and I’ve got finish my laundry, and do my grocery shopping, and so on and so forth.

Looking at what’s already on YouTube, I’m kind of interested in two of the videos from the Perth/Beijing cycle:

  1. Understanding The New Azure Role-Based Certifications – I probably don’t have the spare time to study for and pass any Azure certification exams, but a guy can dream, right?
  2. Mission: Azure Kubernetes Service – Because some other folks I’m working with have been talking about Kubernetes, and I know almost nothing about it.

I’m going to the Microsoft offices in Redmond next week for a workshop related to the specific project I’m working on, so that should be useful. But sometimes I feel like I’m really falling behind with all this Azure and AWS stuff. I’ve been reading The Pragmatic Programmer: From Journeyman to Master in my spare time recently. It’s a classic, but it’s 20 years old, so there are a lot of dated references in it. It’s actually been kind of comforting to read it. I guess I’m more at home with references to 56k modems than references to Kubernetes clusters. There’s actually a 20th anniversary version of the book coming out soon, so maybe I should give up on the old version and wait for the new one.

async and await in C#

I haven’t written many programming-related posts lately. A few months ago, I was doing a bunch of research into stuff related to async and await in C#, and made some notes that I intended to turn into a blog post. Three months later, they’re all still in my Evernote “inbox” notebook. Well, maybe it’s time to finally get around to that post. Of course, now, I barely remember what I was doing back then, so this post is mostly going to be a bunch of links to resources. Maybe it’ll come in handy the next time I need to solve an async/await problem.

When I was trying to figure this stuff out, I found myself reading a lot of stuff by Stephen Cleary. His blog has a lot of useful posts about async programming. His async OOP series is interesting. Those posts led me to look into his Concurrency in C# Cookbook. His MSDN article from 2015 on Brownfield Async Development was relevant to my project too.

Now I’m starting to remember what I was going to write about… It was going to be a post about the challenges of retrofitting async calls into a Web API project that didn’t initially use the async/await patterns. I had to do this due to some changes in another API that I was calling. Those changes aren’t worth getting into here, but I found that async tends to become an “all or nothing” proposition. I was initially running up against some blocking problems, which led me to Stack Overflow, which then led me to Stephen Cleary’s blog post titled Don’t Block on Async Code.

Later, I started hitting some problems that required me to put some effort into limiting concurrency on certain calls, which led me to this MSDN post and this post from Mark Heath. I wound up doing something with SemaphoreSlim. (At least that’s what I think I did…)

Anyway, my project is working fine now, in production, and everyone seems reasonably happy with it, so I guess I got all this stuff right in the end.

FizzBuzz

We’re hiring a new developer in my group at work, and my boss is including me in the interviewing process. It’s been a few years since I’ve done developer interviews, so I’m a bit rusty. I suggested having candidates do a FizzBuzz test on a whiteboard as part of the interview.

Jeff Atwood wrote a good post about FizzBuzz on his blog back in 2007. It seems like an overly simple test, but it can be quite useful. I’ve only been asked to do FizzBuzz once myself, and it was a good experience. The interviewer was really sharp and asked me a lot of good questions about how I could do it differently or why I chose to do something a certain way. He turned a simple 12-line program into a good conversation.

At very least, FizzBuzz should help filter out candidates who are exaggerating on their resumes. If you say you’ve got five years of C# experience and you can’t write a FizzBuzz program, you’re lying. The two candidates we’ve looked at so far both have an MS in Comp Sci, so they’re both better-educated than I am, at least, and they should both be able to handle FizzBuzz.

Anyway, it occurred to me that I never wrote a FizzBuzz program in X++. So here’s a short job to solve FizzBuzz in X++. I might post it to RosettaCode, if I get around to it. Not that the world really needs one more FizzBuzz solution.

static void AjhFizzBuzz(Args _args)
{
    /* Write a program that prints the numbers from 1 to 100. 
    If it’s a multiple of 3, it should print “Fizz”. 
    If it’s a multiple of 5, it should print “Buzz”. 
    If it’s a multiple of 3 and 5, it should print “Fizz Buzz”. 
    */
    int i;
    
    for (i = 1; i <= 100; i++)
    {
        if (i mod 3 == 0 && i mod 5 == 0)
            info("Fizz Buzz");
        else if (i mod 3 == 0)
            info("Fizz");
        else if (i mod 5 == 0)
            info("Buzz");
        else
            info(int2str(i));
    }
}

Adding an exception logger to a Web API project with Autofac and Serilog

I just spent way too much time figuring out how to add a catch-all logger for exceptions to an ASP.NET Web API project, so I figured I’d write up my experience as a blog post, for anyone else who needs it (and for my own future reference).

The goal, specifically, is to log any unhandled exceptions using Serilog. I don’t want to mess with them in any way, I just want to record them in the log. (For this API, most exceptions are already properly handled, but sometimes something falls through the cracks, so I just want to be able to see when that happens, so I can fix it.)

First, this is an old-fashioned ASP.NET Web API project, not a .NET Core project. I’m using Autofac for dependency injection and Serilog for logging.

And I’m using the Autofac.WebAPI2 package to integrate Autofac into the API. My Autofac configuration looks pretty much just like the example in the “Quick Start” section of the page linked above.

Serilog is linked in like this:

builder.Register((c, p) =>
{
    var fileSpec = AppDomain.CurrentDomain.GetData("DataDirectory").ToString() + "\\log\\log-{Date}.log";
    var outpTemplate = "{Timestamp:yyyy-MM-dd HH:mm:ss.fff zzz} [{Level:u3}] {Properties:j} {Message:lj}{NewLine}{Exception}";
    return new LoggerConfiguration()
        .WriteTo.RollingFile(fileSpec, outputTemplate: outpTemplate)
        .ReadFrom.AppSettings()
        .CreateLogger();
}).SingleInstance();

I won’t get into how that works, but you could figure it out from the Serilog docs easily enough.

ASP.NET Web API provides a way to hook into unhandled exceptions using an ExceptionLogger class. This is described a bit here. I found several blog posts describing various permutations on this functionality, but I had to mess around a bit to get it all to work right for me.
I created a class that looks like this:

public class MyExcLogger : ExceptionLogger
{
    public override void Log(ExceptionLoggerContext context)
    {
        var config = GlobalConfiguration.Configuration;
        var logger = (ILogger)config.DependencyResolver.GetService(typeof(ILogger));
        if (logger != null)
            logger.Error("Unhandled exception: {exc}", context.Exception);
    }
}

and I hooked it up to Web API by adding this line to my WebApiConfig Register() method:

config.Services.Add(typeof(IExceptionLogger), new MyExcLogger());

There’s not actually much to it, but I went down the wrong path on this thing several times, trying to get it to work. The (slightly) tricky part was getting the logger instance from the dependency resolver. Constructor injection doesn’t work here, so I had to pull it out of the resolver manually, which I’d never actually tried before.

sorting photos with Python

As part of my MacBook replacement project, I wanted to get a good new backup of my photo library. Photos.app stores photos in a database, not individual files, so if you want a “plain old JPEG” backup of your library, you need to export the photos from Photos.app. There are various ways to do this, but the way I chose left me with a folder full of subfolders organized by “moment name,” so a typical subfolder would be named “Somerville, NJ, May 27, 2018” for instance. That’s kind of a mess, since the folder names sort alphabetically. (And the folder dates are all set to today. Setting the folder dates to the date the photos were taken would solve some problems, but let’s ignore that for now.)

To impose some rudimentary organization on this, I wanted to make a top-level folder for each year, and move all the subfolders for that year into that folder. Not perfect, but at least it gets me from having one folder with 2000 subfolders to 10 folders with 200 subfolders each (approximately), which is a bit more manageable.

I looked around at various ways to do this, and settled on writing a quick Python script, which is shown below. I haven’t written a Python program in years, so this probably isn’t elegant Python. But it works. I’m posting it here in case it’s useful for anyone else. (I first learned Python back in 2003, and have used it on and off since, but never regularly.)

I had also looked for third-party utility programs that would allow me to export from Photos with more options than Photos itself allows. I found an old MacStories article from 2012 that talked about exporting from iPhoto to Dropbox using a tool called Phoshare. I had hoped that maybe Phoshare had been kept up to date, and worked with Photos, but nope. It was last updated in 2012.

I feel like there’s probably some interesting stuff I could do with Hazel too, and/or ExifTool, but I didn’t want to over-complicate things. And I also feel like there’s probably a third-party app out there that would make this much easier, but I didn’t find one that did exactly what I was looking for.

# sort_photo_bu.py
# ajh 2018-05-27
# sort Photos app export into folders by year.
# Export from Photos with subfolder format "moment name". 
# We assume the moment name always ends with the year.
# (Note that this may throw a "destination path already exists" error at some point, but that's probably fine.)

import os, shutil, sys

bupath = "/Users/andrew/Pictures/backup20180527"
for (root, dirs, files) in os.walk(bupath):
    for dirname in dirs:
        if len(dirname) &gt; 4:
            year = dirname[-4:]
            yearpath = bupath + "/" + year
            if not os.path.isdir(yearpath):
                print "Making folder for " + year + "..."
                os.mkdir(yearpath)
            origpath = bupath + "/" + dirname
            #print "moving %s to %s..." % (origpath, yearpath)
            shutil.move(origpath, yearpath)
            #sys.exit()
print "All done."

Learning Statistics

I’m starting to read a book on statistics, and this is from the beginning of the chapter on probability:

If you are reading this book as part of a course in statistics, then you are likely pursuing a degree in higher education. On the other hand, if you are reading this book simply out of enjoyment, then you are crazy. That is one thing we can be certain about.

I guess I’m crazy. Oh well. It’s a pretty good book so far, though I think I’m going to need more to really get going.

I started thinking about learning more about statistics and data analysis recently. It ties in a bit with my attempt to learn how to use Power BI. I have all the basics of Power BI down now, meaning that I can import data and make fancy-looking pie charts and bar graphs, and now I’m poking at the edges of more meaningful data analysis.

It’s kind of hard to figure out where to start with statistics and data analysis. I never took a course in statistics when I was in college, so I don’t really know much to begin with. I did read Larry Gonick’s Cartoon Guide to Statistics years ago, back in the 90s I think. I don’t really remember much about it; maybe I should reread it now.

The book I’m currently reading is meant as a college textbook (per the quote above), and isn’t really meant to stand alone. It purposely doesn’t talk about software tools at all; just the background concepts and a little math. I’m thinking about reading this book along with it, which includes some more practical stuff, using Excel as the tool of choice. (Both books are on Safari, so I can read them for free.)

Of course, as a programmer, I’d eventually like to get to a book that talks about statistics and uses a real programming language for the examples. So maybe Think Stats would be good; it uses Python, which I’ve used before (though I’m probably a bit rusty).

I see a lot of references to R when looking into data analysis and statistics. I know almost nothing about R, so maybe I should look at something like this book.

This is all sort of leading me into data science, which is apparently the sexiest job of the 21st century, according to Harvard Business Review. I’m not really looking for a new job, and definitely not a “sexy” one, but hey, it can’t hurt to learn a bit.

Learning Azure

I have a project going at work that might (or might not) eventually have something to do with Azure. So I’m using this as an excuse to finally learn a bit about it. I looked at a few ebooks and videos that weren’t that helpful, then I stumbled upon a free ebook from Microsoft titled Fundamentals of Azure. It was last revised in 2016, so it’s a little out of date, but still mostly relevant. It does a good job of walking you through all (well, most) of the services offered on Azure, without getting too deep. (But it does get deep enough to actually be useful to a working admin or programmer.) I wrote a short review of it on Goodreads.

I’m using the $50 monthly Azure credit that comes with my Visual Studio subscription, which is more than enough to play around with (assuming that you remember to delete everything you do after you’re done). I’m glad that my company (finally) decided to pay for VS subscriptions for us. There are a few nice benefits that come with that.

If I really needed to get serious about Azure, I’d look at the Microsoft Professional Program in Cloud Admin at EdX or something like that. (Of course, I’m more of a developer than an admin, myself, so that wouldn’t be right for me either way.)

I’d love to be able to do a big project that uses a mix of Azure services, but that’s probably not in the cards. Still, it’s good to be familiar with all this fancy cloud stuff!

How to use a Stopwatch in C# incorrectly

Here’s something I did wrong this week. And, after realizing I’d done it wrong, I remembered that I’d made the same exact mistake in the past. So I’m going to write up a blog post in the hopes that maybe it’ll help me hammer the right way into my head, so I don’t screw up next time.

If you want to do a little quick and dirty performance logging in an application, you can use the Stopwatch class. Just create a new Stopwatch, then stop it when you’re done and check the elapsed time. Easy, right?

Well, here’s what I did wrong: The Elapsed property is a TimeSpan structure, which has a lot of nice properties on it, like Days, Hours, Milliseconds, and so on. For my purposes, I wanted to know how many milliseconds had elapsed. So I showed sw.Elapsed.Milliseconds. Looks good, right? Except that the Milliseconds property is not the whole span in milliseconds. It’s the milliseconds part of the span. If you want total milliseconds, then you need to get sw.Elapsed.TotalMilliseconds.

This is one of those pernicious little bugs that doesn’t cause anything to fail, but instead causes the programmer to make bad decisions. “Hey, it looks like all my web service calls execute in less than a second! Great! I can go home early!” Nope. Turns out some of them were taking more like 20 seconds.

Here’s a little code snippet showing the wrong way and the right way.

Stopwatch sw = Stopwatch.StartNew();
TimeSpan elapsed;
Thread.Sleep(1500);
sw.Stop();
elapsed = sw.Elapsed;
Console.WriteLine("Wrong: {0}", elapsed.Milliseconds);
Console.WriteLine("Right: {0}", elapsed.TotalMilliseconds);

Simple enough, but I’ve done this wrong at least twice so far in my life. Here’s hoping I can remember to do it right next time!