CoffeeScript

I wanted to take a break from my usual .Net and Drupal stuff for a weekend, and try to learn something new. So I started reading Jump Start CoffeeScript today. CoffeeScript is a language, somewhat similar to Python in terms of syntax, that compiles into JavaScript. In theory, it makes JavaScript easier to use, and easier to read. I’m not entirely sold on it yet, but it’s interesting.

The book I’m reading is supposed to be a quick start, that you can get through in a weekend. It walks you through a sample application, an HTML5 game. I’m finding the sample code less useful than I’d like. It’s mostly drawing on the HTML5 canvas. I think I’d rather see code doing more typical stuff that you’d see being done on a typical web site. It’s hard to relate the canvas stuff to anything I’ve actually done before.

That said, it’s a pretty well-written book, and kind of fun. I had hoped to finish it tomorrow, but it looks like I need to take a trip into NYC tomorrow, and I’ll have company, so I won’t be reading on the train. And I’m working on New Year’s Eve, but maybe I can finish it on New Year’s Day!

Amazon Web Services

My boss has gotten enthusiastic about AWS recently. We’ve messed around with a bit with in on a couple of projects, and also just for testing. We used a Linux EC2 instance for the Gisgraphy project I blogged about a while back. And we would up using a dedicated Windows EC2 instance for the gas finder site that we put up after Hurricane Sandy.

In addition to that, I’ve done some experimenting with RDS (for SQL Server) and Elastic Beanstalk. I was initially enthusiastic about both, but, once I got to working with them, I saw enough limitations that I decided to back off. RDS is a nice way to have a low-maintenance SQL Server in the cloud, but the main limitation I found was the inability to easily get a local backup of a large database loaded onto it, and vice versa. While there are ways to get data in and out of RDS, of course, there’s no support for standard SQL backups and restores, to (and from) .bak files. The recommended approach is to script the db out to T-SQL, using the standard export capability, but given the size and complexity of some of our databases, that’s just not workable.

And, as for Elastic Beanstalk, it’s a nice way to get a single ASP.NET application up and running on EC2, with a pre-configured load balancer, but we really don’t have any single site that would benefit from that, and easily fit into that model.

For now, we’ve tentatively decided to move some of our stuff up to AWS, using a few EC2 instances, but really that won’t be much different than what we’re doing now (external hosting on a managed server at PEER1). The main difference will be that we’re planning on breaking some stuff up so that we’ll have a few smaller instances, so we can scale them as needed, and we’ll have the option of spinning up a second copy of the instance running our web service, if needed. I’m not really sure how all this will work out, but I’ll blog more as we go forward, if I learn anything that seems like it would be worth sharing.

Meanwhile, I’ve been watching this screencast series on AWS. Unfortunately, I’ve found that it’s mostly covering ground that I’ve already figured out on my own. But, if you’re new to AWS, and want a guided tour, included a (somewhat leisurely) look at some cloud basics, this might be for you.

Selenium

OK, one more post for tonight. (This is another one I suspected that I may have previously written up, but apparently not.)

I’ve known about Selenium for awhile now, mostly because one of our clients has a “testing guy” and he uses it. I’ve always wanted to be able to do some automated testing of web site projects, but it always seemed like the tools for doing so were too limited or complex. I’ll admit I put off downloading & learning Selenium, largely because I thought it would be a hassle and eat up a lot of time before I could really do anything useful with it. When I finally gave it a chance, though, I was surprised how easy it was to use.

I initially started with WebDriver, which is basically a couple of DLLs that let you “drive” Firefox (or another browser), sending keystrokes and click events, and looking for certain responses. You can get started with WebDriver quickly by grabbing it via NuGet. My first project with WebDriver was a simple console program that launches Firefox, then goes to several of the store locator web sites that use our Bullseye API, does a search at each one, and checks to see if it gets results. Nothing big, but just a useful program that I can run any time I roll out code changes to the API. Previously, I’d been checking this stuff by hand after each rollout.

Today, I took another step, and downloaded Selenium IDE. This is a Firefox plugin that lets you record a series of actions as you do them, then save them to a script. There are plugins allowing you save the script in several languages, including C#. So, I can record some steps, export some C# code, then fix it up to do some reasonable testing. My main purpose today was to record the steps involved in a fairly complex workflow on one of our client sites. It’s a multi-step process (around 20 steps, I think). Just in and of itself, the script is useful to have, as I often need to step through it to establish a new test account, so now I can just “play” it instead of clicking through the whole process myself. But, I would also like to use it to automate some testing of this process. Now that I have a base script, I can go in and replace the values I entered today with variables, so I can abstract things out in such a way that I can run the code repeatedly, testing multiple scenarios. And since I can do this all in C#, I can also then check the database, and see if the values I entered were interpreted and stored in the database correctly.

This may all seem pretty routine to some people, but I have to admit that I’ve never really had a chance to do this kind of testing before. It’s kind of cool!

I think my next project is going to have to be trying WebDriver with browsers other than Firefox.  I’d like to be able to test the same workflow in IE, Firefox, and Chrome, at least. (And if I get really ambitious, maybe I’ll see about iOS browser automation…)

Stumbling my way through the Drupal API

I’ve had to fix some interesting problems at work recently, related to a Drupal site that we’ll be rolling out soon. I just finished fixing one issue that, while seemingly minor, took quite a while to figure out.

I’m really glad to have come up with a good solution. The thing that amuses me most about this is that, after more than eight hours of messing around, the final solution involved writing only about a half-dozen lines of code.

The problem, in a nutshell, is that we have a content type in the system that represents a university. There’s a location field on each node with, minimally, city and country specified. The user can search for locations, using some custom search code, and the search results are displayed via a standard Drupal view. We allow the user to sort the results by one of a few different fields, with the sort drop-down exposed from the view. This works fine, except when sorting by country. On the location record, only the two-letter country code is stored, for instance “AE” for “United Arab Emirates”. So, when you sort by country, it’s really sorting on country code, so “AE” goes to the top, which isn’t really what the client wanted.

Of course, the first thing I did was Google the problem. I found this issue discussion, which pretty much matches my problem. There was a suggestion in the comments there about using hook_views_pre_render to re-sort the results right before displaying them. That works great, if you’re not paging results. But, if you’re pulling results back one page at a time from a large result set, this doesn’t work, since the pre-render hook only gives you the current page.

So I figured out that I really need to sort by country name at the SQL level, while retrieving results. This led to my next problem, which is that, even with the location module installed, there’s no SQL country lookup table in Drupal. The list of country codes and names is just stored in code, in an array, which can be retrieved via _country_get_predefined_list. (You shouldn’t call that directly, though, of course; you should use country_get_list.)

So off I went to find a module that could give me a SQL table with country info in it. The countries module does that, and a bit more. So, I installed that and figured out where the country table was. Then, my next blind alley was figuring out how to join to the new country table in a view. I was hoping I could just add a join to it in the view definition, and go from there. Well, I still don’t know that much about Drupal views, and it didn’t seem possible to do that easily.

So, the next blind alley was to see if I could alter the view SQL with hook_views_query_alter, which seemed sensible. Well, the query object that you get from that hook isn’t a nice simple query object that can easily be changed, so that turned out to be another dead end. (It’s likely possible that I could have figured it out, but it seemed like the wrong approach.)

Then, finally, I stumbled across this SO question. The one answer posted there led me in the direction of modifying the query with hook_query_alter, which can be used to modify just about any query Drupal issues to MySQL. So, finally, I found a workable solution.

hasAllTags('views', 'views_university_search')) {
    $ord =& $query->getOrderBy();
    if (array_key_exists('location_country', $ord)) {
      $query->addJoin('INNER', 'countries_country', 'cc', 'cc.iso2 = location.country');
      $ord = array('cc.name' => $ord['location_country']);
    }
  }
}

So that’s it. I add a join, and replace the ‘order by’ clause. About a half-dozen lines of code. Oh, and I now also understand passing by reference in PHP a little better too!

fun with WSDL and CURL

Ever since the debacle described in this blog post, I’ve made it a point to double-check the WSDL on the SOAP web services for our main product, any time I’m doing a non-trivial rollout, even if I know I haven’t changed anything that should affect the WSDL.

Up until today, I’ve always just done it by bringing up the WSDL URL for each web service in Firefox, and saving it to a text file. There’s only a half-dozen web services, so it doesn’t take that long. But this morning I finally broke down and wrote a batch file to fetch them all, using cURL.

I’ve gotten a bit more enthusiastic about using cURL, and other tools, to simplify things for me recently, since reading this blog post by Scott Hanselman.

random PHP functions

I’m still doing a fair amount of PHP work. Right now, it’s all Drupal, for a site we’re rolling out very soon. I keep stumbling across random PHP functions I hadn’t heard of before, and that turn out to be nice little time savers. Two examples:

  1. curl_setopt_array: I used to just call curl_setopt() a bunch of times to set all my options Now I can set a bunch of options in one fell swoop.
  2. http_build_query: Nothing I couldn’t previously do with simple string concatenation, but this is much cleaner.

Mandrill REST API

OK, here’s another REST API example. In the same system I blogged about yesterday, I also had to use the Mandrill API to send out some transactional emails. Mandrill is a relatively new service from MailChimp that can be used to send out e-mail via simple API calls. It’s pretty nice, and free to use, up to a certain point.

You can actually use it via regular old SMTP, with some control info added into the headers, or you can go ahead and use their REST API. I decided to use the API. Initially, I had no luck with that at all. After thrashing around a bit, I eventually figured out that I had to update to the newest version of EasyHttp. Then, it started working fine, no problem. I didn’t bother looking into the internals of EasyHttp to try and figure out what was causing the problem. I’m kind of curious, but, as usual, it’s more important to just get the thing working.

So, here’s another Gist with some code:

/*
 * Code/Mandrill.cs
 * These are calls to the Mandrill email service.
 * Reference: https://mandrillapp.com/api/docs/
 * 
 * ajh 2012-08-09: new
 */
using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Configuration;
using System.Dynamic;
using EasyHttp.Http;
using System.Net;
using log4net;

namespace Sample.Web
{
    enum MandrillError
    {
        OK,
        WebException,
        HttpNotOk,
        Invalid,
        Rejected,
        Unknown
    }

    public class Mandrill
    {
        static string MandrillBaseUrl = ConfigurationManager.AppSettings["MandrillBaseUrl"];
        static Guid MandrillKey = new Guid(ConfigurationManager.AppSettings["MandrillKey"]);

        public static bool SendActivationEMail(BLL.TrialSignup ts, out string errorMsg)
        {
            string activationLink =
                HttpContext.Current.Request.Url.GetLeftPart(UriPartial.Authority) + "/Register/Activation.aspx?id=" + ts.Id;
            
            //send-template(string key, string template_name, array template_content, struct message) 
            dynamic sendParams = new ExpandoObject();
            sendParams.key = MandrillKey;
            sendParams.template_name = "Secret Project Trial Activation";

            sendParams.template_content = new List();
            
            sendParams.message = new ExpandoObject();
            sendParams.message.subject = "Here's your Secret Project activation email";
            sendParams.message.from_email = "info@SecretProject.com";
            sendParams.message.from_name = "Secret Project";

            sendParams.message.to = new List();
            sendParams.message.to.Add(new ExpandoObject());
            sendParams.message.to[0].email = ts.EMail;
            sendParams.message.to[0].name = ts.Name;

            sendParams.message.track_opens = true;
            //sendParams.message.track_clicks = true;

            sendParams.message.global_merge_vars = new List();
            sendParams.message.global_merge_vars.Add(new ExpandoObject());
            sendParams.message.global_merge_vars[0].name = "NAME";
            sendParams.message.global_merge_vars[0].content = ts.Name;

            sendParams.message.global_merge_vars.Add(new ExpandoObject());
            sendParams.message.global_merge_vars[1].name = "LINK";
            sendParams.message.global_merge_vars[1].content = activationLink;

            errorMsg = string.Empty;
            
            MandrillError merr = SendMessage(sendParams);

            switch (merr)
            {
                case MandrillError.OK:
                    return true;
                case MandrillError.WebException:
                case MandrillError.HttpNotOk:
                    errorMsg = "There was an issue sending your activation e-mail. Please try again later or call us directly.";
                    break;
                case MandrillError.Invalid:
                    errorMsg = "Your email address appears to be invalid. Please try again with a valid address, or call us directly.";
                    break;
                case MandrillError.Rejected:
                    errorMsg = "Your activation email was rejected. Please try again with a valid address, or call us directly.";
                    break;
                case MandrillError.Unknown:
                    errorMsg = "There was an unknown problem sending your activation email. Please try again, or call us directly.";
                    break;
            }
            return false;
        }

        public static bool SendSalesNotification(BLL.TrialSignup ts)
        {
            dynamic sendParams = new ExpandoObject();
            sendParams.key = MandrillKey;
            sendParams.template_name = "Secret Project Trial Sales Notification";

            sendParams.template_content = new List();

            sendParams.message = new ExpandoObject();
            sendParams.message.subject = "Secret Project Trial Account Notification";
            sendParams.message.from_email = "info@SecretProject.com";
            sendParams.message.from_name = "Secret Project";

            sendParams.message.to = new List();
            sendParams.message.to.Add(new ExpandoObject());
            sendParams.message.to[0].email = ConfigurationManager.AppSettings["SalesEmail"];
            sendParams.message.to[0].name = "Secret Project Sales";

            //sendParams.message.track_opens = true;
            //sendParams.message.track_clicks = true;

            sendParams.message.global_merge_vars = new List();
            sendParams.message.global_merge_vars.Add(new ExpandoObject());
            sendParams.message.global_merge_vars[0].name = "NAME";
            sendParams.message.global_merge_vars[0].content = ts.Name;

            sendParams.message.global_merge_vars.Add(new ExpandoObject());
            sendParams.message.global_merge_vars[1].name = "COMPANY";
            sendParams.message.global_merge_vars[1].content = ts.CompanyName;
            
            sendParams.message.global_merge_vars.Add(new ExpandoObject());
            sendParams.message.global_merge_vars[2].name = "EMAIL";
            sendParams.message.global_merge_vars[2].content = ts.EMail;

            MandrillError merr = SendMessage(sendParams);

            switch (merr)
            {
                case MandrillError.OK:
                    return true;
                case MandrillError.WebException:
                case MandrillError.HttpNotOk:
                case MandrillError.Invalid:
                case MandrillError.Rejected:
                case MandrillError.Unknown:
                    break;
            }
            return false;
        }

        private static MandrillError SendMessage(dynamic sendParams)
        {
            ILog _log = log4net.LogManager.GetLogger("Mandrill/SendMessage");

            string url = MandrillBaseUrl + "/messages/send-template.json";

            var http = new HttpClient
            {
                Request = { Accept = HttpContentTypes.ApplicationJson }
            };

            EasyHttp.Http.HttpResponse response;
            try
            {
                response = http.Post(url, sendParams, HttpContentTypes.ApplicationJson);
            }
            catch (WebException ex)
            {
                _log.ErrorFormat("Error: WebException - {0}", ex.Message);
                return MandrillError.WebException;
            }

            if (response.StatusCode != HttpStatusCode.OK)
            {
                _log.InfoFormat("Response = {0} - {1}", response.StatusCode, response.StatusDescription);
                _log.Info(response.RawText);
                return MandrillError.HttpNotOk;
            }

            dynamic rv = response.DynamicBody;
            _log.InfoFormat("email: {0}, status: {1}", rv[0].email, rv[0].status);

            string send_status = rv[0].status;
            if (send_status == "sent" || send_status == "queued")
                return MandrillError.OK;

            // otherwise, it should be "rejected" or "invalid"
            if (send_status == "invalid")
            {
                return MandrillError.Invalid;
            }
            if (send_status == "rejected")
            {
                return MandrillError.Rejected;
            }

            // unexpected...
            return MandrillError.Unknown;
        }
        
    }
}

I hope someone finds it useful and/or entertaining.
In this code, I’m sending out two e-mails, one is a trial account activation email, and the other is just a notification to sales. On the first, I’m looking at the errors closely and trying to return a useful error message, so the customer knows that something has gone wrong and has a clue about how to handle it. On the second, I’m less worried about that, as it’s just a notification to our salesperson.

Zoho CRM REST API

I have about a dozen programming-related items floating around in my head that I’d like to write up short blog entries about. Maybe I’ll manage to get one or two written up tonight.

I’ve been doing some work recently with some random REST APIs. When I’m calling REST APIs from .NET, I like to use EasyHttp. Based on the stats on its NuGet page, I guess it’s not that popular, but it’s updated often enough, and it works well for me.

I recently worked on a project where I had to create a lead in Zoho CRM, using their API. I had a little bit of trouble, but only because I didn’t have quite the right parameters set — it turns out that “newformat=1” is really important. I had looked for some sample C# code for the Zoho API when I started this, and didn’t find any, so I thought I’d post a quick Gist with my code:

/*
 * Sample C# code to create a lead with the Zoho CRM API.
 * ajh 2012-08-23
 */
using System;
using System.Collections.Generic;
using System.Configuration;
using System.Dynamic;
using System.Linq;
using System.Net;
using System.Web;
using System.Xml;
using System.Xml.Linq;
using EasyHttp.Http;
using log4net;

namespace Sample.Web
{
    public class ZohoCrm
    {
        static string ZohoApiBaseUrl = "https://crm.zoho.com/crm/private/xml/Leads/insertRecords"; 
        static string ZohoApiKey = ConfigurationManager.AppSettings["ZohoApiKey"];

        public static bool CreateLead(BLL.SampleLead sl)
        {
            ILog _log = log4net.LogManager.GetLogger("ZohoCrm/CreateLead");

            XDocument xmlData = new XDocument(
            new XElement("Leads",
                new XElement("row", new XAttribute("no", "1"),
                    new XElement("FL", new XAttribute("val", "Lead Source"), "Trial Signup"),
                    new XElement("FL", new XAttribute("val", "Company"), sl.CompanyName),
                    new XElement("FL", new XAttribute("val", "Last Name"), sl.Name),
                    new XElement("FL", new XAttribute("val", "Email"), sl.EMail)
                )));

            string url = string.Format("{0}?authtoken={1}&scope=crmapi&newFormat=1&xmlData={2}", 
				ZohoApiBaseUrl, ZohoApiKey, 
                HttpUtility.UrlEncode(xmlData.ToString()));

            var http = new HttpClient
            {
                Request = { Accept = HttpContentTypes.ApplicationXml }
            };

            dynamic emptyPost = new ExpandoObject();            
            EasyHttp.Http.HttpResponse response;
            try
            {
                response = http.Post(url, emptyPost, HttpContentTypes.ApplicationXml);
            }
            catch (WebException ex)
            {
                _log.ErrorFormat("Error: WebException - {0}", ex.Message);
                return false;
            }

            if (response.StatusCode != HttpStatusCode.OK)
            {
                _log.InfoFormat("Response = {0} - {1}", response.StatusCode, response.StatusDescription);
                _log.Info(response.RawText);
                return false;
            }

            XDocument xdoc;
            try
            {
                xdoc = XDocument.Parse(response.RawText);
            }
            catch (XmlException ex)
            {
                _log.ErrorFormat("Error: XmlException parsing API response - {0}", ex.Message);
                _log.Info(response.RawText);
                return false;
            }

            string msg;
            try
            {
                msg = xdoc.Descendants("result").First().Element("message").Value;
            }
            catch (Exception ex)
            {
                _log.ErrorFormat("Error: Exception reading from API response - {0}", ex.Message);
                _log.Info(response.RawText);
                return false;
            }
            if (msg != "Record(s) added successfully")
            {
                _log.InfoFormat("Unexpected XML result: {0}", msg);
                _log.Info(response.RawText);
                return false;
            }

            return true;
        }
    }

(See also this Gist.)

I’m also using log4net here.  I’m a really big fan of log4net, and I use it in almost all of my projects. I have a pretty standard log.config file I use that sets up a RollingFileAppender, with ten files of 100k each. That’s usually enough to keep a few days or weeks worth of history, depending on the level of logging and the level of activity. And I never have to worry about the log files growing out of control; they just keep rolling over. I think log4net is pretty popular, based on the numbers on their NuGet page.

I try to do a lot of error-handling whenever I’m dealing with a REST API. There are plenty of things that can go wrong. You can see in the code above that I try to trap anything that might throw an exception and log it. I’m never sure if I’m doing this the “right” way. I know some people, when writing a routine like this, would just let the exceptions happen, then have a try/catch around the call to the CreateLead method, and deal with it at a higher level.

Classic ASP – SQL Injection

We still have a few old sites at work that are in classic ASP. One of the problems that tends to occur on these old sites is SQL injection attacks. A lot of old ASP code was written without taking SQL injection into account. It would be great if we could just rewrite all of these sites in ASP.NET, but sometimes the client isn’t interested in doing that.

Well, yesterday, we had a production ASP site get hit. This is a site for an old client that I’ve personally never worked on, so I really knew nothing about it, but I’m getting dragged in, now that we need to clean it up. Looking at the site, I’m actually surprised this hadn’t happened earlier.

Looking at the code, I see a few places where we could be doing a better job of input validation. And also a few places where we’re doing un-parameterized SQL, which is a big no-no if you want to avoid SQL injection. So, I’m going to try to clean some of that up.

I also want to use URLScan on the server, with some SQL injection rules, to try to get some of this stuff caught at the IIS level. I found this article on how to add some rules to the URLScan.ini file to mitigate SQL injection attacks. (I actually first started reading this article, but then remembered that this particular web server is still on IIS 6.)

When I started poking around on the server, I was surprised to see tha URLScan was already installed. However, it was not configured to do any SQL injection prevention. So, Monday morning, I’m going to try to add the SQL injection rules to the ini file, and see if that breaks anything. Then, I hope to have time to tighten up the code a bit and roll out a new version. I can’t say I’m excited to be working on nearly decade-old VBScript, but hey, it’ll be good for a few laughs, right?

Using SSH with Mercurial

We’ve been using Mercurial and Bitbucket at work for quite a while now. Things have been going pretty smoothly, and I don’t regret the choice of Hg over Git or TFS at all. Nor do I regret using Bitbucket as a back-end. They’ve had occasional outages, but probably no more than we’d have with Github or any other web-based service.

I had a bit of a problem pushing a really big changeset today though, so I decided to switch from HTTPS to SSH. It took about 20 minutes to set up, following these instructions. And I had to look here for some additional information on Pageant. I wish there had been a smoother, faster, way to get the whole thing set up, but it’s running smoothly now that I’ve got it all figured out.