Headless browser testing with PhantomJS, Selenium WebDriver, C#, NUnit and Mono

One of the key parts of speeding up the web application development is automating the deployment pipeline. In order to automate deployment, you need to automate your testing. Automated regression testing for a web application typically relies on automating a browser to verify that your application is working as expected. Selenium has long been a staple for teams looking to automate browsers without investing in costly proprietary solutions.

Automating Firefox with Selenium works well and is a tried and trusted solution for automation testing. It’s even possible to do it headless using a virtual framebuffer such as Xvfb. However, the main gripe with browser automation is that it’s slow and everyone’s impatient with slow tests (and rightly so!).

That’s where PhantomJS comes in. PhantomJS is a headless browser built using the popular WebKit library. Because it’s built with WebKit, it acts just like a normal browser except that you can’t see it (unless you ask it to give you a screenshot). There’s plenty of documentation online about doing headless testing with PhantomJS. Since we have a lot invested in the .Net platform, I wanted to see if we could integrate it into a platform that a majority of our team are already familiar with.

Just to make it harder, I also decided that I’d run the tests using Mono on a Linux platform (mostly because my main work laptop is running Fedora and I was too lazy to boot my Windows VM).

Having downloaded the PhantomJS binary, I used nuget to install the Selenium.WebDriver package and tried to get a simple Hello World test going (load up iproperty.com.my and check the title of the page). I found that a lot of the examples and documentation that are available are written using the Java bindings. The C# bindings definitely seem to be second-class citizens when it comes to Selenium. I did find a handy getting started guide for using PhantomJS with Selenium in .Net.

Selenium includes a PhantomJSDriver class which will instantiate PhantomJS for you while setting up your tests (assuming it’s in the working directory or in the PATH somewhere). However, I struggled to get that working on Linux as it assumes various things about the platform (such as the binary name being PhantomJS.exe). There is a PhantomJSDriverService class which may allow customisation on things like the name of the binary but there was no documentation, I’m not a C# guru and I was getting impatient.

My next attempt was to use the RemoteWebDriver class and to manually instantiate the phantomjs binary outside of the test setup. That seemed to work most of the time but not consistently. It seems that there’s some timing issues which lead to HTTP requests to the phantomjs process failing. For testing having this happen is a no-no; flaky tests mean that people will ignore test failures.

Starting PhantomJS in WebDriver mode:

phantomjs --webdriver=4444

Instantiating a RemoteWebDriver to talk to PhantomJS:

_driver = new RemoteWebDriver(
 new Uri("http://127.0.0.1:4444/wd/hub"),
 DesiredCapabilities.Chrome()
);

Occasionally reads from PhantomJS will fail with errors such as:

OpenQA.Selenium.WebDriverException : A exception with a null response
was thrown sending an HTTP request to the remote WebDriver server for
URL http://127.0.0.1:4444/wd/hub/session/2afc1ba0-7ff0-11e2-af83-af91a8b8229c/element.
The status of the exception was ReceiveFailure, and the message was:
Error getting response stream (ReadDone2): ReceiveFailure

Inserting sleep calls into the code and setting PhantomJS to debug mode both helped with the problem which indicates that it’s something to do with timing. After lots of frustration, I decided that it was time to stop being dogmatic about getting this running on Linux and turned to my Windows VM. I went back to the PhantomJSDriver class and the documented methods worked flawlessly. Here’s a simple example of a test:

using System;
using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;

namespace PhantomDemo
{
 [TestFixture()]
 public class Test
 {
 private IWebDriver _driver;

 [TestFixtureSetUp]
 public void FixtureSetup()
 {
 _driver = new PhantomJSDriver();
 _driver.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));
 }

 [TestFixtureTearDown]
 public void FixtureTearDown()
 {
 if (_driver != null)
 {
 _driver.Quit();
 _driver.Dispose();
 }
 }

 [Test()]
 public void TestLogin ()
 {
 _driver.Navigate().GoToUrl(
 "http://www.iproperty.com.my/useracc/Login.aspx"
 );
 _driver.FindElement(By.Id("txtEmail")).SendKeys("andy@andykelk.net");
 _driver.FindElement(By.Id("txtPassword")).SendKeys("xxxxx");
 _driver.FindElement(By.Id("imgbtnLogin")).Submit();

 Assert.AreEqual(
 "Andy",
 _driver.FindElement(By.ClassName("sidebar-text5")).Text
 );

 }
 }
}

And how about the speed? I compared three different methods of running two identical testcases on the same Windows VM : Chrome, Firefox and PhantomJS and these were the results:

Overall, running with PhantomJS should save us precious time in running our automated tests. Once it runs more happily on Linux, it’ll be even better (and yes, I intend to start looking at the source and submitting some patches to help out).