Andy Kelk

Headless browser testing with PhantomJS, Selenium WebDriver, C#, NUnit and Mono

One of the key parts of speeding up the web application development is automating the deployment pipeline. In order to automate deployment, you need to automate your testing. Automated regression testing for a web application typically relies on automating a browser to verify that your application is working as expected. Selenium has long been a staple for teams looking to automate browsers without investing in costly proprietary solutions.

Automating Firefox with Selenium works well and is a tried and trusted solution for automation testing. It’s even possible to do it headless using a virtual framebuffer such as Xvfb. However, the main gripe with browser automation is that it’s slow and everyone’s impatient with slow tests (and rightly so!).

That’s where PhantomJS comes in. PhantomJS is a headless browser built using the popular WebKit library. Because it’s built with WebKit, it acts just like a normal browser except that you can’t see it (unless you ask it to give you a screenshot). There’s plenty of documentation online about doing headless testing with PhantomJS. Since we have a lot invested in the .Net platform, I wanted to see if we could integrate it into a platform that a majority of our team are already familiar with.

Just to make it harder, I also decided that I’d run the tests using Mono on a Linux platform (mostly because my main work laptop is running Fedora and I was too lazy to boot my Windows VM).

Having downloaded the PhantomJS binary, I used nuget to install the Selenium.WebDriver package and tried to get a simple Hello World test going (load up iproperty.com.my and check the title of the page). I found that a lot of the examples and documentation that are available are written using the Java bindings. The C# bindings definitely seem to be second-class citizens when it comes to Selenium. I did find a handy getting started guide for using PhantomJS with Selenium in .Net.

Selenium includes a PhantomJSDriver class which will instantiate PhantomJS for you while setting up your tests (assuming it’s in the working directory or in the PATH somewhere). However, I struggled to get that working on Linux as it assumes various things about the platform (such as the binary name being PhantomJS.exe). There is a PhantomJSDriverService class which may allow customisation on things like the name of the binary but there was no documentation, I’m not a C# guru and I was getting impatient.

My next attempt was to use the RemoteWebDriver class and to manually instantiate the phantomjs binary outside of the test setup. That seemed to work most of the time but not consistently. It seems that there’s some timing issues which lead to HTTP requests to the phantomjs process failing. For testing having this happen is a no-no; flaky tests mean that people will ignore test failures.

Starting PhantomJS in WebDriver mode:

phantomjs --webdriver=4444

Instantiating a RemoteWebDriver to talk to PhantomJS:

_driver = new RemoteWebDriver(
  new Uri("http://127.0.0.1:4444/wd/hub"),
  DesiredCapabilities.Chrome()
);

Occasionally reads from PhantomJS will fail with errors such as:

OpenQA.Selenium.WebDriverException : A exception with a null response
was thrown sending an HTTP request to the remote WebDriver server for
URL http://127.0.0.1:4444/wd/hub/session/2afc1ba0-7ff0-11e2-af83-af91a8b8229c/element.
The status of the exception was ReceiveFailure, and the message was:
Error getting response stream (ReadDone2): ReceiveFailure

Inserting sleep calls into the code and setting PhantomJS to debug mode both helped with the problem which indicates that it’s something to do with timing. After lots of frustration, I decided that it was time to stop being dogmatic about getting this running on Linux and turned to my Windows VM. I went back to the PhantomJSDriver class and the documented methods worked flawlessly. Here’s a simple example of a test:

using System;
using NUnit.Framework;
using OpenQA.Selenium;
using OpenQA.Selenium.PhantomJS;

namespace PhantomDemo
{
  [TestFixture()]
  public class Test
  {
    private IWebDriver _driver;

    [TestFixtureSetUp]
    public void FixtureSetup()
    {
      _driver = new PhantomJSDriver();
      _driver.Manage().Timeouts().ImplicitlyWait(new TimeSpan(0, 0, 30));
    }

    [TestFixtureTearDown]
    public void FixtureTearDown()
    {
      if (_driver != null) 
      {
        _driver.Quit();
        _driver.Dispose();
      }
    }

    [Test()]
    public void TestLogin ()
    {
      _driver.Navigate().GoToUrl(
        "http://www.iproperty.com.my/useracc/Login.aspx"
      );
      _driver.FindElement(By.Id("txtEmail")).SendKeys("[email protected]");
      _driver.FindElement(By.Id("txtPassword")).SendKeys("xxxxx");
      _driver.FindElement(By.Id("imgbtnLogin")).Submit();

      Assert.AreEqual(
        "Andy",
        _driver.FindElement(By.ClassName("sidebar-text5")).Text
      );

    }
  }
}

And how about the speed? I compared three different methods of running two identical testcases on the same Windows VM : Chrome, Firefox and PhantomJS and these were the results:

Browser Run time (average of 3 runs)
Firefox 50.04 seconds
Chrome 38.77 seconds
PhantomJS 30.42 seconds

Overall, running with PhantomJS should save us precious time in running our automated tests. Once it runs more happily on Linux, it’ll be even better (and yes, I intend to start looking at the source and submitting some patches to help out).

Related Posts

16 thoughts on “Headless browser testing with PhantomJS, Selenium WebDriver, C#, NUnit and Mono”

  1. Ditch Selenium and use a Javascript testing library directly with Phantomjs if you want huge performance gains akelk 🙂

    I briefly helped a bunch of hipsters with their CI problems last year and reduced 6+ minutes of unreliable Cucumber, Capybara and Selenium garbage (on Rails) into a few seconds of Javascript tests. Less code, less middle men to screw things up and faster tests. http://casperjs.org is a good starting point

    Reply ↓

  2. Good tip, shanna. I reimplemented the same test suite using casperjs this morning.

    Getting JavaScript driven elements to play nicely was a bit tricky but once I’d tackled that, it was pretty straightforward.

    Timing-wise, the test suite runs on my Windows VM in about 14 seconds. To try and rule out variability of the test subject, I re-ran the Selenium tests and they clocked in at 22 seconds. So still a good saving by using casperjs instead of Selenium.

    Reply ↓

  3. Pingback: Headless browser testing with PhantomJS and CasperJS | Andy Kelk

  5. Can i use the PhantomJs for displaying and interacting with some html pages through a WPF application?
    My requirement is like i want to navigate to an html page in system hard disk , and want to do some interactions like Content selection , right click etc. At present i am using Awesomium Web Browser control , but lot of limitations are there for that .
    So can i use PhantomJs as a webcontrol? Is there any reference or links to achieve the smae?

    Reply ↓

  6. Hi Andy,
    I am new to this tool.I saw video from YouTube to setup the environment and I have written a basic script using selenium/C# i.e. open browser > go to Google> then open YouTube.
    When i build it using VS 2010 its fine. Now when I run in N-unit I am getting the error like
    1. driver.FindElement(By.Id(“gbqfq”)).Clear(); is highligted.
    2. It gives me error like RemoteWebDriver.cs cannot be opned.
    . The server is running and rest of the things like adding .dll files etc is all done. But this error still exists. Can you please help me how to fix this issue.

    Reply ↓

  7. Hi Andy,
    I am new to this Selenium tool. I saw videos from you tube and did environment setup. Now when i try to run even simple script I am getting the following errors.
    1. By.Id is highlighted
    2. Remotewebserver.cs cannot be opened.
    All the .dll files are added and also server is also running. Then also I am getting this error. Can u please help to resolve this issue.

    Reply ↓

      • Yes Andy, I do have these statements in the scripts.Below is the partial script.

        using System;
        using System.Threading;
        using System.Configuration;
        using System.Collections.Generic;
        using System.Linq;
        using System.Text;

        using System.Text.RegularExpressions;
        using NUnit.Framework;
        using Selenium;
        using OpenQA.Selenium;
        using OpenQA.Selenium.Firefox;
        using OpenQA.Selenium.Support.UI;
        public void TheSampleTest()
        {
        driver.Navigate().GoToUrl(baseURL + “/?gws_rd=cr&ei=hSY8UuJ8hNmtB-6mgdAL”);
        Thread.Sleep(200);
        driver.FindElement(By.Id(“gbqfq”)).Clear();
        Thread.Sleep(200);
        driver.FindElement(By.Id(“gbqfq”)).SendKeys(“youtube”);
        Thread.Sleep(200);
        driver.FindElement(By.CssSelector(“em”)).Click();
        Thread.Sleep(200);
        }

        Reply ↓

  8. Yes I have added WebDdriver.dll assembly in the project with the matching version of .Net framework.
    Still after executing it is showing me the same error as mentioned in my first post.
    And I am using visual studio 2010/.Net framework 4.0 and Selenium WebDriver 2.35.0.

    Reply ↓

  9. As the maintainer, I take exception to the notion that the .NET bindings are “second-class citizens” when it comes to Selenium WebDriver. We (well, I, anyway) have worked very hard to maintain feature parity with other language bindings.

    Translating WebDriver examples from Java to C# is extraordinarily easy, almost trivial. There is nearly a one-to-one mapping between the APIs. The major differences are that the .NET bindings use PascalCase instead of the Java camelCase, and that in general, methods in Java that return values and take no parameters have become properties in .NET. That makes the “lack of examples” less of a barrier, as the API translation is incredibly simple.

    Granted, running the .NET bindings using Mono on Linux isn’t a generally tested environment. If you’re looking for somewhere to place blame for that, you can blame me. Selenium is a volunteer effort for me, and there are only so many hours in the day.

    Reply ↓

    • Hi Jim,

      I was more talking about the general adoption of the .Net bindings by the community. Selenium is widely used by the Java community but less so by the .Net community. It’s by no means a reflection on the excellent work done to provide the bindings for .Net within the project. You’re absolutely right that translating Java examples is pretty trivial; however, that may be beyond some single-language programmers who are after pure copy-paste examples…

      As with any open source project, *blaming* a maintainer is a counter-productive exercise – I’m certainly very appreciative of all of the work that’s been done by all of the Selenium project team to create such a rich and widely used testing platform.

      Reply ↓

      • > that may be beyond some single-language programmers who are after pure copy-paste examples

        If you’re a programmer you’re going to have to use your brain. If all you want is “copy-paste examples”, you are a bad programmer, and you should probably not be allowed to call yourself one.

        Reply ↓

Leave a Reply

Your email address will not be published. Required fields are marked *

*
*

Archives