Running Selenium UI tests in an Azure DevOps pipeline

Introduction

There are a few posts already out there dealing with this but I couldnt find one that managed to cover all the steps in enough detail for me. So this post will attempt to rectify that.

What do we need?

• A web site to test. I've got a pretty bare-bones ASP.NET Core web application (just using the Visual Studio 2019 template) that I've been playing around with recent versions of Entity Framework Core in. I'll write the tests against the home page of that site.
• A test project. For convenience I'll be adding the test project to the same solution as the web application.
• A release pipeline. For unit and integration tests youd normally run these as a step in a build pipeline, preventing the deployment if the tests fail. For UI tests though I need something to be deployed in order to test it. I'll be adding the test step to a release pipeline.

The web application

As stated, I'm using the template ASP.NET Core web application in Visual Studio 2019. Heres the project structure. I've added some data context related stuff but you dont need any of that for the purpose of these tests.

Debug this project and you should see something resembling the following.

And thats it. Now I need to publish this somewhere. I'm lucky enough to have an Azure subscription at my disposal currently so thats where this will live. Of course the site can live anywhere publicly accessible.

Lets move on to writing some tests for this web application.

The test project

First, I'll create a new project in our existing solution. Right click on the solution in the Solution Explorer pane and select Add > New Project….

In the first dialogue select Class Library (.NET Core) then click Next.

In the next dialogue give your project a name and a home. Click Create.

Visual Studio will open up the newly created Class1 class. Rename this to something sensible (I've chosen HomePageTests) and lets start adding the dependencies.

I've added the following nuget packages:

• FluentAssertions (Version="5.6.0")
• Microsoft.AspNetCore.TestHost (Version="2.2.0")
• Microsoft.NET.Test.Sdk (Version="16.1.1")
• MSTest.TestAdapter (Version="2.0.0")
• MSTest.TestFramework (Version="2.0.0")
• Selenium.Support (Version="3.141.0")
• Selenium.WebDriver (Version="3.141.0")
• Selenium.WebDriver.IEDriver (Version="3.141.59")

Before I write the actual test method I'm going to add some setup and tear down methods in the class. Not strictly necessary for a class with a single test method but as the test project grows its useful to have this stuff in a single place.

First up, the using statements – I hate it when you see code posted for a class or method without these included.

using FluentAssertions;
using Microsoft.VisualStudio.TestTools.UnitTesting;
using OpenQA.Selenium;
using OpenQA.Selenium.IE;
using System;
using System.IO;

Next, decorate the class with the [TestClass] attribute.

namespace EfCoreSandbox.Tests
{
    [TestClass]
    public class HomePageTests
    {
    }
}

Now I can add some variables. The webAppBaseUrl is the URL for the publicly accessible website. You wouldnt normally hard-code this but rather have it in some kind of configuration store but this will do for now.

`private const string webAppBaseUrl = "https://efcoresandbox.azurewebsites.net/";`

The IWebDriver is the interface that all the Selenium web drivers (the things that open up and actually drive the browsers) implement.

`private static IWebDriver driver;`

Next up, ClassInitialise. This runs, provided its been decorated with the [ClassInitialize] attribute, once before running the tests of the class. In this method I call a method to set up the web driver and then navigate to the web applications URL.

[ClassInitialize]
public static void ClassInitialise(TestContext testContext)
{
    SetupDriver();
    driver.Url = webAppBaseUrl;
    driver.Navigate();
}

Then, on the flip side, the ClassCleanup method. As youd imagine this runs once and after all the other test methods have been run.

[ClassCleanup]
public static void ClassCleanup()
{
    TeardownDriver();
}

Heres the SetupDriver method called in the ClassInitialise method. I wont go in to too much detail here as this isnt a post about Selenium testing but this method configures some options for the IEWebDriver (if you have to test IE automate it – otherwise youll need to use IE and nobody wants that). I get the path to the web driver executable – this differs between the local machine and the Azure platform, environment variable on Azure, local directory on the local PC. Lastly set the driver variable to a new InternetExplorerDriver. If any of this fails call the TeardownDriver method to clean up.

private static void SetupDriver()
{
    try
    {
        InternetExplorerOptions ieOptions = new InternetExplorerOptions
        {
            EnableNativeEvents = false,
            UnhandledPromptBehavior = UnhandledPromptBehavior.Accept,
            EnablePersistentHover = true,
            IntroduceInstabilityByIgnoringProtectedModeSettings = true,
            IgnoreZoomLevel = true,
            EnsureCleanSession = true,
        };

        // Attempt to read the IEWebDriver environment variable that exists on the Azure
        // platform and then fall back to the local directory.
        string ieWebDriverPath = Environment.GetEnvironmentVariable("IEWebDriver");
        if (string.IsNullOrEmpty(ieWebDriverPath))
        {
            ieWebDriverPath = Path.GetDirectoryName(AppDomain.CurrentDomain.BaseDirectory);
        }

        driver = new InternetExplorerDriver(ieWebDriverPath, ieOptions)
        {
            Url = webAppBaseUrl
        };
    }
    catch (Exception ex)
    {
        TeardownDriver();
        throw new ApplicationException("Could not setup IWebDriver.", ex);
    }
}

And the TeardownDriver method. Pretty simple – just clean up the resources of the driver.

private static void TeardownDriver()
{
    if (driver != null)
    {
        driver.Close();
        driver.Quit();
        driver.Dispose();
        driver = null;
    }
}

Time to write a test. I'm keeping it simple and am just going to test that the home page has an <h1> heading tag that contains the string "Welcome".

[TestMethod]
public void HomePageHeadingContainsWelcome()
{
    // Arrange/Act/Assert
    driver.FindElement(By.TagName("h1")).Text.Should().Contain("Welcome");
}

And thats it, put all this together and you should be able to run the test. If you dont already have the Test Explorer window open in Visual Studio open it from Test > Windows > Test Explorer.

In the Test Explorer pane click the Run All button – left most button in the toolbar. The test engine will chug away for a bit opening Internet Explorer and loading the web application and all else being well the test will pass.

Next, let set up the release pipeline to run these automatically.

The release pipeline

While the UI tests will run as part of a release pipeline, that pipeline will pick up and deploy the output of a previous build pipeline. So lets get that set up first of all.

In your Azure DevOps portal, open the project and then select Pipelines in the left hand menu.

Click the Create Pipeline button. I'm not yet a fan of YAML so click the "Use the classic editor" link.

Choose the correct settings for your source repository, for me this is an Azure Repos Git repository. Team project and Repository are both EF Core Sandbox and I'm basing it on the master branch.

Click on the Continue button. I've chosen the built-in ASP.NET Core template but feel free to choose something more appropriate to your needs.

Click Apply to continue. Now I need to do some tweaking to the pipeline.

At the pipeline level I've renamed the pipeline to EF Core Sandbox CI and changes the Agent Specification to windows-2019. Also, given we have no unit tests, I've cleared the Project(s) to test field and removed the Test step in the pipeline. The pipeline should look like this for the moment.

Now I'm going to split the existing Publish task in two. Currently it will publish both the web applications and test projects as zip files. Thats not going to work for the test project so split them up. I've renamed the Publish task as Publish Web App and added a new Publish Tests task, configured as follows.

Importantly, uncheck the Zip Published Projects and Publish Web Projects checkboxes then set the Path to Project(s) field as the path to your tests project only. Mine is set to "**/EfCoreSandbox.Tests.csproj".

Set the Arguments field to "–configuration $(BuildConfiguration) –output $(build.artifactstagingdirectory)".

Click Save & queue to run the build. After a while your Pipelines screen should resemble the following.

The last major part of this process is to set up the release pipeline. Click on Releases in the left hand menu and then on the New pipeline button. In the "Select a template" panel click the "Empty job" link.

I've renamed the stage to Deploy web app and closed the Stage panel.

In the Artifacts panel click + Add. Choose Build as the source type and then select the appropriate project and build pipeline previously configured. Click the Add button.

In the Stage panel click the Deploy web app stage to configure it. Under Agent job click the + (plus) button and add an Azure App Service deploy task. Set this up to publish to your Azure app service. I wont go into too much detail here as your configuration will inevitably differ from mine.

Click the + (plus) button again and this time add the "Visual Studio Test Platform Installer" task. This task should need no additional configuration.

Click the + (plus) button again and the last task to add is a "Visual Studio Test" task. For the Test files field I've specifically chosen the EfCoreSandbox.Tests.dll to run and I've specified that the Text mix contains UI tests. For the Search folder I need to be more specific with the location – changing it to $(System.DefaultWorkingDirectory)/_EF Core Sandbox CI/drop

Give the release pipeline a name (I've chosen EF Core Sandbox Release) and save it.

In the real world Id consider running this pipeline in response to a trigger but for now just click Create release in the toolbar (top right) and then on the Create button.

Click on the Release-1 link that appears at the top of the screen. Mine is Release-3 because I've been mucking about with the pipelines, generally getting things wrong, and have created a few more test releases.

Hover over the Deploy web app box and then click on the Logs button when it is shown. In the Logs screen click the VsTest – testAssemblies item in the list. Scroll down a bit and you should see that the HomePageHeadingContainsWelcome did indeed pass.

So there you have it, Selenium UI tests running as part of your deployment pipeline.

Epilogue

This is all great, but, what about when the tests fail. Lets face it, if you knew all the tests were going to pass every time would you bother writing them?

Let make the test fail. In the web application project change the content of the <h1> element. I've opted for "EF Core Sandbox". If you've got your UI tests project configured to be able to run against the local web application you can run the UI tests locally to confirm that they fail.

Next, I'll update the HomePageTests class to provide me with some additional information when a test fails. In the class add the following variable declaration.

private static TestContext testContextInstance;

Now update the ClassInitialise method to set the variable.

[ClassInitialize]
public static void ClassInitialise(TestContext testContext)
{
    testContextInstance = testContext;
    SetupDriver();
    driver.Url = webAppBaseUrl;
    driver.Navigate();
}

And add the following method to the class.

private static void TakeScreenshot(string fileName)
{
    Screenshot ss = ((ITakesScreenshot)driver).GetScreenshot();
    string path = Path.Combine(Directory.GetCurrentDirectory(), fileName);
    ss.SaveAsFile(path);

    testContextInstance.AddResultFile(path);
}

Finally, for the test class, add the following TestCleanup method that will be executed after every test method.

[TestCleanup]
public void TestCleanup()
{
    if (testContextInstance.CurrentTestOutcome != UnitTestOutcome.Passed)
    {
        TakeScreenshot($"{testContextInstance.TestName}.png");
    }
}

As you've probably guessed these code additions will take a screenshot of the web page if the previous test has failed and then add the file as a result file to the test context instance.

Go ahead and push the changes into source control then kick off a new build – unless your build pipeline is already CI triggered. Once the build has run (and succeeded) create a new release. Once the release run has completed you should see something slightly different, if not unexpected.

Drill down into the logs as before and youll be able to see what went wrong.

As youd expect we can see that the test failed with the message: Expected string "EF Core Sandbox" to contain "Welcome".

What about the screenshot though. If you click Tests Plans in the left hand menu and then Runs youll see the list of completed test runs – the latest failed one should be at the top.

Double-click (yep, I know) on the latest run with the warning icon to see the run summary. From there click the Test results link just above the toolbar and under the Run number. This will list all the tests that have failed. Since we only have one double-click the HomePageHeadingContainsWelcome test. Here youll get the error message and stack trace for the failed test along with, about halfway down the page, an Attachments section that should have one file.

Clicking on the attachment name will download the file. Open the download to view a screenshot of the web page at the point the test failed.

Thats it. If you want the code but dont want to copy and paste all of above sections individually heres a link to the complete HomePageTests class. https://gist.github.com/stuartwhiteford/bc21df9e1b98785beef0a6ed66b8c4f8

Happy testing!