Running Selenium UI Tests in Azure Pipelines

This tutorial guides you through the process of integrating your Selenium UI tests into an Azure DevOps pipeline. By automating your UI tests, you can ensure the quality and stability of your web applications with every code change.

Prerequisites

Step 1: Prepare Your Selenium Tests

Organize Your Test Project

Ensure your Selenium test project is well-structured. Common project types include NUnit, xUnit, MSTest (for .NET), TestNG, JUnit (for Java), Pytest, Unittest (for Python), or Mocha, Jest (for JavaScript).

Your tests should be designed to run independently and report results in a standard format (like JUnit XML).

Best Practice: Headless Execution

For running tests in a CI environment, it's highly recommended to configure Selenium to run in headless mode. This means the browser window won't be displayed visually, making it faster and suitable for server environments.

Example for Chrome (C#):


var options = new ChromeOptions();
options.AddArgument("--headless");
options.AddArgument("--disable-gpu"); // Often needed for headless mode
options.AddArgument("--window-size=1920,1080");

IWebDriver driver = new ChromeDriver(options);

Step 2: Create an Azure Pipeline

Navigate to Pipelines

In your Azure DevOps project, go to Pipelines > Pipelines and click New pipeline.

Connect to Your Repository

Select your repository location (e.g., Azure Repos Git, GitHub) and authenticate if prompted.

Configure Your Pipeline

Choose Starter pipeline or select a template relevant to your project.

You'll be presented with a YAML file that defines your pipeline. This is where you'll add tasks to build, test, and deploy.

Step 3: Add Test Execution Task

Within your YAML pipeline, you need to add a task that will execute your Selenium tests. The specific task depends on your project type and the agent you're using.

Example for .NET (NUnit/xUnit/MSTest)

If you're using .NET projects, the VSTest task is commonly used.


pool:
  vmImage: 'windows-latest' # Or 'ubuntu-latest' if your tests support it

steps:
- task: VSBuild@1
  inputs:
    solution: '**/*.sln'
    msbuildArgs: '/p:DeployOnBuild=true /p:WebPublishMethod=Package /p:PackageAsSingleFile=true /p:SkipInvalidConfigurations=true /p:DesktopBuildPackageLocation="$(build.artifactStagingDirectory)\WebApp.zip" /p:DeployIisAppPath="Default Web Site"'
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'

- task: VSTest@2
  inputs:
    platform: '$(buildPlatform)'
    configuration: '$(buildConfiguration)'
    testResultsFiles: '**\*test*.trx'
    mergeTestResults: true
    failOnMinTestsNotRun: true
    testFiltercriteria: '| Category=UITest' # Optional: filter tests by category
  displayName: 'Run Selenium UI Tests'

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'VSTest'
    testResultsFiles: '**\*test*.trx'
    mergeTestResults: true
    failOnMinTestsNotRun: true
  displayName: 'Publish Test Results'

Example for Node.js (JavaScript/TypeScript) with Selenium-webdriver

You would typically use a script defined in your package.json and run it using the npm or yarn task.


pool:
  vmImage: 'ubuntu-latest'

steps:
- task: NodeTool@0
  inputs:
    versionSpec: '18.x' # Specify your Node.js version
  displayName: 'Install Node.js'

- script: |
    npm install
    npm run test:ui
  displayName: 'Install Dependencies and Run UI Tests'

- task: PublishTestResults@2
  inputs:
    testResultsFormat: 'JUnit' # Assumes your test runner outputs JUnit XML
    testResultsFiles: '**/TEST-*.xml'
    mergeTestResults: true
    failOnMinTestsNotRun: true
  displayName: 'Publish Test Results'

WebDriver Management

Ensure that the correct WebDriver executable (e.g., chromedriver, geckodriver) is available on the build agent. You might need to install it using a package manager or download it as part of your pipeline. Some tasks or extensions can help manage this.

For example, you can use the WebDriverInstaller task from the Azure DevOps Marketplace.

Step 4: Configure Test Agents

Depending on your test setup, you might need to configure specific environments on your build agents:

Step 5: Run and Monitor Your Pipeline

Save and Run

Save your YAML pipeline and trigger a new run. Monitor the pipeline execution to identify any failures.

Analyze Results

After the pipeline completes, navigate to the Tests tab in your pipeline run summary to view detailed test results, including any failures and screenshots.

Failing UI tests can indicate regressions in your application or issues with the test environment itself.

Troubleshooting Common Issues