View Test Results

Introduction

If you have previously gone through the steps to Execute a Test it is quite easy to pull up the results again. Make sure you first login and in the Test Cases grid click on the 'Last Test Execution => Started' column on any test it will pull up the most recent result.

To see historic results click on the test configuration name and then select the run of interest from the page that loads.

Each tab on the results page will be broken down in detail here.

Results Tab

The Results tab is customizable and contains the following possible widgets.

Filters

Test results can be filtered on the following dimensions:

  • Time: Use the slider along the top of the test results to zoom in on a particular time interval during the test. All widgets will filter to that time period including charts, metrics, logging, images, etc.
  • Region: If your test ran in multiple regions filter the results for just one particular region.
  • Resource: Filter all widgets to show results for one specific resource. This includes all metrics and charts. A resource is simply a label to aggregate on that can include URL, JMeter label, Gatling scenario name, etc.

Execution Progress

While the test is executing, this section shows you some "active" metrics. Active means executed during the most recent interval.

Execution Progress

Summary

See a quick high level summary of how the results of this test execution compares with other recent runs of this same test configuration.

Execution Summary

Overview

Metrics and a chart that give you an overview of performance during the test. Each metric can be toggled on/off on the chart. See how the results during this test compare with up to 10 recent test executions as both a percentage change and a sparkline to spot trends. The set of metrics shown is customizable as well.

Execution Overview Chart

Images

The images widget shows you the most recent screenshots captured while running your test. Tools like Webdriver.io, Protractor, Selenium Java, Selenium Javascript, PhatnomJS, and SlimerJS support screenshot capture and these will appear here in a gallery layout.

Images

Click on any image to see it full sized, compare it visually to previous runs of this test, and see related assertions/results/screenshots captured during the same test iteration.

See the new configuration documentation for more details on configuring image diffing.

Image Diff

Image Related Results

Assertions

For scenarios that capture assertion results like Webdriver.io + Mocha, Selenium Javascript + Mocha, and JMeter this widget displays a summary of the test suite results including a listing of all suites, tests, average duration, errors, etc.

Execution Assertions

Results

By Resource

The results are broken down by resource. A resource usually corresponds to a network resource like GET https://www.google.com, a JMeter Label, or a Gatling scenario name + request label, but can also be anything captured with a custom metric. The columns in this grid are customizable.

If you select a resource in the grid, all charts and traces become specific to that resource only.

Resources

Result Analysis

Analyze the results of individual network requests to get a deep dive into any issues. See the slowest results, failed results, the slowest iterations, all results that were a part of a particular iteration of your scenario, results by user, and all iterations of the test where there was an assertion failure. You can also simply export all results to CSV to analyze further offline.

Result Analysis

Traces

Traces allow you to view all the details of a particular connection made during test execution including metrics, data sent, and data received.

Traces

Example Trace

Trace

Additional Charts/Metrics

Any metric can be charted or displayed. Data points are captured every 10 seconds. Like everything else charts are customizable. For example, for Selenium tests we show a "Browser Performance Metrics widget":

Browser Metrics

Test Runner Usage

View details of memory, CPU, and bandwidth utilization across the test runners as well as broken down by test runner. Select a test runner in the grid to make the metrics and chart drill down to that test runner only.

Usage

Logging Tab

The logging widget shows all logging captured during execution, both system and scenario generated.

Logging can be viewed in tail mode to see the most recent log entries at the top.

Logging

Filters can be applied to narrow down the logging displayed by using the various filter controls across the top.

Output Files Tab

If you scenario captures output files (e.g. screenshots), these will be available on the Files tab. To capture only a sampling of the output files from your test, update Edit Configuration action => Advanced Options => Capture All Output by unchecking it. The method by which to capture output files is scenario type specific. See the documentation for the specific scenario type for more details (e.g. Selenium, PhantomJS, etc).

Output Files

Locations Tab

The locations map shows where the test runners utilized to execute your test are located around the globe. If you hover over any region you can see the peak memory and CPU utilization across all the test runners within that region.

Locations

Configuration Tab

See the exact test configuration that was used when running this test.

Test Configuration

Scenario Tab

See the exact scenario (including uploaded files, code, scripts, settings, etc) that was used when running this test.

Test Scenario