Metrics Glossary

This page contains defintions for each standard metric that Testable captures during test execution. The name in parantheses is used in our API. On our website the name outside parantheses is used.

  • Active Concurrent Users (actualConcurrentClients) Counter: Number of concurrent users currently active within your test. Once the test completes this number will always be zero.

  • Bandwidth (bandwidth) Counter: Total bandwidth measured in bytes.

  • Breaking Point (breakingPoint) Counter: If your test hits a breaking point, this reports the number of concurrent users that were executing when the breaking point was hit.

  • Browser Used JS Heap Size (usedJSHeapSize) Counter: For browser based tests this indicates the peak size of the Javascript heap during the test. It is extracted from the browser via window.performance.memory.usedJSHeapSize.

  • Bytes Received (bytesReceived) Counter: Total bytes received in bytes.

  • Bytes Sent (bytesSent) Counter: Total bytes sent in bytes.

  • Commands (commands) Counter: For browser based tests, the number of automation commands executed across all browser sessions.

  • Command Streams (commandStreams) Counter: For browser based tests, the number of browser sessions initiated.

  • Connection Close (connectionCloseMs) Timing: The number of milliseconds from when a request starts until the connection is closed.

  • Connection Open (connectionOpenMs) Timing: The number of milliseconds from when a request starts until the connection is opened.

  • Count (count) Counter: The number of network requests that were executed during your test including HTTP requests, websockets, and TCP sockets opened.

  • CPU (cpu) Metered: The number of vCPUs used by test runners to run your test. This is available by test runner as well as aggregated across all test runners.

  • CPU % (cpuUtilization) Metered: The percent of all available vCPUs utilized while running your test. This is available by test runner as well as aggregated across all test runners.

  • DNS Lookup (dnsLookupMs) Timing: The number of milliseconds from when a request starts until the DNS lookup for the host name is resolved. This metric is not available for PhantomJS, SlimerJS, Gatling, and JMeter tests because the tools do not expose it.

  • Files (files) Counter: The number of files captured during the test.

  • HTTP Content Type (httpContentType) Histogram: A histogram that keeps track of how many responses of each content type is received, based on the Content-Type header. Only applicable for HTTP testing.

  • HTTP Method (httpMethod) Histogram: A histogram that keeps track of how many times each HTTP method is invoked. Only applicable for HTTP testing.

  • HTTP Response Code (httpResponseCode) Histogram: A histogram that keeps track of how many times each HTTP response code is received on a response. Only applicable for HTTP testing. Does not account for connection errors.

  • Images (images) Counter: The number of images captured during the test.

  • Iteration Time (iterationExecutedMs) Timing: The number of milliseconds to execute an entire iteration of your scenario.

  • Iterations Executed (iterationsExecuted) Counter: The number of iterations of your test scenario that were executed during the test.

  • JMeter Active Threads (jmeterConcurrents) Metered: The number of active JMeter threads across all instances/test runners.

  • JMeter Transactions (jmeterTxnCount) Counter: The number of JMeter transactions executed across all instances/test runners.

  • JMeter Transaction [METRIC] (jmeterTxn[METRIC]): The following metrics are also captured for JMeter transactions: connection open, response time, connection close, bandwidth, bytes sent, bytes received, response code, response status, outcome.

  • Max Concurrent Users (maxConcurrentClients) Counter: Peak number of simulated users that were ever executing concurrently during your test.

  • Memory (memory) Metered: The memory used on the test runners, in bytes, while running your test. This is available by test runner as well as aggregated across all test runners.

  • Memory % (memoryUtilization) Metered: The percent of available test runner memory utilized while running your test. This is available by test runner as well as aggregated across all test runners.

  • Outcome (outcome) Histogram: This histogram has two buckets: success and failure. Each network call is either a success or a failure. For HTTP requests, it is successful if a response code < 400 is received. For all other types of connections (e.g. websocket, TCP), success is if the connection opens successfully and closes without any error occuring.

  • Packets Received (dataPacketsReceived) Counter: Number of packets received. For example, on an HTTP chunked response, each chunk counts as a packet. On a websocket, every frame received counts as a packet. On a raw TCP socket, every time the socket data event is fired counts as a packet.

  • Packets Sent (dataPacketsSent) Counter: Number of packets sent. For example, on an HTTP chunked request, each chunk counts as a packet. On a websocket, every frame sent counts as a packet. On a raw TCP socket, every time the socket.write() method is called counts as a packet.

  • Page Load (pageLoadMs) Timing: Time in milliseconds to completely load a page and become interactive. Only applicable for Selenium, PhantomJS, and SlimerJS tests.

  • Page Requests (pageRequests) Observation: Number of resource requests required to load a page. Selenium only.

  • Page Weight (pageWeight) Observation: Total amount of data, in bytes, transferred over the network to completely load a page. Selenium only.

  • Peak Requests/Sec (peakRequestsPerSec) Counter: Peak number of requests per second measured during the test. A request can be any HTTP request, websocket, or TCP socket opened.

  • Requests/Sec (requestsPerSec) Counter: Average number of requests per second. A request can be any HTTP request, websocket, or TCP socket opened.

  • Response Code (responseCode): Histogram: Similar to HTTP Response Code except it has an Error bucket for connection failures and a Connected bucket for non-HTTP test scenarios.

  • Response Time (firstReceivedMs) Timing: The number of milliseconds from when a request starts until the first packet is received from the server. For an HTTP request this is the first packet (of potentially many with chunked encoding). For a websocket it is the time until the first frame AFTER the handshake is completed. For a raw TCP socket it is the time until the socket data event fires for the first time.

  • Speed Index (speedIndex) Observation: The Speed Index is the average time at which visible parts of the page are displayed. It is expressed in milliseconds.

  • Test Runner Bandwidth (testRunnerBandwidth) Metered: Peak throughput, in bytes/sec, required by the test runners to run your test. This is available by test runner as well as aggregated across all test runners.

  • Test Runner Bytes Received (testRunnerReceived) Metered: Peak throughput received, in bytes/sec, required by the test runner to run your test. This is available by test runner as well as aggregated across all test runners.

  • Test Runner Bytes Sent (testRunnerSent) Metered: Peak throughput sent, in bytes/sec, required by the test runner to run your test. This is available by test runner as well as aggregated across all test runners.

  • Test Runner Connections (testRunnerConnections) Metered: Number of TCP connections open on the test runner including all TCP stats (ESTABLISHED, TIME_WAIT, CLOSE_WAIT, FIN_WAIT2, etc). This is available by test runner as well as aggregated across all test runners.

  • Test Runner Connections Established (testRunnerConnectionsEstablished) Metered: Number of TCP connections open on the test runner whose current state is ESTABLISHED. This is available by test runner as well as aggregated across all test runners.

  • Time to First Byte (firstByte) Timing: Time in milliseconds until the first byte is received by the browser when loading a page. Selenium only.

  • Time to First Paint (firstPaint) Timing: Time in milliseconds until the first pixel is drawn on the screen by the browser when loading a page. This is measured using the standard web performance API. Selenium only.

  • Time to First Contentful Paint (firstContentfulPaint) Timing: Time in milliseconds until the browser renders the first DOM element on the screen when loading a page. This is measured using the standard web performance API. Selenium only.

  • Time to Interactive (firstInteractive) Timing: Time in milliseconds when the webpage is both visually rendered and capable of reliably responding to user input. Selenium only.

  • Throughput (throughput) Counter: Average throughput measured in bytes/sec.

  • Virtual Users Started (clientsStarted) Counter: Number of virtual users started during this test. This can be different than the reported Max Concurrent Users because not all virtual users necessarily run concurrently.

  • Virtual Users Finished (clientsFinished) Counter: Number of virtual users finished during this test. If the test is stopped early this number can be lower than the number of virtual users started.

  • Webdriver.io Assertions (wdioAssertions): Histogram: For Selenium Webdriver.io tests that are written with Mocha assertions this histogram tracks the number of assertions that were passed, skipped, and failed.