Custom Metrics and Traces

Introduction

When tests run, all network calls (i.e. HTTP, websocket, TCP, etc) are instrumented to capture a standard set of metrics: latency, bandwidth, packets sent/received, connection success, and more. When writing a script it can be useful to capture your own metrics that are specific to your use case.

Example - Subscribe to Tick

Let’s start with a simple example where we want to capture the latency from subscribing to a symbol using the Testable WebSocket Sample Service and the first price update arriving.

const results = require('testable-utils').results;

const ws = new WebSocket("ws://sample.testable.io/streaming/websocket");

let sentSubscribe = 0;
ws.on('open', function open() {
  sentSubscribe = moment().valueOf();    
  ws.send('{ "subscribe": "IBM" }');
});

ws.on('message', function(data, flags) {
  results().timing('subscribe2tick', moment().valueOf() - sentSubscribe);
  ws.close();
});

The key line in this script is:

results().timing('subscribe2tick', moment().valueOf() - sentSubscribe);

This captures a timing metric, subscribe2tick, in the default namespace (User) with the value being the latency between “now” and the time the subscribe message was sent.

Metric Namespace

Metrics are grouped into namespaces. All the system generated metrics are in the Testable namespace. User generated metrics are in the User namespace by default, but a different namespace can be used.

Valid Metric Names

In order for a metric name to be valid it must:

  1. Not contain the following special characters: ‘-‘, ‘__’. All other valid UTF8 characters are allowed including spaces.
  2. Be 1-255 characters long

Metric names must be unique within the namespace. Users cannot write metrics to the Testable namespace.

Traces

During test execution we trace all connections details (including metrics, data sent, and data received) on some iterations. We try to capture at least one trace for each resource + response status combination each minute of your test. This will help you when analyzing the results to track down errors and better understand what went wrong.

Your script can also capture custom traces for any trace information you might find useful when analyzing results. Same sampling frequency applies (about one trace per resource + status + minute) unless result.forceReportTrace() is called.

const results = require('testable-utils').results;

results('IBM').setTraceStatus('Valid');
results('IBM').addTrace('FirstTick', {}, 'some data here');
results().setTraceStatus('200');
results().addTrace('Error', { header1: 'val1' }, 'error trace');

Metric Types

There are 3 types of metrics that can be captured:

  1. Timing: A timing metric will have the following aggregation functions computed: min, max, mean, count, variance, standard deviation, median (p50), p95, and p99. Additional percentiles can be configured as well.
  2. Counter: Keeps a counter with a running total. You can add/subtract from the counter from your script.
  3. Histogram: Keep a count for multiple buckets.
  4. Metered: A metered metric is intended to capture live utilization of a resource in a particular bucket. Testable uses metered metrics to capture CPU utilization, memory utilization, active connections, and bandwidth per test runner.

Metric Aggregation

Metrics are aggregated on the following dimensions:

  1. Execution: Test execution wide metric aggregation.
  2. Region: Tests can execute in multiple regions. Metrics aggregation is done per region as well as across all regions.
  3. Resource: Every time you make a network request you are accessing a Resource. We associate metrics with these resource labels. For example GET http://google.com or ws://sample.testable.io/real-time/websocket. For custom metrics you can optionally specify a resource which can be any valid string less than 256 characters.
  4. Interval: Test execution is broken into 10 second intervals. Metric aggregation is applied for each interval.

The test results page provides the UI to graph the relevant aggregations (any combination of the above 4 dimensions is supported).

Timing Metrics

To capture a timing in your script:

result().timing({ namespace: 'User', name: 'App Init Time', val: 100, units: 'ms' });

For timings, the following aggregation functions are computed by default: min, max, mean, count, p50, p95, and p99. The set of percentiles can be changed when creating a load configuration.

The units parameter defaults to ms if left blank for timings.

Counter Metrics

A counter keeps a running total for that metric.

result().counter({ namespace: 'User', name: 'My Custom Counter', val: 1, units: 'requests' });

The units parameter is required for counters and has no default.

Histogram Metrics

Histograms can be useful when you want to keep a count for an unknown number of buckets and keep them grouped together. For example, a histogram is useful for tracking HTTP response codes:

result().histogram({ namespace: 'User', name: 'HTTP Response Codes', key: '200', val: 1 });

The bucket can be any valid string that is less than 256 characters. If the value to increment is not provided, it defaults to 1.

Metered Metrics

A metered metric is intended to capture live utilization of a resource in a particular bucket. Testable uses metered metrics to capture CPU utilization, memory utilization, active connections, and bandwidth per test runner.

Testable will calculate the peak value across all buckets in each 10 second time interval during the test. For example, this can be used to observe the test runner with the highest CPU utilization.

For example to use a metered metric to track browser memory usage:

result().metered({ namespace: 'User', name: 'Browser Memory', key: os.hostname(), val: 3423432, units: 'bytes' });

The bucket can be any valid string that is less than 256 characters.

Custom Result Grouping

As noted in the previous section, every time you make a network request you are accessing a “Resource”.

Testable uses the following default format for the resource label:

[METHOD] [BASE_URL][FIRST_TWO_URI_PARTS][...]

This default behavior is intended to avoid a potential explosion of resource labels on which to aggregate metrics. A test can only contain 600 resources before it will be automatically stopped.

Some example URL to resource label default mappings:

  • GET https://www.google.com => GET https://www.google.com
  • POST https://myserver.com/some/path?param1=weee => POST https://myserver.com/some/path
  • GET https://myserver.com/some/long/path/stuff.png => GET https://myserver.com/some/long...

Changing Resource Labels

The default behavior for resource labels is not always desired. Testable provides an API to both override the default behavior or access it for creating new result metrics. In a Node.js script:

const results = require('testable-utils').results;

// returns 'GET https://www.google.com'
results.toResourceName('https://www.google.com', 'GET');

// to override the default behavior and instead use full URLs as resource labels
results.toResourceName = function(url, method) {
	return method + ' ' + url;
}

Attach Custom Metrics to Network Calls

It can sometimes be useful to attach or update a metric associated with a network call (e.g. HTTP GET) to ensure they get aggregated inline with the system captured metrics. The basic results([resource], [url]) API will use whatever resource label is provided or aggregate your metric into the “Overall Results” if none is specified.

There are two ways to accomplish this.

Option 1: Via hook (async friendly)

const results = require('testable-utils').results;
const axios = require('axios');

(async () => {
  // info: { method, url, resource, status, body (only if requested), response (http library API), request (http library API) }
  results.onResponse((result, info) => {
      result.histogram({ namespace: 'User', name: 'My Outcomes', key: 'Good', val: 1 });
    }, 
    // two optional options:
    //   -- body (whether to collect the body and include in info provided to the hook)
    //   -- once (whether to unregister your hook after the next network request)
    { body: true, once: true });

  // make http request; when it finishes the previously registered hook will be called and then unregistered because once = true
  await axios.get('http://sample.testable.io/stocks/IBM');
})();

Option 2: results.current (callback friendly)

Within an event listener for any network API (e.g. http, ws, socketio, etc) the current result can be accessed via results.current. For libraries that return promises, this approach won’t work because the result is already finalized by the time the next line of your code executes.

const results = require('testable-utils').results;
const request = require('request');

request.get('http://sample.testable.io/stocks/IBM', function (error, response, body) {
  if (response && body && body === 'good')
    results.current.histogram({ namespace: 'User', name: 'My Outcomes', key: 'Good', val: 1 });
  else {
    results.current.histogram({ namespace: 'User', name: 'My Outcomes', key: 'Error', val: 1 });
    results.current.setTraceStatus('Custom Error');
  }
});

Note that results.current is null when outside of a network call related event handler.

Use your own success criteria

By default any HTTP request that connects successfully and returns a status less than 400 is considered a successful request. For websockets and TCP sockets any socket that connects successfully is considered successful.

See the previous section for a detailed description of how you can affect the network call result in general.

Option 1: Via hook (async friendly)

const results = require('testable-utils').results;
const axios = require('axios');

(async () => {
  results.onResponse((result, info) => {
      if (info.status < 400 && info.body !== 'bad')
        result.markAsSuccess();
      else {
        result.markAsFailure();
        result.setTraceStatus('Custom Error');
      }
      result.histogram({ namespace: 'User', name: 'My Outcomes', key: 'Good', val: 1 });
    },
    { body: true, once: true });

  await axios.get('http://sample.testable.io/stocks/MSFT');
})();

Option 2: results.current (callback friendly)

const results = require('testable-utils').results;
const request = require('request');

request.get('http://sample.testable.io/stocks/MSFT', function (error, response, body) {
  if (response && response.statusCode < 400 && body !== 'bad')
    results.current.markAsSuccess();
  else {
    results.current.markAsFailure();
    results.current.setTraceStatus('Custom Error');
  }
});

Note that results.current is null when outside of a network call related event handler.

API

The results module provides all functionality related to metric capture.

const results = require('testable-utils').results;
results([resource], [url])

Returns a result object that can capture metrics for that resource/url. resource can optionally group a set of metrics together. Results can also be associated with a url, which does not affect aggregation, but will be available when downloading all results. For example the HTTP module groups the metrics it captures using a resource name of [METHOD] [URL] minus any query parameters after the ? (e.g. GET http://google.com) since it is useful to see all results for each URL aggregated together.

const result = results('my optional custom grouping');
result.timing(name, value[, units])
result.timing(options)

Capture a timing metric. Timings have various aggregators calculated like average, median, percentiles (95th, 99th), and standard deviation.

Namespace defaults to User if not specified. Units default to ms if not specified.

results().timing({ namespace: 'User', name: 'customTimerMs', val: 100, units: 'ms' });
results().timing('latencyMs', 100);
result.counter(name, value[, units])
result.counter(options)

Capture a counter metric. Counters are summed across your test execution as well as per 10 second interval.

Namespace defaults to User if not specified. Units default to empty if not specified.

results().counter({ namespace: 'User', name: 'beeCount', val: 2, units: 'bees' });
results().counter('myCounter', -3, 'units');
result.histogram(name, key, [value])
result.histogram(options)

Capture a bucket key/value into the histogram. The value defaults to 1 if not specified. During test execution you can view the count in each bucket, the total across all buckets, and the percent each bucket is of the total.

results().histogram({ namespace: 'User', name: 'outcome', key: 'failure', val: 1 });
results().histogram('httpMethod', 'GET', 1);
result.setTraceStatus(status[, statusMsg][, isError])

Set the status of the trace associated with this result. A trace must have a status to be valid. Optionally include a status message and an indication of whether or not this an error status.

results().setTraceStatus('404');
result.addTrace(type[, headers][, data])

Capture a trace packet. type can be any string less than 255 characters. headers must be an object if specified, and data is either a Buffer or a String.

Trace packets are visible on the test results page.

results('custom').addTrace('ConnectionOpened');
// ...
results('custom').addTrace('DataSent', { a: 'b' }, 'some data');
// ...
results('custom').addTrace('DataReceived', {}, 'some response');
// ...
results('custom').addTrace('ConnectionClosed');
result.forceReportTrace()

Overrides the sampling nature of traces and forces this trace to be captured and saved. Note that each test execution has a limited number of traces so if you force capture a trace on each test iteration, this limit can quickly be reached.

Force reporting is result specific so if you add a trace to multiple results, each one needs to have this method called separately.

results().forceReportTrace();
result.mergeTracePackets()

Merge trace packets of the same type within a single trace. Headers are merged (a header key added later overrides the same key from earlier) and data concatenated in the order the trace packets were added. This feature is used when capturing HTTP Data Sent and Data Received traces. Because HTTP responses are often chunked and compressed we need to recombine the packets before they can be processed. The first and last timestamp when a packet was added is maintained during the merge.

This feature is result specific so if you add a trace to multiple results, each one needs to have this method called separately.

results('GET http://weee.com').mergeTracePackets();
results('GET http://weee.com').addTrace('DataReceived', { a: 'b' }, 'weee1');
results('GET http://weee.com').addTrace('DataReceived', { c: 'd' }, 'weee2');
result.markAsFailure()

Overrides the default outcome detection to mark this request as a failure. See the set your own success criteria section for more details.

results().markAsFailure();
result.markAsSuccess()

Overrides the default outcome detection to mark this request as a success. See the set your own success criteria section for more details.

results().markAsSuccess();
result.setOutcome(outcome)

Overrides the default outcome detection. See the set your own success criteria section for more details.

results().setOutcome('failure');