Measurements extension 0.2 released

The Measurements extension adds performance measurement features to tests developed with the MuseIDE. The 0.2 release adds several capabilities that are crucial to our goal of using Muse for load testing.

Periodic Measurement Collector

This plugin adds the ability to collect performance measurements at a regular interval. To turn this on for a test, add the Periodic Measurement Collector test suite plugin to the project. The collection interval is configurable in milliseconds. The default is 10 seconds.

Store Measurements To Local Filesystem

Once collected, the measurements need to go somewhere. The extension includes two additional test suite plugins for this purpose. The first is the Store Measurements To Local Filesystem plugin. Add it to the project to enable this feature:

When this plugin is added to the project and applied to a test, it will store all the measurements into a single file after each collection period. The files are named in sequence (ms0.json, ms1.json, … ms999.json) and the JSON format is easily readable by code and, to a lesser extent, by humans.

Each file contains metadata and measurements sections. In this example, the metadata includes the sequence number of the sampled measurements, which happens to match the numerical part of the filenames (0, 1, … 999). Each measurement in the measurements section contains a value and metadata describing the measurement. Most measurements will have (at least) two relevant metadata: the subject, which identifies what “thing” was being observed and the metric, which identifies what property of the “thing” was measured. Looking at the second measurement in the example below, we can see that the avg-dur of all-steps was 203 – in other words, the average duration of all steps completed during the measurement period was 203 (milliseconds). We can also see that the number of (measured) steps completed was 4.

This format was designed for flexibility and to be self-describing – making it very easy to parse and understand a large set of measurements with code. It allows for more metrics and subjects to be added dynamically while a test runs (or be removed, to optimize resource usage). You could even change the sampling period during a test. The open metadata allows tagging of measurements with additional information – in the case of load generation from multiple geographical regions, a region=us-east attribute would identify measurements from a specific load-generating location.

Store Measurements to CSV Files

The JSON format of sampled measurements shown above maximizes flexibility and completeness at the expense of immediate usability. So I also built a test suite plugin to store measurements as CSV files, which may be opened in a spreadsheet and plotted with a few mouse clicks – providing immediately useful analysis of the data. To add this feature to your tests, add the Store Measurements to CSV Files plugin to the project:

When this plugin is active, it will collect measurements and store them in CSV files by subject. There will be one file for each subject, one column in each file for each type of measurement collected for that subject (i.e. each metric) as well as a sequence column (0, 1, … 999).

From the previous JSON format, we could see that there were two metrics collected about the all-steps subject, the completed and avg-dur metrics. Opening the all-steps.csv file in a spreadsheet allows you to easily see those two metrics in columns and observe how those values change over time (the rows). For immediately-useful visualization, you can add a chart with just a few clicks.

Step Measurements Producer

All of the above assumes that there is something producing measurements. In my last post, I introduced a Step Duration Collector for recording (and storing) step durations for each test. That collector is particularly useful for collecting performance metrics on functional test suites (e.g. regression tests). However, for load testing we need a collector that can produce multiple measurements over time from a large number of tests running in parallel. To serve that purpose I created the Step Measurements Producer. Once added to a project, it is configured like this:

Similar to the Step Duration Collector, you can collect measurements on ALL steps (the default), or tagged steps using the Step tag parameter (I think you’ll find the second option much more useful). The producer currently supports average duration and completion count metrics in aggregate (for all steps). Look for more types of step measurements in the next release!

The extension can be added to your Muse projects via the Extensions button in the project navigator. This extension is open source – feel free to examine the source code on my GitHub.

Chris, Chief Engineer @ Web Performance


Add Your Comment

You must be logged in to post a comment.


Copyright © 2018 Web Performance, Inc.

A Durham web design company


For Prices Call (1) 919-845-7601 9AM-5PM EST

After hours? Prefer email? Just complete this form and we will get back to you as soon as possible with a quote. Please note: Technical support questions should be posted to our online support system.

About You
Product Needs

Request a Quote

Our experienced performance engineers have tuned hundreds of systems for companies large and small, and know just where to look, saving you time to market and money. We'll run your website through a complete performance evaluation, then tell you exactly how many users your site can support, including such important details as the effects of "the last mile." We'll also pinpoint potential problem areas and give you a full report detailing what needs to be fixed and where.

About You
Product Needs

Request a Quote

When a test isn't working it can hold up the entire development schedule. Let us help make your testers much more productive.

About You