The Measurements extension adds performance measurement features to tests developed with the MuseIDE. The 0.2 release adds several capabilities that are crucial to our goal of using Muse for load testing.
This plugin adds the ability to collect performance measurements at a regular interval. To turn this on for a test, add the Periodic Measurement Collector test suite plugin to the project. The collection interval is configurable in milliseconds. The default is 10 seconds.
Once collected, the measurements need to go somewhere. The extension includes two additional test suite plugins for this purpose. The first is the Store Measurements To Local Filesystem plugin. Add it to the project to enable this feature:
When this plugin is added to the project and applied to a test, it will store all the measurements into a single file after each collection period. The files are named in sequence (ms0.json, ms1.json, … ms999.json) and the JSON format is easily readable by code and, to a lesser extent, by humans.
Each file contains metadata and measurements sections. In this example, the metadata includes the sequence number of the sampled measurements, which happens to match the numerical part of the filenames (0, 1, … 999). Each measurement in the measurements section contains a value and metadata describing the measurement. Most measurements will have (at least) two relevant metadata: the subject, which identifies what “thing” was being observed and the metric, which identifies what property of the “thing” was measured. Looking at the second measurement in the example below, we can see that the avg-dur of all-steps was 203 – in other words, the average duration of all steps completed during the measurement period was 203 (milliseconds). We can also see that the number of (measured) steps completed was 4.
This format was designed for flexibility and to be self-describing – making it very easy to parse and understand a large set of measurements with code. It allows for more metrics and subjects to be added dynamically while a test runs (or be removed, to optimize resource usage). You could even change the sampling period during a test. The open metadata allows tagging of measurements with additional information – in the case of load generation from multiple geographical regions, a region=us-east attribute would identify measurements from a specific load-generating location.
The JSON format of sampled measurements shown above maximizes flexibility and completeness at the expense of immediate usability. So I also built a test suite plugin to store measurements as CSV files, which may be opened in a spreadsheet and plotted with a few mouse clicks – providing immediately useful analysis of the data. To add this feature to your tests, add the Store Measurements to CSV Files plugin to the project:
When this plugin is active, it will collect measurements and store them in CSV files by subject. There will be one file for each subject, one column in each file for each type of measurement collected for that subject (i.e. each metric) as well as a sequence column (0, 1, … 999).
From the previous JSON format, we could see that there were two metrics collected about the all-steps subject, the completed and avg-dur metrics. Opening the all-steps.csv file in a spreadsheet allows you to easily see those two metrics in columns and observe how those values change over time (the rows). For immediately-useful visualization, you can add a chart with just a few clicks.
All of the above assumes that there is something producing measurements. In my last post, I introduced a Step Duration Collector for recording (and storing) step durations for each test. That collector is particularly useful for collecting performance metrics on functional test suites (e.g. regression tests). However, for load testing we need a collector that can produce multiple measurements over time from a large number of tests running in parallel. To serve that purpose I created the Step Measurements Producer. Once added to a project, it is configured like this:
Similar to the Step Duration Collector, you can collect measurements on ALL steps (the default), or tagged steps using the Step tag parameter (I think you’ll find the second option much more useful). The producer currently supports average duration and completion count metrics in aggregate (for all steps). Look for more types of step measurements in the next release!
The extension can be added to your Muse projects via the Extensions button in the project navigator. This extension is open source – feel free to examine the source code on my GitHub.
Chris, Chief Engineer @ Web Performance
When his dad brought home a Commodore PET computer, Chris was drawn into computers. 7 years later, after finishing his degree in Computer and Electrical Engineering at Purdue University, he found himself writing software for industrial control systems. His first foray into testing software resulted in an innovative control system for testing lubricants in automotive engines. The Internet grabbed his attention and he became one of the first Sun Certified Java Developers. His focus then locked on performance testing of websites. As Chief Engineer for Web Performance since 2001, Chris now spends his time turning real-world testing challenges into new features for the Load Tester product.