One of the challenges in running a large load test is the orchestration of a large number of machines to generate load. This involves a series of steps:
Load Tester does that pretty effortlessly in EC2 – our customers as well as our own test engineers love not having to worry about those steps. It just works. As I evolve our next generation of testing tools, I am revisiting this problem and looking at solutions from a different angle. Last week, I decided to investigate some of the popular Linux orchestration tools (Chef, Puppet, Ansible, etc) to see if they would be suitable for the task of running a distributed load test.
Ansible immediately appealed to me for this task, primarily due to it’s agentless nature – it only needs SSH and Python on the client to manage it. This would mean that I can use any EC2 Linux image. I have recently played a little with Docker as well and came across the ultralight Alpine Linux distro, which struck me as perfect for running a load-generating agent.
I immediately ran into a problem with my first Ansible command – The EC2 Alpine image doesn’t have Python. Did I mention it is an ultralight distro? As it turns out, Ansible really only needs SSH. Python is needed for most tasks (which are performed with Alpine modules written in…Python), but thanks to the raw module, I was able to use Ansible to install Python and then all the rest of the Ansible modules will work.
Before doing anything else, I want to update Alpine with the latest packages. Alpine uses the apk package manager, so normally I would use Ansible’s apk module for this. But, with Python still MIA, I used the raw module instead. Note that the ONLY recommended use of the raw module is for systems that do not yet have Python on them. This Ansible task will get it done:
(note that all of these steps require sudo, so you either need to add a “become: yes” line to each task or add it before the tasks declaration, to become sudo for all tasks)
- name: Ensure updated raw: apk update
But it failed. I logged into the Alpine instance and tried the update manually. I found that the update fails with these errors:
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/main/x86_64/APKINDEX.tar.gz
fetch http://dl-cdn.alpinelinux.org/alpine/v3.7/community/x86_64/APKINDEX.tar.gz
fetch https://mcrute-build-artifacts.s3.us-west-2.amazonaws.com/alpine-packages/3.7/testing/x86_64/APKINDEX.tar.gz
ERROR: https://mcrute-build-artifacts.s3.us-west-2.amazonaws.com/alpine-packages/3.7/testing: Permission denied
WARNING: Ignoring APKINDEX.4dabf18d.tar.gz: No such file or directory
v3.7.0-151-gf417903f18 [http://dl-cdn.alpinelinux.org/alpine/v3.7/main]
v3.7.0-153-g3d6fac5d3f [http://dl-cdn.alpinelinux.org/alpine/v3.7/community]
1 errors; 9056 distinct packages available
The cause is an errant line in the /etc/apk/repositories file in this AMI (it references an inaccessible apk repository). Fortunately, this line is easily removed with a bit of grep and sed. I inserted another Ansible command before the update step, again using the raw module:
- name: Ensure apk repository settings are correct raw: if echo $(tail -n 1 /etc/apk/repositories) | grep -q "testing"; then sed -i '$ d' /etc/apk/repositories; fi
Yay! The system is updated, so now I can install Python. This is the last time I will use the raw module:
- name: Ensure Python and pip are installed raw: apk add python2 py2-pip
If I didn’t need Docker, then I could stop here. For our load-generating engines, I won’t need Docker, but I will for running Chronograf, which I will eventually use for viewing live metrics during a test. That is a post for another day. Setting up Docker is easy now, with Ansible at full strength, so let’s get to it:
Now that Python is present, I can use the apk Ansible module to install Docker:
- name: Ensure docker is installed apk: name: docker state: present
Knowing that it is present, I can start it with the service module:
- name: Ensure docker is running service: name: docker state: started
Finally, we can install the dependencies for the docker Ansible module. This is not strictly required, but future tasks, such running containers, is easier with the docker module. That module needs the docker-py Python package. Since I installed pip alongside Python in the previous step, I can use the Ansible pip module:
- name: Ensure docker-py is installed pip: name: docker-py
Done. This Alpine/EC2 image is now ready to run Docker containers. Enjoy!
Chris
When his dad brought home a Commodore PET computer, Chris was drawn into computers. 7 years later, after finishing his degree in Computer and Electrical Engineering at Purdue University, he found himself writing software for industrial control systems. His first foray into testing software resulted in an innovative control system for testing lubricants in automotive engines. The Internet grabbed his attention and he became one of the first Sun Certified Java Developers. His focus then locked on performance testing of websites. As Chief Engineer for Web Performance since 2001, Chris now spends his time turning real-world testing challenges into new features for the Load Tester product.