Impact of Zend Optimizer on PHP Performance
Load Testing SugarCRM and the Zend Optimizer
Christopher L Merrill
©2007 Web Performance, Inc.
The performance of our reference PHP application
load (a default SugarCRM
installation) showed no measurable improvement after installing the
article measures the performance impact of the Zend
Optimizer on a real-world processor-bound PHP application
under load. Our measure of performance is
. We define that as the number of simultaneous users that the system can support while
meeting the specified performance criteria. The performance critera for
this test require that all pages load within 6 seconds and no errors
are encountered in the application.
answers the question "Why use the Zend Optimizer?"
with this statement: "The standard Zend run-time compiler used by PHP
is indeed extremely
fast, generating code that is usually 2 to 10 times faster. But an
application that uses the Zend Optimizer typically executes another 40%
to 100% faster."
We test this assertion using SugarCRM as
our reference application. According to the SugarCRM website
, SugarCRM is
"the market-leading comercial open source CRM application". SugarCRM is
written in PHP
. Based on previous testing, SugarCRM becomes quickly
processor-bound on a typical installation. Measurement showed that the
majority of the CPU load (~75%) was consumed in the Apache httpd
processes, which makes it a good
choice for testing the impact of a PHP optimizer
initial results were analyzed, we contacted Zend for assistance in
verifying that our configuration was correct. They suggested that we
explicitly set the values for some Optimizer settings in the php.ini
making these changes we retested and found no change in performance.
have invited Zend to suggest a readily-available application that can
demonstrate the claimed performance increase but, at the time of this
writing, had not received a response.
report is concerned with the performance of the system under load.
It does not make any claims about the performance of an idle
system. The measurements may be quite different when a single user
accesses the system.
The purpose of the report is not to
estimate the capacity of a
SugarCRM system or make any predictions about expected performance.
Every SugarCRM deployment will be different. The purpose is to measure
performance before and after installing the Zend Optimizer to determine
performance benefits of the optimizer on a real-world processor-bound
The analysis is based
the results of a series of load tests performed in our test lab.
skip the methodology, you can jump straight to the analysis
Zend Optimizer version 3.3.0a was
installed using the procedure on the Zend website
were run using PHP 5.1.6.
The server is running
version 4.5.1e of SugarCRM installed via the standard installer
from phpinfo() was captured for the baseline
The load-testing software is Web
Load Tester 3.5.5447. The Web Performance Server Agent (same
version) was installed on the server.
Hardware and OS
SugarCRM server is
running on a Dell Poweredge SC1420 with 2 x 2.8 GHz Xeon processors and
2GB of RAM. It has an 800MHz bus and 1MB of L2 Cache. All factory BIOS
settings were restored prior to testing. This includes the
hyperthreading support which is enabled. The floppy drive support in
the BIOS disabled (no floppy drive is installed). The server contains a
Western Digital 80G (WD800JD) SATA disk drive and uses the on-board 1G
network adapter. The operating system is the standard server install of
The load-generating engines
(3) are each
running on a Dell Poweredge 300 with 2 x 800MHz Pentium III processors
with 1G of RAM. Each engine is running the Web Performance Load Engine
3.5 Boot Disk version 3.5.5275. For these tests, one load engine would
have been sufficient.
The engines and
networked via a Dell 2324 Powerconnect switch. The server is connected
to a 1Gb port and the load engines are each connected to 100Mb ports.
load test controller (the Load Tester GUI) is running
on Windows XP SP2 on an IBM Thinkpad Z61m. It is connected via a
second switch on a 100Mb port. This has no impact
the results, since the controller bandwidth requirements are small, but
is mentioned here for completeness.
testcases selected for this load test mirror those selected for the SugarCRM Load
Please note that the user mix, think
times etc. may or may not reflect
your SugarCRM implementation. The rated user capacity of the system
is not likely to reflect any specific real-world deployment. To a large
extent, these choices are irrelevant to this test, since the goal is
not to evaluate the performance of SugarCRM, but rather determine the
impact of the optimizer on before-and-after results.
is available for examination -
version of Load Tester can view the testcases, load
configurations and detailed raw metrics. It cannot view the test
reports, which is why they are linked from this page (see references
The Load Configuration was created with these
- Four testcases, as described above
(virtual users) are distributed among the testcases as descibed
- VUs use the recorded think times, which
are typically ~2-6 seconds per page. Think times are larger for some
pages pages, such as adding a contact or a note, or reading a note.
VU is simulating a 5Mbps network connection (e.g. cable/DSL connection)
test starts with 20 users and adds 1 user every minute. VUs start at
intervals within each minute.
- 10 second sample period
test parameters were determined after preliminary tests were used to
approximate system capacity. The tests were terminated shortly after
the system had reached maximum capacity.
The database was restored to an identical state prior to
each test using the procedures outlined in the SugarCRM Load
Each test run
followed these steps:
- Restore database to pre-configured state with 100 users and 250 accounts, 1000 contacts and 3000 notes.
- Start the server monitoring agent
warm-up test (5 users running the View Note testcase for 3 minutes)
the load test
For the before (unoptimized) and after
(optimized) tests, 3 runs were completed under each configuration. The best
run of the 3 was selected as
representative of the configuration.
However, the results of the 3 showed minimal differences. For
completeness, they are included
in the test
Our analysis shows the
capacity of the system in baseline and optimized configurations to be
30. This analysis, along with a variety of charts is produced
automatically in Load Tester's
Load Test Report. Consult the User
section of the test reports for more details on this
analysis (see references
). The charts shown
below are extracted from that report section. The red line on the chart
maximum page duration at each user level measured during the test. The
location where that line crosses the duration
indicates the point where performance no longer met
the criteria. Had any errors occurred during the test, the yellow line
would depict the error rate and indicate when it crossed the error threshold
. Note that the
optimizer test encountered some page durations that were
higher than the baseline test, resulting in a different scale on the
Baseline User Capacity analysis
Zend Optimizer User
To confirm the findings from the User Capacity analysis,
we next look to the Page Duration charts. These show the minimum,
maximum and average page durations during the test. These charts are
section of the
load test reports. Note that the optimizer test encountered some page
durations that were higher than the baseline test, resulting in a
different scale on the
Note that capacity limit (30
users, which occurred at ~12 minutes), both systems showed similar page
durations - maximums of 5-6 seconds and averages ~2 seconds. As the
system was pushed past the rated capacity, you can see that the
optimized system actually showed higher maximum and average page
durations. This was consistent across 3 the test runs performed on each
Baseline page durations
Zend Optimizer page durations
As expected, the limiting factor in the application
performance is processor utilization. The charts below (extracted from
the Servers section of the load test reports) show several
processor-related measurements that were collected during the test
using the Advanced
module. The light-blue plot depicts the CPU
utilization (%). In both tests, the CPU is nearing 100% utilization
when the page duration threholds fail.
Optimizer processor measurements
the conditions tested, the reference PHP application (SugarCRM) showed
no measureable performance gain from the use of the Zend Optimizer.
- phpinfo() output
for baseline test
load test report for baseline test
output for Zend Optimizer
- complete load test report for the Zend Optimizer
- Load Tester repository
file (to be used with Web Performance Load Tester for further data
Feedback & Comments
Comments about this report may be posted at the company blog post.
v1.0 - 1st public release (5 Nov 07)
v1.1 - email cleanup (23 Jan 09)