{"id":1627,"date":"2010-11-15T10:00:05","date_gmt":"2010-11-15T14:00:05","guid":{"rendered":"http:\/\/www.webperformanceinc.com\/load_testing\/blog\/?p=1627"},"modified":"2011-10-06T18:13:01","modified_gmt":"2011-10-06T22:13:01","slug":"drupal-database-scalability","status":"publish","type":"post","link":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/2010\/11\/drupal-database-scalability\/","title":{"rendered":"Drupal: Caching and Database Scalability"},"content":{"rendered":"<p><strong>Note<\/strong>: This is Part 4 of an ongoing series on Drupal performance and load testing.  If you haven&#8217;t already, read the <a href=\"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/2010\/09\/optimizing-drupal-introduction\/\">introduction<\/a>.<\/p>\n<h3>Summary<\/h3>\n<p>We measured Drupal&#8217;s performance with respect to database size, demonstrating flat performance regardless of the size of the database.\u00a0 We also got some good data demonstrating Drupal&#8217;s behavior with caching.<\/p>\n<h3>Procedure<\/h3>\n<p>We re-created our previous test platform: a stock Drupal installation on an Amazon Elastic Cloud m1.large instance with both the Alternative PHP Cache (APC) and Drupal&#8217;s built-in caching capabilities.\u00a0 In this test, however, instead of scaling the number of simultaneous users, we instead held the test at 400 users, but varied the amount of content in our database.<\/p>\n<p>We also used a different test scenario.\u00a0 As in the previous scenario, each user visited a single &#8220;hot&#8221; page, presumably a page that had been linked from another high-traffic site.\u00a0 However, each user then browsed to a different arbitrarily selected page, forcing Drupal, over time, to serve the entire contents of its database.\u00a0 In the graphs below include one line for each of the three pages.<\/p>\n<p>It should be noted that the machines in question run with 7.5 gigabytes of memory &#8212; theoretically enough to cache in RAM the entirety of the database we threw at it.<\/p>\n<p>We were looking for any sign that Drupal would struggle to service a large database.<\/p>\n<h3>Results:<\/h3>\n<p>The first test group simply adjusted the total size of the database in bytes:<\/p>\n<ul>\n<li>10,000 nodes, up to 100 comments per node &#8212; just over <strong>1 gigabyte<\/strong> of plain text content.<\/li>\n<li>50,000 nodes, up to 100 comments per node &#8212; a little shy of <strong>7 gigabytes<\/strong> of plain text content.<\/li>\n<\/ul>\n<p>To put this number into context, 7 gigabytes is equivalent to 15,000 full-length novels, larger than all but the most popular blogs and wikis, but certainly dwarfed by a multi-terabyte resource like wikipedia.<\/p>\n<p>The following is the graph of average page duration over time against the 10,000 node database.<\/p>\n<p><a href=\"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-content\/uploads\/2010\/10\/10000nodes_time_based_page_durations.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-1636\" title=\"10000nodes_time_based_page_durations\" src=\"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-content\/uploads\/2010\/10\/10000nodes_time_based_page_durations-300x177.png\" alt=\"10000nodes_time_based_page_durations\" width=\"300\" height=\"177\" srcset=\"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-content\/uploads\/2010\/10\/10000nodes_time_based_page_durations-300x177.png 300w, https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-content\/uploads\/2010\/10\/10000nodes_time_based_page_durations-500x296.png 500w, https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-content\/uploads\/2010\/10\/10000nodes_time_based_page_durations.png 745w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a>What&#8217;s happening here?\u00a0 The two pages with fixed URLs (the front page and headline article, which was large and therefore a little bit slow), both behave very consistently.\u00a0 But the &#8220;page&#8221; that was actually configured to dynamically crawl all of the stories in the database shows interesting behavior.\u00a0 For the first half of the test, page durations seem to increase gradually, and then quickly drop to one tenth of a second.\u00a0 This test features slightly over 20,000 test case completions, so the simplest explanation is that roughly halfway through our test we fully populated Drupal&#8217;s built-in cache and just started returning every page request from the cache.\u00a0 If you look very closely, the average page durations of the other two steps in the test case decrease very slightly as load on the server was reduced.<\/p>\n<p>The next graph shows average page duration over time against the 50,000 node database.\u00a0 (Ignore the swap in colors between the green and blue lines.\u00a0 I also don&#8217;t care that the scale on the y-axis has changed &#8212; this is caused by relative sizes of the spikes in the graph, which in this case seem to be nothing more than ordinary noise.)<\/p>\n<p><a href=\"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-content\/uploads\/2010\/10\/50000nodes.png\"><img loading=\"lazy\" decoding=\"async\" class=\"aligncenter size-medium wp-image-1639\" title=\"50000nodes\" src=\"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-content\/uploads\/2010\/10\/50000nodes-300x167.png\" alt=\"50000nodes\" width=\"300\" height=\"167\" srcset=\"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-content\/uploads\/2010\/10\/50000nodes-300x167.png 300w, https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-content\/uploads\/2010\/10\/50000nodes-500x279.png 500w, https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-content\/uploads\/2010\/10\/50000nodes.png 729w\" sizes=\"auto, (max-width: 300px) 100vw, 300px\" \/><\/a>This graph looks like a reverse of the previous graph.\u00a0 Why?\u00a0 Because at the beginning of this test Drupal is still using the cache from the previous test.\u00a0 Once it exhausts the first 10,000 table rows it must begin constructing pages again and this more than doubles the page duration.\u00a0 Even after the 20 minute mark of the test, average page durations are still a little lower than what we saw in the 10,000 node test, probably because some virtual users continue to revisit the cached pool.<\/p>\n<p>Technically, this is a mistake on my part &#8212; good science would seem to require that I clear the cache at the beginning of each test run.\u00a0 Nevertheless, I feel we&#8217;ve discovered what we were looking for in this test run: Drupal scales cleanly with increasing database size, and configuration details dominate scalability issues in a default Drupal installation.<\/p>\n<h3>What about the logical node count?<\/h3>\n<p>When talking about database scalability, there are actually two things that interest us: scalability by total number of bytes and scalability by the logical number of database rows.\u00a0 Drupal also performed flawlessly up to <strong>650,000<\/strong> nodes (but with no comments).\u00a0 There was no discernible performance degradation across this range.<\/p>\n<h3>Conclusion<\/h3>\n<p>We weren&#8217;t really expecting to discover any horrible scalability bugs in Drupal (or MySQL, for that matter), and we didn&#8217;t.\u00a0 In fact, there was no detectable performance penalty associated with increased database size.\u00a0 By any conventional measure, Drupal demonstrated excellent scalability.<\/p>\n<p>&#8212; Lane, engineer at Web Performance.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Note: This is Part 4 of an ongoing series on Drupal performance and load testing.  If you haven&#8217;t already, read the <a href=\"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/2010\/09\/optimizing-drupal-introduction\/\">introduction<\/a>.<br \/>\nSummary<br \/>\nWe measured Drupal&#8217;s performance with respect to database size, demonstrating flat performance regardless of the size of the database.\u00a0 We also got some good data demonstrating Drupal&#8217;s behavior with caching.<br \/>\nProcedure<br \/>\nWe re-created our previous test platform: a stock Drupal installation on an Amazon Elastic Cloud m1.large instance with both the Alternative PHP Cache (APC) and Drupal&#8217;s built-in caching capabilities.\u00a0 In this test, however, instead of scaling the number of simultaneous users, we instead held the test at 400 &hellip; <a href=\"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/2010\/11\/drupal-database-scalability\/\">Continue reading &raquo;<\/a><\/p>\n","protected":false},"author":53,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[40,131],"tags":[96,91,237,13,72],"class_list":["post-1627","post","type-post","status-publish","format-standard","hentry","category-case-studies","category-drupal-2","tag-caching","tag-drupal","tag-load-testing","tag-performance","tag-scalability"],"_links":{"self":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts\/1627","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/users\/53"}],"replies":[{"embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/comments?post=1627"}],"version-history":[{"count":34,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts\/1627\/revisions"}],"predecessor-version":[{"id":1632,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/posts\/1627\/revisions\/1632"}],"wp:attachment":[{"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/media?parent=1627"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/categories?post=1627"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.webperformance.com\/load-testing-tools\/blog\/wp-json\/wp\/v2\/tags?post=1627"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}