Magento Enterprise Edition (EE) 1.13 Benchmarking Guide
Contents
Customer Needs and Expectations
The focus of the Magento Enterprise Edition 1.13 release is performance and scalability. The benchmarking results presented here demonstrate that we have addressed the following concerns:
- The eCommerce landscape is changing, and merchants must provide customers with an online shopping experience that meets their performance expectations
- As merchants grow their businesses and increasingly larger enterprise merchants are adopting Magento, Magento Enterprise Edition must scale to handle increased traffic volume and larger catalogs
Magento's Plan
We identified the following areas for enhancement in Magento Enterprise Edition 1.13:
- Improve indexing for catalogs of all sizes
- Make re-indexing operations invisible to the shopper by executing them in the background
- Improve full page caching support
- Reduce page load times for key shopping flows (checkout)
- Enable merchants to serve heavier traffic volume without need to purchase additional hardware
What Magento Accomplished
Magento Enterprise Edition 1.13 delivers the following improvements over Magento Enterprise Edition 1.12:
- Incremental re-indexing has been introduced with Magento Enterprise Edition 1.13, and scenarios where full-re-indexing was required have been limited. This means that operations that previously took hours can now be completed in minutes
- 53% improvement in completion times for a full re-index with a 500,000 SKU catalog
- 35% improvement in "Place Order" performance
- 65% improvement in page load times across shopper flow pages that were tested.
- 33% improvement in orders per day that can be processed on the same hardware configuration as Magento Enterprise Edition 1.12
- 31% improvement in page views per day supported on the same hardware configuration on Magento Enterprise Edition 1.12
Multi-Node Deployment Topology
Before we introduce you to the benchmarks, an overview of our multi-node benchmarking facility is in order. We started with a basic multi-node cluster with load balancer and caching and separate DB node, and installed Apache. This is a familiar hardware configuration for the Magento developer community. Once provisioned, we installed both Magento Enterprise Edition 1.13 and Magento Enterprise Edition 1.12 on the cluster to compare their performance.
This is a representation of the multi-node Magento Enterprise Edition 1.12/1.13 installation used for performance testing. It consists of four physical nodes, three of which are virtualized.
We used HP ProLiant SL230s Gen8 servers with 8x16GB DDR3 and 2x Intel Xeon E5‐2660 CPUs for our benchmarking.
We used a physical disk of a usable 1.2 TB managed by a RAID10 controller. The load balancer we used is nginx.
The cluster resides in our Las Vegas data center, connected to the Internet via a gigabit connection. We tested the cluster using the popular Gatling suite from our engineering offices in Austin, TX.
Software Components
Scenarios
This section presents the merchant scenarios we simulated for our testing. These scenarios are based on real-world experience and industry standards with respect to shopper flows and catalog sizes.
Shopper Flows
We used real-world, established eCommerce metrics to simulate shopper flows.
- 94% of eCommerce shoppers visited a storefront but did not purchase any products
- 65% of shoppers added items to their carts but abandoned them
- Of the 6% who did purchase products (otherwise known as the conversion rate), half checked out as guests without signing in, and half signed into the storefront.
Catalog
The catalogs we simulated also reflect real-world, established eCommerce experience.
- We simulated small and medium-sized companies with a 50,000-item catalog with 27 product categories.
- We simulated large companies 500,000 items in the catalog with 2,000 categories.
- As many eCommerce sites offer different types of products, we specified physical (simple) products, virtual products, and downloadable products, with 60% of the products in each catalog being physical.
- We benchmarked each catalog on one website/storefront.
Target Merchant Profile
The simulated merchant profile we benchmarked against represents the profile of our enterprise customers. We established these metrics based on the day-to-day experience of large merchants operating a successful eCommerce business. Again, we used what we consider to be a typical hardware and software configuration.
- 50K visitors / day
- 1M Page views / day
- 18000 orders / day
- 3000 orders during peak 4 hours
- 1000 concurrent users
- Standard HW/SW configuration
Benchmarking Results
These benchmarks were generated during extensive testing of our multi-node configuration running Magento Enterprise Edition 1.13 described above. Our configuration was modeled after what a commercial hosting partner would put into production, and the results reflect accurate gains over Magento Enterprise Edition 1.12.
The duration of the test sessions is 72 seconds, which significantly stresses the multi-node configuration.
Incremental Re-indexing
In Magento Enterprise Edition 1.12, any change to a product would result in a full re-index. Magento Enterprise Edition 1.13 introduces a new feature--incremental re-indexing. With incremental re-indexing, only those items that were changed or added will be re-indexed, reducing the processing time to a fraction of what was required before.
Take the example of a merchant with a catalog containing 500,000 products. In Magento Enterprise Edition 1.12, any change to a product would result in a full re-index operation. In Magento Enterprise Edition 1.13, incremental re-indexing means the merchant will only re-index items that were changed. The test focused on measuring the improvements provided by the incremental re-indexing feature in Magento Enterprise Edition 1.13. The table below compares improvements to common admin actions, such as changing a product description, prices or inventory.
Full Re-indexing
As part of the benchmarking effort, we also measured the improvements in the full re-indexing feature on Magento Enterprise Edition 1.13, where indexing a 500,000-item catalog was 53% faster than Magento Enterprise Edition 1.12. Faster re-indexing means less load on the system and that changes to the catalog propagate faster to the storefront.
- 500,000 products
- 2,000 categories
- One store
- One catalog update during test run
Individual Re-indexers
The Magento Enterprise Edition 1.13 indexer component contains a number of individual indexers such as Product Flat Data and Product Price. When a Magento admin changes the price of a product, it is only necessary to execute the Product Price indexer for pricing changes to propagate to the frontend. The completion times of these individual indexers were measured in the benchmark environment for Magento Enterprise Edition 1.13 and Magento Enterprise Edition 1.12.
This section presents the results for full re-index completion times for the individual indexers in Magento Enterprise Edition 1.13 compared to Magento Enterprise Edition 1.12.
- URL Rewrite failed to run in Magento Enterprise Edition 1.12 but takes only 0.15 sec in Magento Enterprise Edition 1.13
- Catalog description: 500,000 products, 2000 categories, one storefront
Page Load Times
When Magento Enterprise Edition 1.12 and Magento Enterprise Edition 1.13 were compared, with both running on our multi-node benchmarking configuration, Magento Enterprise Edition 1.13 loaded pages 65% faster than Magento Enterprise Edition 1.12.
Guest checkout and registered user checkout are two flows that are crucial to storefront operation. This section presents the results of page load time measurements for these two flows.
Guest Checkout Flow Page Load Times
In the guest checkout flow pages, Magento Enterprise Edition 1.13 provides a substantial decrease in load times over its predecessor, most of the time more than twice as fast.
Registered Checkout Flow Page Load Times
The bar chart below presents the improvements in page load times for registered checkout flow.
Page Views and Orders
In addition to page load times, the benchmark also focused on measuring throughput improvements, particularly page views per day and orders per day.
During our testing, which simulated a storefront running at peak hours, EE 1.13 executed 33% more orders and 31% more page views than Magento Enterprise Edition 1.12 on the multi-node benchmarking configuration. Notably, Magento Enterprise Edition 1.13 served 47K pages during the test run (10 minutes).
- 10 minute session time
Observations
What we noted during the benchmarking tests:
- The MySQL instance did not show any significant signs of CPU or I/O load during the tests.
-
- The CPU was under 10% and no queries exceeded a 2-second threshold.
- Redis and Memcached instances did not exceed a CPU load of 10% during the tests.
- Web nodes showed high levels of CPU utilization under high load.
- We anticipate achieving stable scaling by adding additional web nodes to the cluster until the services themselves begin to degrade.
Conclusion
Magento Enterprise Edition 1.13 was engineered for performance--and clearly delivers on the goal as measured by important metrics.
- Indexing is improved to enable faster operations without impacting shopping experience. Merchant administrators can add and update products as needed while ensuring product URLs, promotions, navigational menus, and product search tools are always up to date.
- The checkout process is improved by reducing page load times for browsing and placing orders. Faster checkout can significantly improve your customers’ shopping experience and customer satisfaction, and potentially improve your conversion rate.
- Faster page load times means Magento Enterprise Edition 1.13 can support more page views per day and more orders per day, potentially increasing your conversion rate.
In addition, we have focused on providing these benefits without the need to upgrade existing hardware, improving your return on assets and investment.
Appendix
Page Load Times
This benchmark report presented analysis of the results produced by the Gatling load generator tool. In this appendix, you can see the actual results reported by Gatling. The results were obtained under the following test setup, which was designed to push Magento to its limit.| Concurrent session (users) | 1000 |
| Peak load duration | 10 minutes |
| Session duration | 72 seconds |
| CPU utilization | 95% |
CPU utilization is high because session duration was set to 72 seconds for the test. Session duration was set to a low value to increase the active load on Magento. A typical e-commerce site will see session durations of five minutes or more. Preliminary experiments by the benchmark team have confirmed that using a five-minute session duration reduces the CPU utilization to 50-60%.
| Magento EE v1.13 | ||||||||||||
| Requests | Total | OK | KO | Min | Max | Mean | Std Dev | 95th Pct | 99th Pct | Req/s | ||
| Global Information0 | 62161 | 62087 | 74 | 80 | 12910 | 1735 | 1869 | 5780 | 8060 | 75 | ||
| product page0 | 28573 | 28573 | 0 | 80 | 9390 | 709 | 996 | 2670 | 4920 | 34 | ||
| Click add to cart1 | 10458 | 10458 | 0 | 440 | 7170 | 1868 | 1139 | 4010 | 4980 | 13 | ||
| Click add to cart Redirect 12 | 10458 | 10458 | 0 | 960 | 12910 | 3816 | 2239 | 8050 | 9960 | 13 | ||
| click checkout3 | 3486 | 3486 | 0 | 1100 | 10800 | 3459 | 1965 | 7130 | 8970 | 4 | ||
| Registered Checkout:Login and Pick Address4 | 3366 | 3366 | 0 | 200 | 7380 | 1155 | 1482 | 4870 | 5900 | 4 | ||
| Registered Checkout:Login and Pick Address Redirect 15 | 3366 | 3366 | 0 | 700 | 11440 | 2605 | 1662 | 5640 | 7710 | 4 | ||
| Guest Checkout start6 | 120 | 120 | 0 | 360 | 2490 | 908 | 487 | 1860 | 2420 | 0 | ||
| Registered:Pick Billing Address 17 | 120 | 120 | 0 | 390 | 4240 | 1457 | 974 | 3510 | 4210 | 0 | ||
| Registered:Pick Billing Address 2-18 | 120 | 120 | 0 | 410 | 3690 | 1218 | 685 | 2460 | 3070 | 0 | ||
| Registered:Pick Billing Address 2-29 | 120 | 120 | 0 | 400 | 3600 | 1201 | 714 | 2700 | 3510 | 0 | ||
| Registered:Go to pick Shipping Method10 | 120 | 120 | 0 | 340 | 2900 | 1072 | 633 | 2330 | 2700 | 0 | ||
| Guest:Pick Billing Address 111 | 120 | 120 | 0 | 610 | 4620 | 1688 | 941 | 3580 | 4440 | 0 | ||
| Guest:Pick Billing Address 212 | 120 | 120 | 0 | 380 | 3530 | 1049 | 589 | 2050 | 2750 | 0 | ||
| Guest:Go to pick Shipping Method13 | 120 | 120 | 0 | 300 | 2080 | 796 | 401 | 1530 | 1960 | 0 | ||
| Set Shipping Method (Flatrate) 114 | 240 | 240 | 0 | 510 | 4830 | 1780 | 1066 | 3930 | 4720 | 0 | ||
| Set Shipping Method (Flatrate): Goto Payment15 | 240 | 240 | 0 | 300 | 3150 | 905 | 516 | 1940 | 2300 | 0 | ||
| Set Payment Method (Check/MO) 116 | 240 | 240 | 0 | 670 | 5290 | 1787 | 987 | 3660 | 4920 | 0 | ||
| Set Payment Method (Check/MO) 217 | 240 | 240 | 0 | 300 | 2760 | 904 | 518 | 2010 | 2440 | 0 | ||
| Set Payment Method (Check/MO) 318 | 220 | 220 | 0 | 370 | 8180 | 2696 | 1725 | 5810 | 7590 | 0 | ||
| /checkout/../success/19 | 240 | 240 | 0 | 250 | 2830 | 931 | 583 | 1970 | 2530 | 0 | ||
| /checkout/../success/ Redirect 120 | 74 | 0 | 74 | 1030 | 8350 | 2473 | 1620 | 5590 | 7310 | 0 | ||
| Magento EE v1.12 | ||||||||||||
| Requests | Total | OK | KO | Min | Max | Mean | Std Dev | 95th Pct | 99th Pct | Req/s | ||
| Global Information0 | 48044 | 47887 | 157 | 40 | 40580 | 4985 | 3290 | 10650 | 13870 | 58 | ||
| product page0 | 21327 | 21286 | 41 | 40 | 20040 | 4557 | 2820 | 9740 | 12350 | 26 | ||
| Click add to cart1 | 8340 | 8324 | 16 | 40 | 12130 | 4216 | 2217 | 7520 | 9000 | 10 | ||
| Click add to cart Redirect 12 | 8324 | 8320 | 4 | 40 | 16500 | 7022 | 3449 | 11780 | 13500 | 10 | ||
| click checkout3 | 2780 | 2779 | 1 | 920 | 12350 | 6311 | 2829 | 9760 | 10950 | 3 | ||
| Guest Checkout start4 | 90 | 90 | 0 | 390 | 3690 | 2103 | 795 | 3210 | 3410 | 0 | ||
| Registered Checkout:Login and Pick Address5 | 2690 | 2690 | 0 | 180 | 19120 | 3298 | 4984 | 15470 | 16940 | 3 | ||
| Guest:Pick Billing Address 16 | 90 | 90 | 0 | 600 | 7750 | 4348 | 1605 | 6430 | 7400 | 0 | ||
| Registered Checkout:Login and Pick Address Redirect 17 | 2690 | 2688 | 2 | 40 | 24360 | 5433 | 3350 | 11780 | 14060 | 3 | ||
| Guest:Pick Billing Address 28 | 90 | 90 | 0 | 520 | 8240 | 3412 | 1423 | 5480 | 6810 | 0 | ||
| Guest:Go to pick Shipping Method9 | 90 | 90 | 0 | 310 | 4780 | 2334 | 888 | 3300 | 4450 | 0 | ||
| Registered:Pick Billing Address 110 | 90 | 90 | 0 | 370 | 10090 | 3657 | 2453 | 8660 | 9600 | 0 | ||
| Registered:Pick Billing Address 2-111 | 90 | 90 | 0 | 540 | 17670 | 4148 | 2741 | 10220 | 12510 | 0 | ||
| Registered:Pick Billing Address 2-212 | 90 | 90 | 0 | 530 | 12640 | 3983 | 2411 | 8290 | 11590 | 0 | ||
| Registered:Go to pick Shipping Method13 | 90 | 90 | 0 | 360 | 6290 | 2546 | 1401 | 5050 | 5980 | 0 | ||
| Set Shipping Method (Flatrate) 114 | 180 | 180 | 0 | 450 | 7640 | 3926 | 1884 | 6550 | 7450 | 0 | ||
| Set Shipping Method (Flatrate): Goto Payment15 | 180 | 180 | 0 | 380 | 6290 | 2451 | 1256 | 4270 | 4830 | 0 | ||
| Set Payment Method (Check/MO) 116 | 180 | 180 | 0 | 570 | 8290 | 3923 | 1996 | 6460 | 7300 | 0 | ||
| Set Payment Method (Check/MO) 217 | 180 | 180 | 0 | 330 | 5230 | 2318 | 1238 | 4210 | 4810 | 0 | ||
| Set Payment Method (Check/MO) 318 | 180 | 180 | 0 | 940 | 40580 | 12673 | 7957 | 25910 | 29540 | 0 | ||
| /checkout/../success/19 | 180 | 180 | 0 | 230 | 3330 | 1437 | 854 | 2870 | 3150 | 0 | ||
| /checkout/../success/ Redirect 120 | 93 | 0 | 93 | 780 | 10470 | 5031 | 2251 | 7920 | 8880 | 0 | ||
Estimating the Number of Visitors
Gatling does not report the number of users that cycled through during the test. However, it is possible to estimate the number using the following formula: