Developing With Solr Integration For Bookdonors

Project Overview

Digital Six was delighted to design and develop Bookdonor’s new website using Magento and a number of integrations. The company has a huge inventory – with over 250,000 products currently in stock and plans to dramatically increase this to 400,000 in the next 12 months. The main challenge of this project was making sure that the high volume of product didn’t slow down the website’s loading time. Slow loading websites result in a poor user experience which affects conversion rates. It can also affect the website’s rankings in Google’s organic results, so it was vital that Bookdonor’s webpages were able to load quickly.

While we developed the website, we were able to detect the causes of the speed issues and find the best solutions. A powerful indexing system was one of the top priorities in order to provide a powerful search. Apache Solr was the ideal choice for allowing visitors to quickly search by Title, Author or ISBN and find the books they’re looking for.

Initial Set-up

From the start we were sure to advise our client that they would need a powerful server. Not all issues can be solved by raw server power, however. Often there are cases where you need to work smarter rather than just faster.

Working smarter involves profiling of the site as it is in use. This is particularly true for complex applications like Magento where event calls and plugins can make application execution flow unpredictable. We used NewRelic to profile the site during development, it’s invaluable for quickly identifying the source of any slowdowns.

Advanced Search Implementation With Apache Solr

The first problem we encountered was that the product lists and search were becoming progressively slower. This wasn’t so much a problem with Magento, the bottleneck was the complex queries required to retrieve the data stored in MySQL. To get around this problem, we switched to a separate indexing system – Apache Solr. Solr is a very powerful indexing system used by a number of high profile sites and services including Netflix, eBay, and NASA. It scales extremely well to large amounts of data, and has much better text-based search than Magento’s default text search. After setting Magento up to use Solr for search and category listings, the site became much faster even with hundreds of thousands of products.

Solving The Second Speed Issue

As yet more products were added we identified a second cause for slowdown. This time the slowdown was in the Magento code itself. Bookdonors not only has hundreds of thousands of books but hundreds of thousands of authors. As the list of authors grew, Magento’s standard handling of attributes became less and less efficient. Magento’s standard handling would probably have been efficient for attributes like Size or Colour that might have only had 5-50 options, but they were causing a lot of slowdowns when trying to load thousands and thousands of author names. We were able to improve performance considerably without altering functionality by changing how Magento cached the data.

We also found that many standard Magento plugins proved inefficient when using such large data sets. We had to rewrite one to utilise the Solr search index, and add caching to some others. In the end we were able to get all of the pages of the site loading quickly and efficiently.

Importing Data

Importing and updating hundreds of thousands of products can also strain the server. To prevent the site from slowing down under heavy load, or the site becoming unresponsive due to locked tables and so on, we had to build a custom import system to ensure that updates were imported as efficiently as possible. In total there were three sets of data imported – Categories, Products, and Product-Category Assignments. The plugin we used to interface Solr with Magento wasn’t particularly efficient at handling automated changes to large numbers of products, however. We ended up reverse-engineering the plugin and using its internal reindexing systems to reindex updated products in an efficient manner.

The Results

The implementation of these plugins have helped streamline processes including daily uploads of new products. The new and improved Bookdonors website now delivers an exceptional user experience across all devices. Visitors can now quickly and easily find the books they’re looking for with the advanced search tool, or enjoy browsing within the various categories. We look forward to seeing this fantastic enterprise grow from strength to strength. Take a look for yourself at bookdonors.co.uk