Ludicrous Speed: WordPress Caching with Redis

When I first started hosting my own sites, I had no idea what caching was or why it was important.  Then I wrote a couple of popular blog posts, and my server crashed.

Fast forward a few years, and I’m running a few different websites on a few different servers.  Some get a steady stream of traffic; others get a huge spike now and again when a post is picked up by Hacker News.  In both cases, configuring your cache is a great way to keep things from failing.

I’m very happy to say that I’ve finally set up a hugely efficient front-end page cache for WordPress.  It doesn’t use a plugin. It doesn’t require hacking core. And it serves my website in 3 ms. 1

Thanks to a clever configuration of Nginx and Redis, WordPress has achieved … ludicrous speed!

WordPress

WordPress is my application of choice for just about everything.  It’s easy to set up, easy to extend, and best of all, it’s open source.

You’ll notice one thing I really care about is the fact that open source exists and helps train a new generation of software monkeys on how to build the Internet.  Fact is, if WordPress were closed source, I would not have the job I do today.

WordPress is built on top of a whole slew of technologies.  The data is stored in MySQL, and PHP is used to script both the retrieval of that data and its shoehorning into an HTML template.  Unfortunately, this many moving parts means requests can be somewhat slow.

For comparison to the 3 ms reference above, my site takes about 4 (four) seconds to render when it’s not cached.

That is a huge difference between the cached and non-cached performance – 3 ms versus 4000 ms. WordPress is fast, but dynamic page generation is only as fast as the server powering it.  Even on the most streamlined machine, rendering a page dynamically will take time.

Nginx

Nginx is an open-source, event-driven web server.  I use it on all of my servers because it’s incredibly more efficient than Apache for handling multiple connections.  It’s fast, doesn’t use too much memory, and can be used to proxy requests to PHP or even other handlers like Node and Ruby.

Since we’re working with WordPress, we’re also working with PHP.  For reference, a pretty stock Nginx configuration file looks something like this:

Redis

Redis is an open source, in-memory, key-value database server – meaning it’s an optimized hashtable for your system.  If you’ve ever heard of Memcached, Redis is similar.  You can store anything you want – in memory – and retrieve it crazy fast.

Unlike Memcached, though, Redis is fairly easy to install and configure. 2

First things first, you’ll need to install Redis.  There are some stock instructions available on the Redis website.  There are also some handy, albeit slightly outdated, tutorials lying around the Internet. I used the Centos 6 tutorial as a base for my configuration, even though I run Ubuntu on my server.

Feel free to use whatever installation instructions, tools, or configuration you want (Debian users can just run apt-get install redis-server), just get Redis installed.

To let PHP (and thus, WordPress) interact with Redis, we use a library called Predis. You can grab the entire library if you want, or you can use a really clean all-in-one-file version. This version is the predis.php that will be referenced in the cache file in the next section:

Magic

In a nutshell, I put a single PHP script in front of WordPress on my server.  This script interfaces with Redis directly to handle the page cache:

  1.  If this isn’t a postback (form submission), the visitor isn’t logged in (based on the presence of a WordPress cookie), and Redis has an existing cached copy of this URL request, return the cache
  2. If this is a postback (form submission) or the request url is postfixed with ?r=y, flush the cache of the page and load WordPress
  3. If the visitor is logged in (based on the presence of a WordPress cookie) and the url is postfixed with ?c=y, flush the cache for the entire site and load WordPress
  4. If the visitor is logged in, load WordPress

The script we load in front of WordPress is fairly simple.  It checks the above criteria and either pulls data from Redis or loads WordPress:

Since we’re running this script before loading WordPress we skip all of the overhead associated with firing up MySQL and loading up the entire WordPress environment.  Instead, we return whatever HTML we rendered beforehand and respond to new visitor requests faster than without a cache.

But, if certain requirements are met, we still load WordPress like normal.

The biggest trick here is to get our script loaded before WordPress without hacking any core WordPress files.  Thanks to Nginx, we can set up an alias for the regular index.php file.  This modified Nginx configuration file will load wp-index-redis.php instead of index.php whenever a visitor requests it.  Also, this configuration will rewrite WordPress requests through wp-index-redis.php so rewritten page/post/queries will pass through the cache as well:

The one thing that needs to skip the cache, however, is the entire /wp-admin directory. If you pay close attention to the configuration file above, you’ll see that we’re allowing admin requests to pass back to the regular index.php file.

Summary

When a request hits my site, Nginx passes it directly to a single PHP script. This script checks an in-memory cache to see if we can load a saved version and, if so, loads it. If we need to refresh things, the script passed the request to WordPress and caches the result for future reference.

This is all accomplished through some clever rewriting of Nginx’s configuration such that we don’t need to hack WordPress directly. Now, visitors get lightning-fast responses to requests, and WordPress core can continue to update without any fear of overwriting cache-specific changes.

Check out the source of my Jumping Duck Media website to see just how quickly the cache returns a response.

On any given request, the cache returns in 3-10ms.

On any given request, the cache returns in 3-10ms.

Notes:

  1. Yes, you read that right. Three milliseconds. This is exclusive of the time it takes to perform a DNS lookup or actually deliver the page. But it is proof that the bottleneck is no longer the server itself.
  2. I have been working with professionally on the web for over 7 years and have not once been able to configure Memcached properly.  Redis, on the other hand, I have had 0 issues with. If Redis confuses you, feel free to swap out my advice with Memcached and keep on trucking.

Comments

  1. Michael says

    Hi,
    Thanks for the detailed post.
    Just a quick question: have your considered using accelerators like Varnish instead? And if you did, what’s the pros and cons?

    • says

      I avoid Varnish because it’s more often than not used as a bandaid to fix an otherwise poorly-configured server. That said, there’s no reason not to use it … I just feel that it’s overused and haven’t pursued it myself.

      • Alex says

        The solution here (full-page caching on the application level) seems like an unconventional mix. I’d be interested to see what the penalty is for performing the caching logic in PHP versus something purpose-made like Varnish

        • says

          It’s unconventional, yes, but I tried it for a couple of reasons. Firstly, I already had Redis configured on my test box so there was little to set up. Second, I am in the process of implementing a WordPress object cache that uses Redis (vs Memcached) for storage. Keeping both the object cache and page cache in the same tool is my attempt to maintain a smaller stack in general.

      • says

        The whole point of HTTP is to be architected such that intermediate proxies such as Varnish are transparent accelerators.

        “I just feel it’s overused.” — e.g. You wanted to use nginx and Redis, and needed to justify away from the simple solution.

        It should be a smell that instead of doing full-page caching in Varnish, you’ve instead rolled your own full-page application layer caching. Varnish handles POSTs, it handles logins (configured via VCL). This solution is a hack. I applaud your performance, but it’s rather nonsensical.

        Furthermore, memcached is not difficult to run — it’s a single daemon that requires less setup than Redis! And no persistent storage.

        Final note: “Redis is an open source flat file database server – meaning it’s an optimized key value store for your system.” — this is somewhat nonsense. Redis is a fast datastructure-oriented KV store that happens to be able to persist state to disk, but is by no means a ‘flat file database server’ and I’m not sure how that implies that it’s an optimized KV store.

          • says

            Actually… nothing says “troll” to me quite like a comment reply that’s posted nearly a year later by someone who signs it with nothing but a first name, and clearly has no comprehension of what the original well-structured comment even meant.

    • says

      I know a web host that uses this type of setup using redis and nginx. It is gatsby dot im.

      But for my personal site, I like nginx cache as it is simpler and involves fewer technology.

  2. says

    I’m still unclear as to what is difficult about “yum install memcached php-pecl-memcache” (or, “apt-get install memcached php5-memcache” for you Debian users). I’m totally willing to accept that you’ve had problems with Memcached, I just have never run into them myself, so I’m very curious. Were you trying to install it on an IIS server or something?

    • says

      When I was first learning Memcached, the version installed via yum was incompatible with the version of PHP I had running. In order to fix it, I had to build from source and … that went far less than swimmingly. Also, as you pointed out, I often work on IIS boxes as well, and Redis plays much nicer on Windows Server than Memcached (though I have seen both implemented in the same way). Like I say time and again, Redis is an alternative as both it and Memcached can serve the same purpose on the server. I just happen to be much more comfortable with Redis.

  3. says

    To set up Varnish for WordPress is almost trivial. I assume it is the same for nginx’s reverse caching. This yields the question: why bother with Redis ? Varnish does full-page caching out of the box and it actually excels at this task. Your claim that Varnish is “more often than not used as a bandaid to fix an otherwise poorly-configured server” may be right but it doesn’t imply that it should not be used when it’s obviously the best tool for the job.

    • says

      Not saying it shouldn’t be used, just that it’s often used too quickly to solve what might be the wrong problem. The implementation I’m working on is a blend of front-end page caching and back-end object caching. This article was merely addressing the front-end part of the cache.

  4. says

    Personally I’d use a more standard reverse proxy cache but this is an interesting alternative. I’m curious to see how it stacks up against varnish and nginx’s caching.

    Now that you’ve improved your backend performance you should look at your front end issues. WP Minify should be able to help with your 21 stylesheets and 25 JS files.

    • says

      I’m in the process of building a new theme for the front end. Multiple CSS and JS files is already taken care of, just not yet released. Planning a future post on how I streamline that development using Grunt, actually.

      • says

        You might also want to try profiling your plugins. I thought about it some more and, unless you’re on a really slow shared hosting environment, no WP post should take 4 seconds to load. First think I’d look at is the Twitter plugin. They are notorious for slow synchronous loading, even with their own caching.

        • Israel says

          What Dan said. I try to keep WP under 75ms on small servers using a combo of object caching, good DB query strategy, and banning/fixing inefficient plugins. Typically WordPress will only hit the DB on cache expirations if you get things set up well.

          Between that and proper use of Varnish we can run up to 10k requests per second at around 0.5-1ms per request – if we ever have that much bandwidth to push content through.

          I’ll also echo what others have said – if you skip PHP and have nginx go straight to Redis or Memcache, you’ll have a huge improvement. Here are a couple links to their nginx modules, respectively:

          http://wiki.nginx.org/HttpRedis
          http://wiki.nginx.org/HttpMemcachedModule

          And here are some pretty diagrams that apply equally to both since they’re just being used as key-value stores: http://www.igvita.com/2008/02/11/nginx-and-memcached-a-400-boost/

  5. chris says

    Hi Alex,

    Have you used this alongside the Photon Image caching feature in Jetpack? If not, do you think there is the possibility of a conflict?

  6. says

    You are still invoking PHP when you don’t actually need to. – You can cut this speed down and load even further by interfacing with Redis directly from Nginx :)

  7. says

    There are two memcached extensions for PHP — ext/memcache and ext/memcached. Last time I checked, ext/memcache was easier to build on Windows.

    Since you seem to be on Windows more often, I hope you’re aware of http://downloads.php.net/pierre/ which allows you to skip the compile step since it offers all the dlls you need. On your flavour of Linux, there should be also more up to date repositories with PHP and extensions so you don’t have to go through compilation to try out a new feature.

    I also second the question how the memcache daemon is possibly harder to configure than redis — not that either is hard, the simplicity of memcache seems to outshine many daemons I use on a daily basis.

    Anyway — lbnl, nginx also has a cache which works with memcache directly:
    http://wiki.nginx.org/HttpMemcachedModule

    This may or may not be available in your distribution though and may require you to compile nginx from sources.

    But the advantage: no need to even call PHP to deliver a page which should give you the biggest boost there is. I think this is similar to what the previous comment suggested using Redis.

    There are also countless of tutorials how to set this up with WordPress so e.g. a post update clears the cache or the frontpage, etc. from the cache.

    • says

      True. I would argue to use a responsive design over a separate mobile them anyway. Buy if you do have a mobile theme, you could use Nginx to detect mobile and swap where appropriate.

        • says

          The logic for caching here should be above WordPress – i.e. not in the mobile theme at all. WordPress has to start up before the theme is loaded (so that WordPress’ API is available) and by then, you’ve lost what benefits you were gaining from caching. The page cache layer should be before WordPress, plugins, or the theme ever load.

    • says

      Once I’m through experimenting, I’ll probably script a Vagrant VM to set this all up so it will be easier for others to play with.

      • says

        If you haven’t you should add page modspeed to your Nginx server. It will reduce the number of HTTP requests for CSS/javascript automatically.

  8. Phil says

    I’m actually amazed at some of the critiques I just read in the comments. C’mon people, the man implemented a solution that allowed his stack to skip a whole bunch of SQL commands. The crime is that he rolled his own, instead of using Varnish or some other reverse proxy?

    This isn’t hard. In fact, if you are even the tiniest bit of a coder, you can do what Eric did in less than a day. I did almost exactly the same thing on my company’s blog (http://www.collegeworks.com/blog/ – my bottleneck is the second-to-lowest-price Amazon EC2 instance we use) except for the Ngenix part, which isn’t really the point of the article anyway. It took my about 3 hours to write my cache controller and a plugin to allow WP to flush my Redis cache on all the appropriate WP admin tasks. Again, this isn’t hard.

    • woozyking says

      This is a free world, everyone’s allowed to use different solutions, whether they’re conventional or not.

      I can tell the author of this article is a writer, but he does not really understand Redis as @Xorlev mentioned. Redis is simply an in memory key-value store that provides more practical and efficient (if applied correctly) data structures (than Memcached, APC) for various use cases (for instance: http://instagram-engineering.tumblr.com/post/12202313862/storing-hundreds-of-millions-of-simple-key-value-pairs or http://code.flickr.net/2013/03/26/using-redis-as-a-secondary-index-for-mysql/). Redis is not new, it’s around for a while now. See http://www.slideshare.net/phpguru/redis-101-10043219 for a good intro for Redis and Redis in PHP.

      I personally like this hack when I first saw it from http://www.jimwestergren.com/wordpress-with-redis-as-a-frontend-cache/ (largely because I love Redis) but did not adopt it for production environment because it just needs more polishing (maybe implement an interface in WordPress plugin like W3 Total Cache for an easier setup than this?).

      • says

        Actually, I understand Redis quite well. But the point of the article wasn’t to explain what Redis is, what it does, or how it stores data.

        From the point of view of data storage, any NoSQL implementation is fairly equivalent – it’s the fact that Redis stores data in memory rather than on disk that makes it fast. Redis does have the advantage of granular data types (something both Memcached and traditional flat-file – on disk – systems lack).

      • Phil says

        Your right, everyone is entitled to their opinion. That is why I did a double take on the critiques.

        Redis is a solid technology, but I guess the real issue for some here is that it is way lower in the stack than something like Varnish. That means you need to get your hands a little dirty to utilize it effectively in your solution. Some guys have done that, and if the results are good, what’s the harm?

        I personally don’t use any of the WordPress+Redis solutions that I have seen others write. I didn’t like Jim Westergren’s solution because it required Predis, and I prefer using phpredis for my PHP+Redis solutions. Whoops, I expressed an opinion, I guess I better get ready for the criticism :). Here’s another opinion. I prefer a Redis+phpredis solution that utilizes igbinary in specific instances where I choose to serialize things more complex than strings.

        Maybe easy setup and config are more important to others than they are me to me. That’s OK with me. FWIW, I am willing to “suffer” a more complicated setup in order to get the results that I am looking for, which are sometimes very specific to me. I love being able to cache a large object in memory, and I use that technique all the time. Since I have Redis running for that, it is simple for me to take it further and cache other things, including generated markup all the way to full pages. So I agree with the author of this blog post that WordPress benefits from a front end cache. Not that Varnish isn’t appropriate, of course it is. But apples and oranges. YMMV.

  9. says

    Are there any issues when running suphp? Normally most php caching solutions like memcache or APC cannot run with suphp.

    • says

      I have zero experience with suPHP, so I can’t say for sure. But from the looks of things, it’s an Apache module … so you wouldn’t be using it with Nginx in the first place.

      • says

        Why not?

        I run suPHP for the security, and nginx for the protection against ddos and speed boost. It works well and do this on multiple machines. I’d never run a machine without suphp TBH. It’s saved my tushy too many times to count.

        No worries on not knowing the details. I was just curious. :)

        • says

          Every piece of documentation I can find says suPHP runs either as an Apache module or as an add-on to Apache’s mod_cgi. The only reference I can find regarding Nginx and suPHP is using it as a proxy to Apache.

          I don’t run Apache on any of my boxes, so I’ve never bothered to pick up suPHP. If you can run it without Apache, I’d love to see a blog post on the topic so I can give it a try.

  10. says

    Have found an issue using the “Contact Form 7″ plugin. I have to add a condition to not cache the page the contact form is on like your feed example above.

    Is there a way to skip the cache on any POST regardless if its via regular comment post or a JSON post?

    Thanks for putting this up by the way, the speed from this combo IS ridiculous!

    • says

      Yes. See the line that sets $submit? It’s currently checking just the cache control header to see whether or not we have a comment submission. You could just as easily check to see if the $_POST global is non-empty:

      (($_SERVER['HTTP_CACHE_CONTROL'] == 'max-age=0' || ! empty( $_POST ) ) ? $submit = 1 : $submit = 0);
  11. says

    Setting up varnish takes sometimes for newbie like me :D. I am using CentminMod for CentOS for easier nginx setup and I configured WP-FFPC with memcached. My blog loads faster … pretty happy with nginx ^_^

  12. says

    Another question I have: w3 Total cache does a lot beyond caching, but DB caching and expiration settings are good, are they not? Should I use that with the above tutorial or is it not needed?

    • says

      It really depends on your goals. My Redis system is specifically for front-end page caching. If you need object caching (which speeds up the back end) I would recommend using a more fully-fleshed-out system. As-is, the above code was merely a prototype and won’t fit every use case.

  13. John Elekman says

    Hi Eric, great solution. Just a quick question. I have noticed a couple of issues with comment form. From time to time the comments form tend to load cached details of another user and so exposing somebody else details including email. Is there a way to tell this setup not to cache form content?

    • says

      There would be. Keep in mind that this solution was an experiment and this particular caching issue is one of the few known bugs in the implementation. I don’t have a fix for it as of yet, but if you devise one I’ll gladly update the post.

      • Phil says

        I looked at this a while ago. I couldn’t think of anything reliable, except to add a filter on the Word Press comment_post_redirect hook. I could then insert an additional query string arg in the URI that Word Press redirects to when a comment is submitted. Seeing my specific arg in the URI, I know not to retrieve that page from cache.

        Of course, once I was at the point to be using plugins to work with caching, I went ahead and added cache management functions for various Word Press events (i.e., wp_insert_comment, transition_comment_status, save_post, etc.) so that pages are deleted from cache automatically.

        • John Elekman says

          Hi Phil,

          Do you have details about your solution. I really want to use redis but because of the comments form caching issues and exposing other people’s email address, am staying away.

          Could you kindly post DETAILS of your solution?

          • says

            I would definitely encourage you both to continue this discussion elsewhere. Comments on my blog post aren’t necessarily the best venue for an extended discussion of alternate code implementations. That said, I’d love to see a link to such discussion because I’m very interested as well.

          • Phil says

            John, the code I use at https://github.com/mrboson/php-redis-controller so go ahead and take a look. It’s functionally very similar to what Eric did here, but I wrapped all the magic into a class. There’s a sample.php which shows how it works for any PHP site. There’s a wordpress-sample.php showing how I use it with WordPress.

            Feel free to comment or leave questions in the github repo.

  14. says

    Great post, followed the instructions and my wordpress site is blazing fast.

    However, I do have a question. For mobile devices we use a different theme. How would I go about implementing this in Redis? I’m guessing that when you set the cache key in the database, you put some sort of extra field, indicating it is the mobile cache. However, I’m not sure how to adapt the hset function.

    I solved this problem temporarily with a function that checks whether the user agent is mobile and if yes, do not set the cache. But that off course is not ideal.

    • says

      I would recommend merging your themes to make the regular site theme responsive rather than using any kind of server-side logic to switch out the theme. Logic running on the server means you’re not switching from the cache … so it will always be slower.

    • says

      A little unfair to call this a copy of something I’d never heard about. But it’s nice to see someone else is doing the same kind of work.

      As for performance – make sure you’re logged out. If you’re logged in, you bypass the cache entirely. As for Redis’ performance on your box, I can’t speak to that. What I can tell you is that the system above (using a refined version of the code posted on GitHub (https://github.com/ericmann/wordpress-redis-backend) is just as fast as the other caching systems for WP (i.e. Memcached).

    • says

      Depends on your objective. The Nginx microcache is a great system as well, but there are advantages and disadvantages to using each system.

Trackbacks

Leave a Reply