Drupal, Varnish and Edge side includes (ESI)

Caching in Drupal has leaped forward in the last few years with the maturity of various advanced caching strategies.

The drink of choice for the Axis12 team has been Varnish as it is a great reverse proxy that is highly configurable with a great community driving inovation.
Currently there are three ways to integrate varnish:

1. Use the Varnish module which provides …
2. Use Pressflow which is an optimized version of Drupal developed by Four Kitchens which has varnish support built in and deals with …
3. Use Drupal 7 which has built in varnish support

While all of these options are good they don't really deal with the biggest problem performance hit on your Drupal system - logged in users (That's a slight mistruth, the Drupal 7 option does address this slightly). That is because they essentially turn themselves off when a user logs in as they need to serve things specific to them dynamically such as personal home page and the facility to change passwords etc.

At Axis12 we are about to kick off a project that will tackle this problem through the use of Edge Side Include (or ESI) in the Drupal ESI module. ESI allows you to embed callback url's into your html that will be intepreted by Varnish and populated with the result from the callback. This allows more granular control of display including different refresh rates for each ESI.Here is the link the Drupal ESI module page

The rough plan so far……

Goal:

• Create a Edge Side Includes module that will support reverse proxies with this functionality.
• It will be built in a abstracted and pluggable fashion so both the reverse proxies and the Drupal caching layer are exchangeable

Initial Implementation:

Our initial effort is going be to look at Varnish as seems to be well adopted in Drupal, but will design with the idea pluggable extensions being added when needed.
Considerations:

1. If you only have one ESI call on a page this is very straight forward
2. We don't have a option 1 situation.
3. If you have multiple calls you could just do one call and use JavaScript to do DOM manipulation (again we can't due to accessibility on the site) the jquery (or equiv) can be passed back with the ESI data

Process flows:
Requests:

1. [path]esi.php[with arguments]
2. controller picks up request loads appropriate config
3. The config is chosen using settings admin UI but on save we write that information to both the db and to a settings/config file in a know location on disk, the esi.php is then able to read that and have all the information necessary to draw from the chosen cache store without having to bootstrap
4. a cache lookup is performed based url hash (plus salt or token or something/something)
5. If hit return the data
6. Fail invoke Frupal bootstrap and generate the data

Page building:

1. At the highest point in any given template create a controller ESI calling this will include all of the other subsequent ESI calls on the page. This is where we will front load all the caching
2. All other ESI calls will only call the data needed
3. Initially we will use a get with args call e.g. [path to esi]/esi.php?q=node/[nid](or user etc)& args=/module_function:salt:variables/module_function:salt:variables/etc
4. The salt could be a user_id, a node_id, other caching needs
5. variables are extra data need in full bootstrap etc
6. We should look at using blocks where possible for our ESI as placement it easy and UI configuration