Making Sessions Cluster-aware

2009-05-19

The Problem

Once your application scales to the point that a single server isn't able to serve all your visitors anymore, sessions can become a problem.

The default files-based session handler was only designed for a single server and so this must be taken into account if your application is to continue to run flawlessly.

In this article I try to show some of the options available for dealing with sessions in a cluster.

The Solution

One thing that quite often isn't taken into account once sessions need to change from the default files based session handler is session locking. Without session locking a race condition exists where two requests may trample on each others session updates. This is made far worse in an AJAX type application since the number of requests is frequently an order of magnitude or more greater than a normal dynamic site.

Depending on the session handler and how this is written, this can result in one of three options:

It works as expected
The session data from one request may be lost.
The session data is corrupted.

Generally the first or second outcomes are most likely as PHP has a couple of optimisations in how it uses its session handlers.

As far as I can tell, if the session data hasn't changed, then the custom session_write function isn't called, this means that for requests where the session data doesn't change, the risk is far less than the worst case would suppose (assuming the session_write handler is used to write and its not just placed in the session_close handler). This would suggest that if we wanted to be safe we could have two custom session handlers, one that supports locking for session writers (so they can't trample over each other) and one that ignores locks where only reading is required.

Option 1 - Add a session-aware load balancer

A normal loadbalancer will just shunt connections to different machines in turn, either in a roundrobin algorithm or using a more intelligent technique.

Being session-aware means that the value of the session cookie (or session id in the url) is taken into account in the routing of each request within the cluster. Every request with the same session id will be routed to the same backend server within the cluster. This has two side effects. First, that the standard session handling of PHP (or other scripting language) will continue to work the same way as it did on a single server, the second that if the server goes down, then any open sessions that point to that server will be lost. This can mean that important user data is lost and your app appears unreliable in your users eyes.

For this reason, this option should only be used when there is no real other choice.

Option 2 - Store Sessions in a cluster-wide filesystem.

The default files based session handler can be made cluster-aware if we place it on a cluster-wide filesystem. The only requirement this filesystem needs is that it needs to support file locking. This means that many NFS implementations may be out. One option that is available however is that of GFS. GFS is the cluster filesystem made popular in the RHEL (and by extension CentOS and Fedora) distribution. I'm not going to go into detail here, but I mension it for completeness.

Option 3 - Store Sessions in a Database

PHP allows alternative session handlers to be used in place of the default file-based session handler. There are many examples available on the net and most appear to work fine when under light load. However, the majority available all suffer from one flaw; the don't include any locking of the session data. This means that if two requeests for the same session arrive at the same time, it is very likely that one will trample over any changes to the session made by the other request.

If using MyISAM or Memory tables, adding locking will stop all other requests that touch any session will be unable to run in parallel. The reason for this is that MyISAM and Memory tables both use table level locking. So once locked, no other query can run on the same table until its unlocked.

The alternative to this is to use InnoDB tables. These support row level locking and as such a lock should only block requests from a single session id.

Option 4 - Store Sessions in Memcache

Storing sessions in memcache is fairly trivial to do. You simply install the memcache extension and an extra session handler is made available.

The only draw back as far as I can tell is that memcache doesn't seem to support locking of the session data. So trampling on other requests session data is quite likely if a lot of requests are made in a short space of time, whether this is actually a problem is hard to know for sure though. A lot of big sites are using this option without reporting problems.