Lessons from a server upgrade

Recently, the server hosting the Ethical Co-op website died, and needed to be replaced. It launched in 2005, running on a standard LAMP system.

Since it was written in the days of PHP4, and was working well, I’d never upgraded it to PHP5. Everything was running smoothly on PHP4, but this had reached end of life, and running PHP4 on the server was proving more and more of an obstacle when trying to install other, newer code.

So, when the server died late on Saturday night, and with the weekend usually being a very quiet time, I decided to use the opportunity to finally move to PHP5.

The server has needed to be replaced or reinstalled perhaps three times since 2005. I’m not a natural system administrator – I have a programming and database background, and have never done system administration professionally, so the first few times were a bit shaky. However, I’ve improved as I’ve gone along, and the installation went very smoothly this time.

I have everything carefully documented. There are a lot of tweaks to the vanilla system, various extra PEAR libraries installed, and all are carefully documented, so setting up from scratch is a breeze.

I upgraded the code from PHP4 to PHP5 which (considering my coding style carefully honed from its roots in ZX-81 BASIC) was trivial. Everything was back in good time by the end of the weekend, my tests reported no problems, and I went to bed.

Monday the problem reports started.

It didn’t take long to identify the problem. POST variables were being chopped off. Consistently nothing more than around product 200 was being returned. I’d never encountered a problem like this before, but after Googling I identified a culprit. Suhosin, the Hardened PHP project, has a post.max_vars setting which defaults to 200. A quick look at my config and I see This server is protected with the Suhosin Patch.

A perfect fit. This must be the cause of the problem!

Cursing the paranoid security, I spent much of the day investigating Suhosin. A Twitter search returned “Fuck Suhosin!” as the first result, making me more sure I’d identified the culprit.

None of the Suhosin settings were appearing in my phpinfo() output, which apparently is normal behaviour for the defaults, and explicitly setting them in php.ini, which seemed to solve the problem for most people out there, had no effect.

The server by default had the Suhosin patch installed, and in my attempts to get come control over the configuration I installed the Extension, which finally gave me control.

At this point I realised my first mistake. There may have been 200 products, but each product has five variables, so the cutoff was actually happening at 1000, not at 200.

And so I made my second mistake. If suhosin.post.max_vars is by default set to 200, it couldn’t be the cause of the problem. Still, there was no way to know that the setting was 200 on my system, and in the default configuration file that appeared after I installed the extension, this line appeared

;suhosin.post.max_vars = 1000

Commented out, but indicative that the default was 1000, not 200 as the documentation indicated. By now I really was convinced that Suhosin was the spawn of all evil.

However, no matter what changes I made to these and other possible Suhosin settings, nothing made a difference.

Eventually I had to accept it wasn’t Suhosin, and I had wasted most of the day. It had been unclear to me from Suhosin’s documentation exactly the difference between the patch and extension was, and I had also been misled by some posts claiming that the patch was responsible for the POST var limit, when it appears after all it isn’t, and this is only implemented if the extension is installed.

By that stage there was such a mess from the day’s errors and the downtime that I was ready to cancel the week’s deliveries (the Ethical Co-op delivers once a week, with most of the orders arriving on Monday and Tuesday).

Bernhard of Meglakor had been helping me, and was much quicker than me to realise Suhosin was not the cause. I was convinced it was the cause; after all, it had seemed a perfect match, and it was the only thing I could find that limited POST variables.

But it wasn’t.

Earlier this year, PHP 5.3.9 was released. One of the minor changes made during that release was the following:

Added max_input_vars directive to prevent attacks based on hash collisions.

There it was! PHP now limits any input variables to 1000 by default! After all my worrying about upgrading the code, it was a minor configuration change in a minor version that caused all the problems.

A classic case of being led astray by mis-identifying the problem, and then being less open to alternatives because of the investment in the identification!

A simple change, and everything was working again.

Thanks Bernhard for clearing Suhosin’s name, and then identifying max_input_vars.

Related Posts:

4 Comments to “Lessons from a server upgrade”

  1. Marius
    March 21, 2012

    Ian, upgraded to a shiny new PHP but still running an old WordPress (3.1.4) … not good at all :)


  2. greenman
    March 21, 2012

    Marius, I just knew someone was going to comment on that :). I could never upgrade beyond 3.1.4 being stuck on PHP4, which was part of the reason for the urgency. No excuse any more.

  3. Jeff
    March 21, 2012

    The problem with Open Source, is: staying current with the core.

    Only alternative for you, is “to probably backup more.”


    (Oops, now I said that… ;)

  4. greenman
    March 24, 2012

    I see this was probably the reason for the change.