speed of PHP 5.1 and 4.3

With reference to Gallery2 Performance Tips, it's mentioned that migrating to PHP5 should gain some performance. I was trying it out today but found that my album page loads actually slower. e.g. on a slow connection, it took 5sec(4.3) vs 6sec(5.1) to load the same album page.

I'm wondering what's you experience in switching to PHP5.1. I've to point out I'm on a slow server (a P3 machine), not sure if it's the cause or not though.

Gallery version: 2.1.2
PHP version: 4.3.10 / 5.1
Webserver (e.g. Apache 1.3.33): 2.0.54
Database (e.g. MySql 4.0.11): 4.1.11
Activated toolkits (e.g. NetPbm, GD): hidden album
Operating system (e.g. Linux): GNU/Linux
Browser (e.g. Firefox 1.0): Firefox 1.5