Performance
|
Glooper
Joined: 2005-09-21
Posts: 225 |
Posted: Thu, 2009-08-13 22:06
|
|
Hi, I've got s site that is going live next week that uses gallery3. It's all going well, I've just updated the code to the latest version on github, and it's looking good. I was hoping to get my shopping cart integration added in time...but it's still not ready so that will have to wait until the next release of the site! Anyway.. I've got a few worries about the performance of Gallery3. At the moment it's taking about 2s seconds to load a page... are there any ways I can currently finetune the performance of gallery3? Or is performance of Gallery3 something that you'll be looking at in a future iteration? If so will this be months or weeks away.. just so I can go back to the client and give a small idea of timing of when these issues will be sorted. Cheers |
|
Posts: 344
What's the site link?
Posts: 225
well.. it's not quite ready yet so I can't give you a URL... will be up next week sometime though.
I was kinda looking for general hints and tips with regards to gallery3 rather than specific ones to this new site
Ben
Benjamin Albert Smith - Photography
Posts: 25953
Well it is hard to diagnose a issue when you say it is slow and we can't even see the site.
FYI from the README ( then never gets read ):
Do you have any data from your webserver admin on why it is slow?
What is the servers specifications?
Is it slow in some other respects?
Is it shared with others?
Do other sites on the same server run 'fast'?
Slow in comparison to G1 or G2?
Is the bottleneck the DB or the Apache web server?
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 225
fair comment
...
ok.. you can have a look at the site in it's pre-release form here
http://60.234.40.50/~capturei/index.php
Just don't tell the people it's for!!
this site is on the same server as...
http://www.captureitnz.co.nz/
which has good latency...
Chrome is telling me it takes around 2s latency from a page request to a response.
Obviously there is a lot more going on in the gallery site than the captureit one which only has basic php stuff in. So I can't ever see it matching the speed of the capture it one.
I'm not sure whether the bottleneck is the DB or apache... I'll see what logs I have on the server now..
Ben
Benjamin Albert Smith - Photography
Posts: 225
oh.. and it's fine.. I'm willing to live a bit on the edge! ;)
To be fair I didn't want to release this site within the next week... it was intended for a few months time... unfortunetly there was a change of plan as GATE are doing an event next week so need the website in time for it! Bloody customers moving the goal posts forward! :P
Benjamin Albert Smith - Photography
Posts: 16503
I started this post yesterday and left....
Do you happen to have G2 running on the same server? If so, you should notice that G3 is faster than G2. Just as a comparison.
And what exactly are you counting as 2 seconds? That seems really fast to me, my G3 sites load fast and total page load is about 5 seconds. Right now G3 is pretty heavy. A totally uncached page (depending on thumbnails you have, etc) is about 400KB according to YSlow
I see your page is around 360KB so you've slimmed it down a bit. Is your site you gave an the other example of the site being on the same server making any database calls? Is the database on the same server? How loaded is the server over all?
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here
Posts: 225
I'm looking at the pages initial latency. My image downloads I've stuck in an ajax loader so I'm not so worried about the speed they come across. the JS and CSS seem to download nice and fast too... but the initial latency of the page is what I want to try and decrease. Generally it takes around 2 seconds from clicking a link before any data starts to be received by my web browser (sometimes this shoots up to around 6-8 seconds).. This is obviously the time where Gallery is generating the pages and doing a lot of work in the back end.. talking to the database etc...
To try and speed things up I've done the following so far.
- removed a lot of the js and css that wasn't used by my theme (when logged in as admin I re-add it so that maintenance tasks and album editing etc.. can still be done). Though I can't see this speeding up the latency by much unless the combined JS and CSS are created every call (I assume they are cached?).
- removed a lot of the theme calls that aren't used by my installed modules (the album top, sidebar, album bottom, header bottom etc..)
- only used a few key modules. Currently using only the organise, user_homes and one for my contacts page. I'll probably re-add tags, search and the shopping basket ones at a later date.
The other website you've seen doesn't make calls to a database and only has some very basic PHP to mainly split out the common components between pages. So I agree it's unfair to ever consider G3 to perform as well as that. And unfortunetly I cannot tell how loaded the server actually is, being as It's rented space on a datacentre in Auckland. I'm pretty certain the database is on the same server too, which according to my "directadmin" control panel for the site is a quad core2 intel 2.4ghz with around 3gb memory.. can't seem to find out what version of linux it is. And you are right.. it is running faster than G2 did... that's part of the reason why I was eager to use g3
... I just want to try and squeeze even more performance out of it to improve the responsiveness of the site! 
What I'm after is any other ways in which I can reduce that latency...(other than getting better hardware!)... Are the created pages currently cached? or are they recreated each time? (if they are cached then removing the theme calls isn't going to have an effect?)... Is there perhaps some indexes I can put on the database to help improve things? Is there anything in the core code that perhaps I can look at that might be a known bottleneck?
Ben
Benjamin Albert Smith - Photography
Posts: 16503
Currently, I don't believe there's any caching being done in G3 except for your combined CSS and JS files, which do get recreated it any of the included JS or CSS files are modified. So it is making checks for all that stuff all the time. Which you could further eliminate by changing lines like:
<?= $theme->css("yui/reset-fonts-grids.css") ?>to "normal" css and possibly combine all the JS and CSS into one file manually
However, even though that is extra work that Gallery is doing I don't think it's very expensive. It's probably database and database performance you're hitting. My suggestion, live with it, or get a different host, a VPS, www.linode.com or a host that doesn't host the database server on the same server as the sites, www.pair.com
But honestly for you being in NZ, those sites actually load for me relatively fast compared to other NZ sites I've visited. Most NZ and AU based sites I visit from the US load pretty slowly.
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here
Posts: 225
Cheers Nivekiam...
..
Using a premade JS and CSS helped a hell of a lot... the latency has gone from around 2+ seconds (sometimes up to 8s) to a speedy consistent 400ms's
...
Will probably switch back tomorrow sometime whilst making more changes to the site, so have a quick look now if you can and see the difference.
http://60.234.40.50/~capturei/index.php
I'll make sure when going live later in the week to use pre-made JS and CSS's
... probably minify the js too ;)
Ben
Benjamin Albert Smith - Photography
Posts: 16503
Looks like that area could use some investigating and tuning then
Thanks for the feedback. BTW, that's one kick ass theme
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here
Posts: 716
If I remember correctly, the premade JS and CSS files are cached in the database, so if your server has database performance issues then that's probably were the latency issues were at.
Posts: 25953
I agree!
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 225
Cheers floridave and nivekiam...
... Should be even better when I get the new graphics for it. currently taking photos to replace the rather computer generated backgrounds with real images and complete the last two front page squares. Unfortunetly I won't be able to share the theme as it's for a customer... but I promise to share some elements of it in the future. ;) To be fair the G3 framework is a breeze to write themes for. 
rWatcher... Just been looking at the code to work out how the combined JS and CSS stuff works... you are right it is stored in the database making use of the rather notorious longblob. Though this is also accessed when actually downloading the js and css files yet the latency was reasonable for that.. certainly not the 1.6 seconds that it was taking to access when initially creating a page... My guess is that there is a problem creating the key for the cache and therefore its actually combining and zipping the JS's and css's each time (though I might be wrong). I guess it could also be a problem reading the stats for files (which it uses to make the keys), so could be an issue on the server I'm using still.
After I've got this site release out the way next week, hopefully I'll have some time to have a look at it in more depth to see If I can find out if there is a bug in the code or it some issue on the server. Also, instead of using blobs can I recommend sticking the combined js and css's in the var directory and referencing them directly.
Ben
Benjamin Albert Smith - Photography
Posts: 25953
Great! looking forward to this and glad you like the 'new' framework.
Will ping bharat and tamldal to join this thread.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 358
Yea, we thought about putting them as files, but wereworried about having files hanging around the files system. So Bharat and I decided to put it in a caching mechanism
http://www.timalmdal.com
Posts: 225
Can you not delete old files as you would database rows?
Whilst in development I guess the number of css and js files would constantly be increasing as you change bits and pieces.. but in production sites surely they wouldn't change? And the performance increase of files over blobs would be beneficial?
Actually.. I checked my cache database and it had 866 rows in there... the total size of my database is 76.4mib and the cache table holds nearly all of it at 75.6mib! So I'm not sure the cache is deleting rows anyway?
I'm willing to help with coding any of this if need be... I've got a number of potential clients that I'm looking to use Gallery3 so I will have to recode this area for performance in any production sites anyway.
Let me know what you think... I of course could be missing something blatantly obvious!
Ben
Benjamin Albert Smith - Photography
Posts: 358
We could, but we decided not to...
One of the beautiful things about about using git as our source management system, is how easy it is to fork a project and integrate patches from the forks. Where this comes into play is if you really disagree with our approach on something (and this is a really good example); you just fork the project and implement the feature as you choose. If your implementation is good enough (robustness of code, following our coding standards) and adds to the robustness of the Gallery3 product, we can easily integrate the change into the main code base.
So you might be right, but we have picked this path (we believe that it meets the needs of 80% of our users) for the moment and our focus is on getting a release out the door, we probably won't change it... so much other stuff to do. It is worrisome that the cache is not getting cleaned or we are generating that many versions of the combined js and css files. I've raised ticket https://sourceforge.net/apps/trac/gallery/ticket/644 for that.
What I would suggest is create a fork, that uses files for the css and js. We can always pick it up and integrate it later if it thats a direction that makes we want to go.
Thanks for all your investigation
http://www.timalmdal.com
Posts: 225
Cool.. sounds good.. I'll start work on a fork sometime in the next couple of weeks
Ben
Benjamin Albert Smith - Photography
Posts: 16503
Hey Glooper, on that same server you were having this issue with, do you happen to have root access?
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here
Posts: 225
No unfortunetly not. it's a hosted server in Auckland.
Benjamin Albert Smith - Photography
Posts: 16503
Well location isn't relevant, but without root access then you can't check a few things we're trying to look at in terms of performance for the generation of the combined JS and CSS on some servers.
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here
Posts: 225
well.. location would be relevant if the server was sitting right next to me... ;)
but yeah.. sorry... no root access. Let me know if there are any php scripts that you want to run on it... maybe you can extract the information using a php script?
Ben
Benjamin Albert Smith - Photography
Posts: 16503
Nah, we'd need to know available resources to the web server. How much memory does Apache have available to it. How much is used. Is swap getting hit, etc.
Do you know if that's a typical shared host or a VPS (Virtual Private Server)?
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here
Posts: 225
its a shared host... through these guys
http://www.hostingdirect.co.nz/services/web-hosting
the unmetered package... currently I have two website on there, though I imagine there's loads on there from other customers,
from the control panel I can get the following information about the server
Processor Name Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
Vendor ID GenuineIntel
Processor Speed (MHz) 1197.043
Processor Name Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
Vendor ID GenuineIntel
Processor Speed (MHz) 1197.043
Processor Name Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
Vendor ID GenuineIntel
Processor Speed (MHz) 1197.043
Processor Name Intel(R) Core(TM)2 Quad CPU Q6600 @ 2.40GHz
Vendor ID GenuineIntel
Processor Speed (MHz) 1197.043
Total Memory 3108068 kB
Free Memory 1010588 kB
Total Swap Memory 7727248 kB
Free Swap Memory 7727204 kB
System Uptime 26 Days, 12 Hours and 8 Minutes
Apache 2.2.13 Running
DirectAdmin Running
Exim 4.67 Running
MySQL 5.1.37 Running
Named 9.3.4 Running
ProFTPd 1.3.2a Running
sshd Running
dovecot 1.2.4 Running
Unfortunetly I can't see how much memory apache has been allocated. Though have just seen it has 4 quad core processors though ... nice...
.
Ben
Benjamin Albert Smith - Photography
Posts: 7985
Sorry for the long silence. Is this performance problem still happening? Can you point me at a site where I can see it happening? I'd love to profile it with Firebug so that I can figure out which asset requests are slow.
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git | help! vote!
Posts: 225
I've temporarily sorted it out by hand combining the css and js files, removing the head call in the theme and just referencing the combined js and css directly. It was happening here...
http://www.gatephotography.co.nz/
It seemed to me that the js and css were being combined each time causing the initial response to a page request to take a while. This would also explain why the cache database was so full of css and js files, it was up to about 74mb. It did seem worse with this theme than the default so I'm not sure if something in my css or js or both was causing it to recreate each time (though I can't think what).
I have a test environment on the site now.. so tomorrow morning I'll set it up and revert back to using the normal js and css process. I'll send you a link when it's up...
Ben
Benjamin Albert Smith - Photography
Posts: 225
I can't seem to replicate it anymore!
I think it's because I've cleared down the cache table. Like I said there was about 74 mib in there before. Maybe in clearing it down it has solved the problem. there's now only 8 entries in that table.. there was 800 odd before
Ben
Benjamin Albert Smith - Photography
Posts: 225
Currently working on a new site and have had performance issues again. I thought I'd narrow down the problem a bit more. It's definitely a problem with the caches table. I ran a mysql query log to find out what is taking a while and it is the following two queries used to retrieve the javascript and css.
100219 9:55:45 1 Query SELECT *
FROM `ml_caches`
WHERE `key` IN ('2f61be3647c22fee824894ab6823ca31')
100219 9:55:46 1 Query SELECT *
FROM `ml_caches`
WHERE `key` IN ('13caa5a3eac4cdcdf5c4c128b4e42b93')
I timed these queries in a mysql client and they are taking around 1.34 seconds to complete each.
I think what's happening is that the more information you put into the caches table it slows it down a lot. (blobs have a habit of doing that). I checked the size of this table and I currently have 583 rows in it. I think the reason it's not clearing is because I am constantly changing the JS and CSS files to get the site right, and each change is adding a new entry into the cache table.
I've just cleared the cache table.. and allowed it to rebuild the js and css files. The table now consists of 4 rows and the sql statements run in the normal 0.00 seconds.
Basically there's no problem when it is running in a production environment and nothing changes. the only problem is when you are doing changes to the CSS and JS files.
For a solution...When it checks the cache could it not check to see if the number of rows is above a threshold and clear the table if it is?
Ben
Benjamin Albert Smith - Photography
Pikitia - New Zealand Photography
Posts: 32509
@Glooper:
Please file a bug. The caches table should have an index on the 'key' column. All our lookups are primarily on the key column and it's silly not to use an index.
With an index, those queries will be super fast even if there are thousands of rows in the caches table.
http://sourceforge.net/apps/trac/gallery/ -> new ticket.
Thanks!
Posts: 225
Bug added to tracking system.
Benjamin Albert Smith - Photography
Pikitia - New Zealand Photography