thumbnails not generating properly? (scalability?)

mparas

Joined: 2012-06-06
Posts: 2

Posted: Tue, 2012-06-12 21:57

Hi, everyone. Long time reader, first time asker...

I have a question regarding the scaling of Gallery. I've seen that there are websites out there that have upwards of 1 million images in their galleries. While we're not quite there yet, we seem to be having some issues with roughly 80k. What's happening is that thumbnails are intermittently not being generated properly. The specific error Gallery is throwing at us is ERROR_LOCK_TIMEOUT on the regeneration of derivatives. Because of the way Gallery works (or at least the way I believe it to work), it does fix itself over time (after refreshing a page x number of times, all thumbnails for that page will generally show properly). This issue only started presenting itself recently as our image count got higher.

Right now, our 80k images are sitting in a single album. Is this reasonable or unreasonable? Will breaking it up into sub-albums help at all? I've tried showing fewer images per page (from 18 to 12) but that didn't solve the problem.

Here's some info about our server architecture (which may also be causing the hiccups):
* Gallery files live on a file server mounted over NFS on 3 load-balanced web servers.
* Web servers use Varnish
* PostgreSQL 9 database using pgpool to manage connections

And here's the general Gallery info

Quote:

Gallery version = 2.3.1 core 1.3.0.1
API = Core 7.54, Module 3.9, Theme 2.6, Embed 1.5
PHP version = 5.3.3 apache2handler
Webserver = Apache
Database = postgres7 8.4.9 3 9.0.7, lock.system=database
Toolkits = ImageMagick
Acceleration = partial/21600, partial/21600
Operating system = Linux web-prod-7s 2.6.32-220.7.1.el6.x86_64 #1 SMP Wed Mar 7 00:52:02 GMT 2012 x86_64
Default theme = matrix
gettext = enabled
Locale = en_US
Browser = Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5
Rows in GalleryAccessMap table = 10
Rows in GalleryAccessSubscriberMap table = 84516
Rows in GalleryUser table = 3
Rows in GalleryItem table = 84516
Rows in GalleryAlbumItem table = 3
Rows in GalleryCacheMap table = 598

Any and all advice is appresh'd!

suprsidr

Joined: 2005-04-17
Posts: 8339

Posted: Tue, 2012-06-12 22:12

80k in one album.... wow, yea I would break into categories/dates etc. and organize them.
Any time gallery fetches dependents that's a whopping query.

http://codex.gallery2.org/Gallery2:Performance_Tips
http://codex.gallery2.org/Gallery2:How_Tos#Performance

and I know joe7rocks has a incredibly huge site.

-s

mparas

Joined: 2012-06-06
Posts: 2

Posted: Wed, 2012-06-13 18:01

Yeah, I figured there was probably something not-so-obvious that was causing things to break. We're going to go ahead and break things up into sub-albums and hope that alleviates stuff.

Thanks for the reply!

mparas Joined: 2012-06-06 Posts: 2	Posted: Tue, 2012-06-12 21:57
	Hi, everyone. Long time reader, first time asker... I have a question regarding the scaling of Gallery. I've seen that there are websites out there that have upwards of 1 million images in their galleries. While we're not quite there yet, we seem to be having some issues with roughly 80k. What's happening is that thumbnails are intermittently not being generated properly. The specific error Gallery is throwing at us is ERROR_LOCK_TIMEOUT on the regeneration of derivatives. Because of the way Gallery works (or at least the way I believe it to work), it does fix itself over time (after refreshing a page x number of times, all thumbnails for that page will generally show properly). This issue only started presenting itself recently as our image count got higher. Right now, our 80k images are sitting in a single album. Is this reasonable or unreasonable? Will breaking it up into sub-albums help at all? I've tried showing fewer images per page (from 18 to 12) but that didn't solve the problem. Here's some info about our server architecture (which may also be causing the hiccups): * Gallery files live on a file server mounted over NFS on 3 load-balanced web servers. * Web servers use Varnish * PostgreSQL 9 database using pgpool to manage connections And here's the general Gallery info Quote: Gallery version = 2.3.1 core 1.3.0.1 API = Core 7.54, Module 3.9, Theme 2.6, Embed 1.5 PHP version = 5.3.3 apache2handler Webserver = Apache Database = postgres7 8.4.9 3 9.0.7, lock.system=database Toolkits = ImageMagick Acceleration = partial/21600, partial/21600 Operating system = Linux web-prod-7s 2.6.32-220.7.1.el6.x86_64 #1 SMP Wed Mar 7 00:52:02 GMT 2012 x86_64 Default theme = matrix gettext = enabled Locale = en_US Browser = Mozilla/5.0 (Macintosh; Intel Mac OS X 10_7_4) AppleWebKit/536.5 (KHTML, like Gecko) Chrome/19.0.1084.56 Safari/536.5 Rows in GalleryAccessMap table = 10 Rows in GalleryAccessSubscriberMap table = 84516 Rows in GalleryUser table = 3 Rows in GalleryItem table = 84516 Rows in GalleryAlbumItem table = 3 Rows in GalleryCacheMap table = 598 Any and all advice is appresh'd!
	XUser login Username: * Password: * Create new account Request new password
suprsidr Joined: 2005-04-17 Posts: 8339	Posted: Tue, 2012-06-12 22:12
	80k in one album.... wow, yea I would break into categories/dates etc. and organize them. Any time gallery fetches dependents that's a whopping query. http://codex.gallery2.org/Gallery2:Performance_Tips http://codex.gallery2.org/Gallery2:How_Tos#Performance and I know joe7rocks has a incredibly huge site. -s

mparas Joined: 2012-06-06 Posts: 2	Posted: Wed, 2012-06-13 18:01
	Yeah, I figured there was probably something not-so-obvious that was causing things to break. We're going to go ahead and break things up into sub-albums and hope that alleviates stuff. Thanks for the reply!

thumbnails not generating properly? (scalability?)

XUser login