Maintenance/Build all thumbnails/resizes - locking error

TopSoftbe

Joined: 2005-03-28
Posts: 15
Posted: Wed, 2006-03-08 23:02


Gallery version (not just "2"):2.0.3 core 1.0.0.3
PHP version (e.g. 4.3.11):4.4.1 cgi-fcgi
PHPInfo Link (see FAQ):
Webserver (e.g. Apache 1.3.33):Microsoft-IIS/6.0
Database (e.g. MySql 4.0.11):mysql 4.1.10a-nt
Activated toolkits (e.g. NetPbm, GD):Exif, ImageMagick, Gd, NetPBM
Operating system (e.g. Linux):Windows NT SERVER2003 5.2 build 3790
Browser (e.g. Firefox 1.0):Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; InfoPath.1; .NET CLR 2.0.50727)

In Maintenance/Build all thumbnails/resizes, Gallery crawls very fast trough the existing resizes, then starts building the new ones, but then stops after a few minutes. When I refresh the page, I am able to go a little bit further. But it never goes to the end. I have nearly 13.000 pictures, I am at 8500 for the resizes. The errors I get are:

Last Run Details:

Error (ERROR_LOCK_TIMEOUT)

in D:\Inetpub\wwwroot\gallery2\modules\core\classes\GalleryStorage\DatabaseStorage.class at line 2391 (gallerystatus::error)
in D:\Inetpub\wwwroot\gallery2\modules\core\classes\GalleryStorage\DatabaseStorage.class at line 906 (mysqldatabasestorage::_getlockclearance)
in D:\Inetpub\wwwroot\gallery2\modules\core\classes\GalleryStorage.class at line 595 (mysqldatabasestorage::acquirewritelock)
in D:\Inetpub\wwwroot\gallery2\modules\core\classes\DatabaseLockSystem.class at line 88 (gallerystorage::acquirewritelock)
in D:\Inetpub\wwwroot\gallery2\modules\core\classes\helpers\GalleryLockHelper_simple.class at line 185 (databaselocksystem::acquirewritelock)
in D:\Inetpub\wwwroot\gallery2\modules\core\classes\GalleryCoreApi.class at line 2113 (gallerylockhelper_simple::acquirewritelock)
in D:\Inetpub\wwwroot\gallery2\modules\core\classes\helpers\GalleryDerivativeHelper_advanced.class at line 692 (gallerycoreapi::acquirewritelock)
in D:\Inetpub\wwwroot\gallery2\modules\core\classes\GalleryCoreApi.class at line 843 (galleryderivativehelper_advanced::rebuildcache)
in D:\Inetpub\wwwroot\gallery2\modules\core\classes\helpers\GalleryDerivativeHelper_simple.class at line 66 (gallerycoreapi::rebuildderivativecache)
in D:\Inetpub\wwwroot\gallery2\modules\core\classes\GalleryCoreApi.class at line 830 (galleryderivativehelper_simple::rebuildcacheifnotcurrent)
in D:\Inetpub\wwwroot\gallery2\modules\core\classes\BuildDerivativesTask.class at line 92 (gallerycoreapi::rebuildderivativecacheifnotcurrent)
in D:\Inetpub\wwwroot\gallery2\modules\core\AdminMaintenance.inc at line 104 (buildderivativestask::run)
in ??? at line 0 (adminmaintenancecontroller::runtask)

I have tried the database lock and the file lock, it makes no difference whatsoever.
Am I the only one with windows? Anyway, other users have the same problem under linux as far as I can see. Is there another solution to this problem instead of manually restarting the process and rebuilding some hundred resizes each time before it blocks?
Regards.

 
TopSoftbe

Joined: 2005-03-28
Posts: 15
Posted: Fri, 2006-03-10 13:58

I am at 10005 now...
Someone has an intelligent idea? Is there some time-out setting I can tweak?
Regards.

 
ckdake
ckdake's picture

Joined: 2004-02-18
Posts: 2254
Posted: Tue, 2006-03-14 05:50

Turn on buffered debug mode so we can see the actual error getting generated: FAQ: How to set/use Gallery in debug mode?
--
http://ckdake.com/

If you found my help useful, please consider donating to Gallery.

 
TopSoftbe

Joined: 2005-03-28
Posts: 15
Posted: Tue, 2006-03-14 19:15

I have tried with 'buffered' and with 'logged', but the program just stops and generates no log whatsoever. I mean, with buffered, I have a beautiful debug report each time I do something, but when rebuilding the resizes, there is nothing, the program just stops after some 100 to 200 pictures and that's it.
Anyway, I have resized all my pictures now, just by restarting the process and advancing a few hundreds of pics each time. I think I will wait for V2.1, can live with this for a while.

 
ckdake
ckdake's picture

Joined: 2004-02-18
Posts: 2254
Posted: Wed, 2006-03-15 05:56

Another report here: http://gallery.menalto.com/node/45886

this may be a bug type issue and not just for you. Will follow up further..

--
http://ckdake.com/

If you found my help useful, please consider donating to Gallery.

 
xrmb
xrmb's picture

Joined: 2005-10-02
Posts: 10
Posted: Wed, 2006-03-15 13:47

I posted two topics regarding this problem, I'm glad that more people have the problem and get answers...

Here is my logfile. All I do is opening my gallery. Sometimes I get an Gallery error message, sometimes an empt page. I switched to file based locking, so I can remove old locks, doesn't really help, but at least it gets the gallery running again for a few minutes.

http://xrmb.home.comcast.net/gallery.log (800 kb)

Let me know what else you need to debug it.

 
fabs
fabs's picture

Joined: 2004-05-10
Posts: 171
Posted: Wed, 2006-03-15 16:30

ok- so now we are three as ckdake posted my former thread before already.
let me know if anyone figures it out.

 
xrmb
xrmb's picture

Joined: 2005-10-02
Posts: 10
Posted: Thu, 2006-03-16 18:44

I changed the gallery.class:

function guaranteeTimeLimit($limit) {
if ($limit <= 0) {
$limit = 30;
}
$limit*=5;
$now = time();
if (empty($this->_timeLimit) || ($this->_timeLimit - $now < $limit)) {

it just guarantees more time to certain functions... might not be the smartest place, and should also not be considered as fix, bit at least it works now...

 
ckdake
ckdake's picture

Joined: 2004-02-18
Posts: 2254
Posted: Mon, 2006-03-20 04:01

This may be a related thing as well: http://gallery.menalto.com/node/46051#comment-176035

We may just need to be guaranteeing more time in a couple of places:
-refresh capture date
-build all thumbs/resizes

--
http://ckdake.com/

If you found my help useful, please consider donating to Gallery.

 
nickctn

Joined: 2005-01-29
Posts: 15
Posted: Mon, 2006-06-12 22:05

Hi,

Which file/funciton should I be modifying in order to allow more time for the "refresh capture date"?

Thanks.

My site // My Gallery

 
sykong

Joined: 2006-04-07
Posts: 21
Posted: Mon, 2007-02-19 04:24

I'm also facing this problem. I searched around on the forums and the support page. It seems that there is no real solution to this problem? Especially when there are a lot of photos involved.

 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Mon, 2007-02-19 10:49

sykong

if you're using g2.2, we can try to diagnose the problem. if you're using a prior version, i'd suggest you upgrade to g2.2 and check if the problem is still there.

--------------
Doumentation: Support / Troubleshooting | Installation, Upgrade, Configuration and Usage

 
ichthyous

Joined: 2006-06-16
Posts: 324
Posted: Mon, 2007-02-19 15:55

I am also having a similar problem...when I rebuild thumbnails in some albums the bar stops at about 40%-50% and just stays there. It doesn't happen in every gallery and no error is reported. I tried switching graphics processors and lowering the image quality but that didn't help. I have been having many problems with being able to edit replica photos so I think this might have something to do with that. With some replicas, when you go to edit them you get an "item not found" error. The only solution to this that I can see is to delete the item and start over. I think that the thumbnail rebuild is getting stuck when it hits a replica with missing data. Are any of you who are seeing problems with thumbnail rebuild using replicated items in your albums as well? Thanks

New York photos by Andrew Prokos

 
ichthyous

Joined: 2006-06-16
Posts: 324
Posted: Mon, 2007-02-19 15:59

Sorry forgot to post my info:

Gallery version = 2.2-svn core 1.1.16
PHP version = 4.4.4 apache
Webserver = Apache/1.3.37 (Unix) mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 PHP/4.4.4 FrontPage/5.0.2.2635.SR1.2 mod_ssl/2.8.28 OpenSSL/0.9.7a
Database = mysqlt 5.0.27-standard-log, lock.system=flock
Toolkits = LinkItemToolkit, Gd, Thumbnail, ImageMagick, NetPBM
Acceleration = none/86400, none/86400
Operating system = Linux vps.andrewprokos.com 2.6.9-022stab078.23-enterprise #1 SMP Thu Oct 19 14:54:39 MSD 2006 i686
Default theme = PGlightbox
gettext = enabled
Locale = en_US

I tried delteing db cache and template cache as well as optimizing the DB...did not help.

Thanks

Washington DC Photography by Andrew Prokos

 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Mon, 2007-02-19 16:48

ichthyous

please create a new forum topic for your questions.
can you give us steps to reproduce the replica issues?
starting with no replica, how do i get an error involving replicas?

--------------
Doumentation: Support / Troubleshooting | Installation, Upgrade, Configuration and Usage

 
ichthyous

Joined: 2006-06-16
Posts: 324
Posted: Tue, 2007-02-20 03:22

Well my question is really the same...rebuilding thumbnails doesn't work. I am not sure it's the replicas that are causing the problem, but I have been having problems with some of my replica items. When I try to edit them I get an "item not found" error. When I try to rebuild thumbnails in an album that has some of these problematic replica items it gets stuck. I will try to find a specific replica item that is causing problems and delete it, then run the rebuild again and see if it goes this time

New York location photography by Andrew Prokos

 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Tue, 2007-02-20 03:31

it would be great if you could provide instructions on how to get errors / issues starting with an album that doesn't show any issues before.
if we can't reproduce the problem, it's very hard to diagnose / understand it.

--------------
Doumentation: Support / Troubleshooting | Installation, Upgrade, Configuration and Usage

 
ichthyous

Joined: 2006-06-16
Posts: 324
Posted: Sat, 2007-02-24 20:42

Sorry to take so long to get back to this...it's definitely not the replicas that are causing this problem as when I delete the replicas that are giving errors I still can't rebuild thumbs. I am still unable to rebuild thumbs in some of my folders...it seems to me to be the ones that hold a large number of images. For instance the architectural photography section of my site (link below) gets stuck at about 60% during the rebuild if I choose the same thumbnail size, or at about 40% if I choose a different thumbnail size. It seems to be a memory issue of some sort. All the thumbs up to the point where it stops are rebuilt, but after that point they are not. No error is produced. As I said I tried lowering the image quality in ImageMagick and also switching to GD but it didn't change. I never used to have this issue before Thanks

Architectural Photography by Andrew Prokos

 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Sat, 2007-02-24 23:34

if it's a memory issue, ensure that php is configured to log all errors to the webserver's error log such that you see all messages in there.
gd is certainly the wrong toolkit if you have memory problems. imagemagick doesn't use PHP's memory pool and is thus not affected by php's memory limits.

--------------
Doumentation: Support / Troubleshooting | Installation, Upgrade, Configuration and Usage

 
ichthyous

Joined: 2006-06-16
Posts: 324
Posted: Sun, 2007-02-25 20:15

I am using Image Magick, not GD, so I guess the memory theory's out too. Have you ever seen this issue before? If so, how was it resolved?

New York Landmark Stock Photography by Andrew Prokos

 
TopSoftbe

Joined: 2005-03-28
Posts: 15
Posted: Sun, 2007-02-25 22:55

I started this thread a long time ago, and I have upgraded in the mean time to version 2.2-RC1. Now I decided to delete some predefined image sizes and after that rebuild all the thumbnails and resizes...
Nothing has changed: I still have the same problem: the system rebuilds some 200 images and then falls asleep. I click back in my browser, click again on rebuild and the system goes very quickly to where it stopped, does again a few hundred rebuilds and stops again. No errors, nothing. I have some 15.000 images, it took me all day to rebuild all resizes.
The only thing that may be different from other installations is that I have my own server, so I do the rebuild 'locally'.
Can someone tell me what is needed to try to find the solution to this long-standing problem? Perhaps the fix could then be incorporated into V 2.2 final.
Thanks in advance.

Gallery version = 2.1 core 1.1.0
PHP version = 4.4.1 cgi-fcgi
Webserver = Microsoft-IIS/6.0
Database = mysql 4.1.10a-nt, lock.system=database
Toolkits = Exif, ImageMagick, Gd, NetPBM
Acceleration = none, full/604800
Operating system = Windows NT TOPSERVER2003 5.2 build 3790
Default theme = matrix
Locale = en_US
Browser = Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.2; SV1)

 
mindless
mindless's picture

Joined: 2004-01-04
Posts: 8601
Posted: Mon, 2007-02-26 23:05

Here are things you can do that might help us find a solution:

1) Provide updated debug information now that you're using 2.2-RC-1:
a) error stack trace
b) attach or link to debug output using "logged" debug mode in config.php. save the change in config.php to turn on logged debug output just before you click "run task" so we only get output for that action, not other requests while you navigate to site admin / maintenance.

2) Try with both file locking and database locking in Site Admin / General and let us know if that makes any difference

3) Deactivate Gd and NetPBM (only imagemagick active now) and see if that makes a difference

4) Try with only NetPBM active (deactivate imagemagick) and see if that makes a difference

 
ichthyous

Joined: 2006-06-16
Posts: 324
Posted: Tue, 2007-02-27 18:10

I tried switching to DB locking, switching graphics processors, and disabling GD and NetPbm...didn't change anything. Not sure what an error stack trace is...please xplain. I will try logged debug output later on tonight after business hours. Thanks

Architectural Photography by Andrew Prokos

 
TopSoftbe

Joined: 2005-03-28
Posts: 15
Posted: Tue, 2007-02-27 19:04
Quote:
Here are things you can do that might help us find a solution:

OK, I will try all the suggestions, but this will be a rather lengthy process. Please allow me a day or two, then I will come back with the results. (I work during the day, so it will be nightshift work... :-)

 
mindless
mindless's picture

Joined: 2004-01-04
Posts: 8601
Posted: Wed, 2007-02-28 03:33

ichthyous, an error stack trace is what you see in the first post of this topic, starting with "Error (ERROR_LOCK_TIMEOUT)"