Rebuilding images/thumbs is most slow and unstable part of application... arrrrgh
|
Serge D
Joined: 2009-06-11
Posts: 2436 |
Posted: Fri, 2010-12-31 00:08
|
|
Sorry, have to vent this so I can get back into New Year spirit... Rebuilding images/thumbs is most slow and unstable part of application * over time it can cause 500 error on the server, then all hell will go loose |
|
Posts: 7985
Sorry to hear it. It sounds like your server is not doing well when we run a long series of graphics operations. The 500 errors and "unable to fork" errors are typically only seen when your server is out of resources in some fashion. Go to Admin > Dashboard when rebuilding -- what's your server load?
We can slow down the rebuild to go a little easier on your server if it's a problem but most of the errors you're seeing I think come from your system just getting overworked.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 25961
It still seems to work for me on the 2 hosts I have as well as 3.0 and a git version. What specifically should be updated?
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 2436
@floridave: enable strict mode to see error in admin section - missing parameter for getform function
@bharat: Current Server load: 3.65 4.08 4.63
as I said, rebuild process should be more intelligent and more error prone. #3+ points are still apply regardless of my server load...
rebuild is one of the critical tasks and current handling is just not enough.
I am running rebuild now, and it seems to be going... yesterday it was dead in the water and process would bring down whole g3...
PS. I suspect that when server reports to many errors in one day, it may contribute to the problem. log file can grow big time...
Posts: 7985
Rebuilding the image when it's first accessed is a huge undertaking -- I think you're underestimating how hard it is. For starters, we have no way of knowing when an image is accessed because that happens at the Apache2 level, we only know when a PHP page is accessed which refers to an image. So if, for example, you have a page with 9 thumbnails on it but the thumbnails haven't been generated, you have to build all 9 thumbnails *before* you render the page because you won't know the exact dimensions of the thumbnail before you draw the page (because things like the square thumbnails module can affect the thumbnail dimensions). So now your page loads will not only be really slow, but the framework has to be heavier to a) deal with the fact that your underlying data files are missing and b) figure out how to correctly generate them. This will sound very familiar to you if you used G2 -- that is exactly the way we went in G2 and I can tell you: it was very hard and resulted in a ton of code.
In the forums I don't hear a lot of people complaining about the stability of the rebuild task. I'm sure they're not thrilled with it, but look at the forums and the feature requests, they're asking for a ton of other features. If there's a particular failure that you can isolate that we can fix, I'm totally willing to fix it, but a general "make it more robust" isn't helpful if I can't even make it fail. On my test environments, it doesn't fail. Tell me more about what exactly is going wrong and I'll make it better.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
@bharat: Ok, few fixes then:
* let's make it that number of images to be processed is real (not just initial count) and adjusted when task is paused/canceled
* solve famous "one item left" problem
* make rebuild of the item/album a standard operation, no extra module should be needed just for that and it would solve load problem as only one item/album would be affected if necessary.
* enhance [save] function that it would fix folder structure if necessary (thumbs, resizes)
* add code execution protection so if such unfortunate event occur it would not bring down entire engine
* I have no idea why (as there is no errors), situation when process is stack in "Starting up" step should be resolved.
From the code, g3 mostly falls back to default error handling... this is not good coding practice. try/catch better to be used, especially for the parts when execution is potentially resource intensive and/or could harm overall server health - rebuild is one of them.
Posts: 7985
I get that you're frustrated, but I need specific actionable tasks. I can't just "make it better" because I don't know what to do. If there's a specific problem, I'll not only fix it, but I'll fix everything around it that I can find. But I am not able to reproduce any of these problems and I'm not excited about bloating the code adding random error handling for situations that I don't know are occurring, it just makes the code harder to maintain. I am happy to improve this but I need specific, reproducible problems to fix. To address your points:
1) As far as I know the number of images to be processed is real- every time you go to the Admin > Maintenance page it recalculates that value directly from the database. If there are problems with that, I've been unable to reproduce them.
2) I don't know how to reproduce the "one item left" problem. If somebody can reproduce it for me, I'm happy to fix it.
3) I think this is counter to our philosophy. Very few people need this feature, it doesn't make sense to have it show up on the context menu for every album when it'll never get used. Why is there such a demand for this feature? I highly doubt that people use it much.. can you make a case for why it should get so much attention?
4) The Fix task can fix a variety of things -- what needs to be fixed in this case? Can you provide specific scenarios for where the folder structure would need to be fixed? I don't know what needs to be done in this case.
5) As far as I know, we've done this. What's missing?
6) Again, I don't know what the problem is here.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
and I was trying to get back to my happy place
if I knew exact problem, I would probably fix it myself. what we are discussing here is a review of the code for save operation (which generates resizes/thumbs, right?).
you know that a) it takes forever to process high volume of items - reported on many ocasions by many people
b) as soon as server is "overworked", import along with "simple" processing of "dirty" items would fail miserable... try to minimize available resources for the php process to simulate the situation for high number of items.
status_message is not updated
it was mentioned in another thread and may be related to "random" logic involved for selection of item to process. at some point, random query would not give you correct item to start
it is an admin function, more or less. but it would greatly benifit the functionality, as it would not require rebuilding entire list of items to fix a few.
we have resque module which works, I think, opposite way.
repair function need to validate that item tree has corresponding photos, and then if there are missing resize/thumb folders, recreate them.
I have had strict error mode enabled and it would prevent rebuild 99% times. I have disabled it and was able to rebuild it. so if php environment has it enabled by default (latest version of php?), we have possible problem
well, you know the code. what would prevent code to go from "starting up" to "processed 1 out of ..."
Posts: 2436
Going through the code for graphics::generate :
a) it may be useful to break up process in two steps - separate loop for thumbs and for resizes (or even 3 steps with separate step for movies). Progress would be more visible and process would have higher chance to succeed overall.
b) there is a possible bug - album with no cover - dirty flag would not be reset as function exists prematurely - would this be a case for "one lingering item"?
c) just optimization question - why rules are requested for each operation for each item (since it is one DB read for each)? why not to do it once for the whole batch or cache it as you collect the rules?
d) crazy question - would it be possible that when getimagesize is called, image file is still being written/locked by the library and therefore preventing file data access? potential lockup?
e) out of efficiency - why thumbs are built against full image and not a resize? Full image is ~2Mb (in my case), when resize is 92Kb. yes, granted, order would be important, but code can enforce it easily
gallery_task.php : rebuild_dirty_images():
a) is $ignored count properly taken in consideration here
why not just take
$task->percent_complete = min((int)(100 * ($completed + count($ignored)) / $total_count), 100);Posts: 290
the rebuild_items section could be modified/re-written to use my modified scheduler module to run on the crontab rather than through the browser. and it could be limited to only running a handful of images on each execution to keep load down. granted this will slow down the rebuild process, but for items where a thumbnail/resize hasn't been generated yet, you could have a place-holder "thumb not generated" image in it's place. $item->thumb_url() function should be able to detect the existence of the image file it wants to give you, and spit out a url to a place-holder instead of the actual file (which obviously doesn't yet exist).
just my $.02 worth.
Dan
danneh.org :: Gallery3
Posts: 2436
I have one item stuck right now.
What do you want to know about it?
Posts: 7985
I made https://sourceforge.net/apps/trac/gallery/ticket/1571 to track this issue for 3.0.1
re the "one item left" bug, can you describe what you're seeing? I assume that when you go to Admin > Maintenance it says that there's one item that needs to be rebuilt, but when you run the rebuild task it doesn't resolve it. Is that right? If so, then run graphics::find_dirty_images_query() and make sure that it's returning the 1 item:
$dirty_count = graphics::find_dirty_images_query()->count_records();If that returns a count of 1 (it should), then use the last query that it ran and then let's look in the database to figure out which item is stuck:
Kohana_Log::add("error", Database::instance()->last_query());Then run that query in the db and let's find the stuck item and see if we can figure out what's going on.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
Is 1
Posts: 7985
Can you run that query in the DB and show us what comes back?
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
records_found=1
as for item list
does produce item id (id=NNNN) which then failed with error
where xxxxx.jpg is in thumbs folder...???
why is thumb image is referenced here I do not know
original file does exist
Posts: 7985
Sorry I should have been more specific, I'd like to see the entire item. Try this:
Note that \G at the end -- that gives us more readable output. It definitely looks like something is wrong with that image, so let's figure out what's going on.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
PMed
Posts: 7985
Thanks, Serge. I'm up to my ears getting the replacement organize module working:
http://gallery.menalto.com/node/100045
I'll try to get around to this in the next couple of days.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 7985
Ok, I'm assuming that the stack trace above correlates to the item that's not built yet. The item that you sent me in the PM has the path "pxxxx/bxxxx/9x.jpg" so can you check to see if you have a file called var/albums/pxxxx/bxxxx/9x.jpg?
The failure is happening in the composite call so this is probably watermarking related. It's possible that your composite toolkit is unable to handle watermarking this particular image. What graphics toolkit are you using, and can you share the source image with us? My guess is that the composite operation failed resulting in the graphics toolkit spewing out an empty image and then the next operation in line failed because there was nothing to work with. If so, then that's http://sourceforge.net/apps/trac/gallery/ticket/1283 which is a difficult ticket to fix - we have to get into the Kohana graphics libraries and rewrite enough of them to be more robust.
Anyway. I agree with you that this area needs to be improved, but I simply don't have the resources to completely overhaul this with the time and priorities that I have. Without good test cases, that will only make it worse. As folks find concrete, reproducible issues I will absolutely fix them and gradually this whole area will get better.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
could you remove file reference please...
and, yes, original file exists
I had that happening in the past without watermark as well
simptoms were similar
ImageMagic
why then error says that there is no thumb image?
if so, shouldn't it just fail with thumb build and record that?
Judging by the ticket, we are coming back to original statement that image processing need to be reviewed to make it more error protected, intelligent
until then, you would hear from unhappy people...
Posts: 2436
And I would consider this more important than Organize module
just my PoV...
Posts: 7985
I understand what you're saying. I read every single forum post and reply to a large percentage of them, so I pretty much spend *all* my time right now hearing from unhappy people. I have to prioritize my time -- for every person who's unhappy about rebuilding issues, there are 20 who are unhappy with organize ccompletely failing. In this release, I've done almost all of the coding:
https://sourceforge.net/apps/trac/gallery/milestone/3.0.1
With the time I have, I must focus on concrete issues that are fixable. You and I know there are much bigger/deeper problems that we need to tackle to make theming better, improve embedding, fix authentication, rebuild some of the core libraries, etc to take G3 to the next level.
In the meantime, I would be more than happy to fix this problem if you can run the failing composite operation by hand and show me what the command and arguments were, and what error you got back. Armed with that information, I can at least make that part of the process a little more robust.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
I have gallery in the state when it is impossible to fix last item
you can give me code with any necessary debug hooks and I can try to run it for you so necessary information is collected
in the mean time you can satisfy other unhappy people
I care less about complexity of the theming as I think I have overcome/accustomed to majority of it by now. it is complex project with great potential, it is just pitty that graphics operations, which are one of the core functions, would be in such state... off course it is just my install and nobody else seems to run into it... or, should I say... yet...
And thank you for 2 of my tickets being reviewed in 3.0.1
Posts: 7985
Assuming that you're using ImageMagick, you have to go in and add logging calls before every exec() call in the ImageMagick driver so that we can get an idea of what commands are getting run.
Edit system/libraries/drivers/Image/ImageMagick.php and (for example) in the composite function, add this extra line:
218 public function composite($properties) 219 { Kohana_Log::add("error",escapeshellcmd($this->dir.'composite'.$this->ext).' -geometry ' . escapeshellarg('+'.$properties['x'].'+'.$properties['y']).' -dissolve '.escapeshellarg($properties['transparency']).' '.escapeshellarg($properties['overlay_file']).' '.$this->cmd_image.' '.$this->cmd_image) 220 if ($error = exec(escapeshellcmd($this->dir.'composite'.$this->ext).' -geometry ' . escapeshellarg('+'.$properties['x'].'+'.$properties['y']).' -dissolve '.escapeshellarg($properties['transparency']).' '.escapeshellarg($properties['overlay_file']).' '.$this->cmd_image.' '.$this->cmd_image)) 221 { 222 $this->errors[] = $error; 223 return FALSE; 224 } 225 return TRUE; 226 }Then in the logs we'll see exactly what operation it's running. But since compositing happens after other operations, you'll need to trapdoor all of them so that we can get a complete picture of the operations leading up to the failing one. Then we can build a small script that just calls these functions directly and see what error imagemagick is throwing, and from that we can hopefully figure out how to fix it.
If I can reproduce it, I can and will fix it.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 129
Following this thread for some days now: very interesting.
My 2 cents: Serge is right!
The famous one-item-left bug exists (have seen it multiple times on multiple installs) and the rebuild option is not satisfying (rebuild 'all' causes huge load on big installs and will break sooner or later). And if the folder structure breaks (e.g. due to a broken move with the organize module) you are in deep troubles.
I'm not writing to complain, I just want to point out that Serge is not the only one thinking like this.
Given the huge contributions of Serge I'd rather see some of his concerns addressed then feature requests of xyz that has no clue about G3.
Let me know if I can help somehow.
Posts: 7985
Let's see if we can track down this specific issue, and perhaps when I fix it I can fix a class of similar issues...
The problem with the graphics toolkits is that it's really hard to reproduce the wide variety of failure cases and then do the right thing. For example, if the ImageMagick composite operation is actually failing here, then what should we do? We can't generate the image that the user wants because they want a composite. Right now we skip the image and say that it's broken, but that also is a pain because it hangs around forever nagging you and you can't fix it. Generally what we try to do is to help the user understand what changes they have to make to their graphics toolkits to avoid getting into this situation in the first place, but that totally depends on the exact failure...
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
nothing recorded
Posts: 7985
Did you trap every function, or just composite? You'll have to do this for every function in the ImageMagick driver (assuming you're using ImageMagick, else you'll have to do it for whichever driver you have active).
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
looks like I have no choice but to fix it myself...
I have tried to reapply watermark and now I am stuck with just 10% of pictures processed and process is dying any time I am trying to rebuild.
old saying is that "rescue from drowning is by person himself"...
Posts: 7985
Somebody needs to debug your setup to understand what's going wrong. The instructions I gave above should work (but you'll have to extrapolate for the rest of the driver). I can make up a debug driver for you to try, or (much faster) if I can get ssh access to your site I can debug it in place to figure out what's going wrong, without damaging your live site.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
debug driver would do. I think you may need to consider start debug trace from task level and not just driver itself
ImageMagick version 6.2.4
Posts: 7985
Ok, try unpacking http://www.menalto.com/.outgoing/gallery/ImageMagick.php.zip in system/libraries/drivers/Image then re-run the task. You should get a ton of log lines in your var/logs. We'll need to identify the last set of operations from that and then run those by hand to figure out what's going on.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
placed it, but I think I have bigger issue in the task itself, as "rebuilding images" dialog stays in "starting up" state and does not move
nothing in error log
Posts: 7985
Ok, try replacing modules/gallery/helpers/gallery_task.php with this version:
http://www.menalto.com/.outgoing/gallery/gallery_task.php.zip
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
wow, we have movement...
have you added something more than just debug?
few sample lines sent via PM
Posts: 7985
Nope, just debugging statements. Sometimes when I add a debugging statement and an operation that was failing mysteriously starts working, it's because PHP was unstable/crashing. Check your system error logs, perhaps Apache2/PHP segfaulted. Debugging statements can change when (and if) the crash occurs because it introduces some I/O. What version of PHP are you using? Sometimes also an older version of a bytecode accelerator can result in wonky behavior like this.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 7985
..also, this type of behavior underscores why it's so hard to make this code stable -- if PHP itself is unstable in some way, it can be very hard to work around it reliably.
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
we are not out of the woods yet... we just succeed to push operation further
Host name: Linux infong 2.4 #1 SMP Wed Nov 4 21:12:12 UTC 2009 i686 GNU/Linux
Operating system: Linux infong 2.4 #1 SMP Wed Nov 4 21:12:12 UTC 2009 i686 GNU/Linux Linux infong 2.4 #1 SMP Wed Nov 4 21:12:12 UTC 2009 i686 GNU/Linux
Apache: Unknown
PHP: 5.2.14
MySQL: 5.0.91-log
Server load: 10.19 10.49 9.42
Graphics toolkit: imagemagick
Posts: 7985
Totally. Do you have a new log for me to look at?
---
Problems? Check gallery3/var/logs
file a bug/feature ticket | upgrade to the latest code! | hacking G3? join us on IRC!
Posts: 2436
not exactly...
it was working for few hours and sometime during that it died
I want from 915 to 502 unprocessed items
I have canceled the stale task, backed up log, deleted it from the server as 1.5Mb is a little to big
reruning it
Posts: 2436
I sent you a log for item 91
Posts: 44
I don't know if any work has been done on this topic since the last comment. I just wanted to add that I've been having problems with this as well. I didn't have problems with Gallery 2, even when I chose to resize all the images to 2 different sizes and new thumbnail sizes. I trashed my Gallery 2 install by accident and upgraded to gallery 3. I did a fresh install via Fantastico, manually updated it, and imported the 306 images via server add. I kept getting "306 of your images are out of date" message, and kept trying to update the images. I'd hit the update link, see a dialog box and walk away from my computer. Come back some time later (sometimes hours, sometimes the next day), and it would get hung up somewhere. Hit pause, mess with the site some more, all the while the message "306 of your images..." displays at the top. Right now, I'm running the rebuild process for the 7th time, and I've only got 27 out of date images according to the maintenance task page, of course it says 306 at the top of the page. Also, the notification saying my Gallery was out of date also didn't disappear until I manually hit the "check now" button on the dashboard.
Posts: 25961
As a general rule we have had nothing but issues with users that have used that install method for G2 & G3.
Please start a new thread after installing/upgrading and experimental version. Don't use GD as a image toolkit.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team