I successfully imported 20,000 photos from Gallery2 into Gallery3. All was well. I imported at 165px thumbnails since it said that would go faster (took a LONG time and I had to resume often, but importing is done once so that's ok). Then I changed all my thumbs to 200px as that looks better. That finished.
Next, I noticed one of my albums was displaying funny. I don't know how to describe it but it was just totally whacked. It would display thumbs but also a fullsize image (24MP!) in the album index. Other albums were not affected. I decided to create sub albums to hopefully fix that. It got worse. I began to organize but when I tried to move a bunch of photos, it would fail so I learned to copy in smaller chunks. Then I created a second sub-folder (to reduce the 1200-odd photos in the root of this album). Then I got big problems.
Looking at my /var, it seems the DB lost sync with the actual file system, as it doesn't even show the 2nd album I created, nor does it show that the first sub-album has all the photos I supposedly copied. So now I'm bummed out because I just want this to work.
I tried the "fix" thing twice but that didn't help. Here is the log from today. I can try to take screenshots if my descriptions are lacking. I'm afraid I have DB corruption though I have no idea what caused it, how to fix it, or how to prevent in the future.
Please help!
Quote:
<?php defined('SYSPATH') or die('No direct script access.'); ?>
2013-10-30 20:13:19 -04:00 --- error: Failed to convert placeholder for missing image: Age 12-17 months
2013-10-30 20:13:19 -04:00 --- error: Failed to rebuild thumb image: Age 12-17 months
2013-10-30 20:13:19 -04:00 --- error: Failed to convert placeholder for missing image: Age 12-17 months
2013-10-30 20:13:19 -04:00 --- error: Caught exception rebuilding images: Age 12-17 months
@todo UNREADABLE_FILE
#0 /home/kevin_moore/kevinallenmoore.com/photo/modules/gallery/helpers/graphics.php(255): photo_Core::get_file_metadata('/home/kevin_moo...')
#1 /home/kevin_moore/kevinallenmoore.com/photo/modules/gallery/helpers/graphics.php(223): graphics_Core::_update_item_dimensions(Object(Item_Model))
#2 /home/kevin_moore/kevinallenmoore.com/photo/modules/gallery/helpers/gallery_event.php(126): graphics_Core::generate(Object(Item_Model))
#3 /home/kevin_moore/kevinallenmoore.com/photo/modules/gallery/helpers/module.php(402): gallery_event_Core::item_created(Object(Item_Model))
#4 /home/kevin_moore/kevinallenmoore.com/photo/modules/gallery/models/item.php(416): module_Core::event('item_created', Object(Item_Model))
#5 /home/kevin_moore/kevinallenmoore.com/photo/modules/gallery/controllers/albums.php(133): Item_Model_Core->save()
#6 [internal function]: Albums_Controller->create('14')
#7 /home/kevin_moore/kevinallenmoore.com/photo/system/core/Kohana.php(331): ReflectionMethod->invokeArgs(Object(Albums_Controller), Array)
#8 [internal function]: Kohana_Core::instance(NULL)
#9 /home/kevin_moore/kevinallenmoore.com/photo/system/core/Event.php(208): call_user_func_array(Array, Array)
#10 /home/kevin_moore/kevinallenmoore.com/photo/application/Bootstrap.php(67): Event_Core::run('system.execute')
#11 /home/kevin_moore/kevinallenmoore.com/photo/index.php(116): require('/home/kevin_moo...')
#12 {main}
** moderator snip - we really don't need to see a bunch of repeating errors **
Posted: Thu, 2013-10-31 00:54
Posts: 27300
I would suspect that your server just halted the creation of the thumb of a specific image and the full sized image is used as the thumb. Without a working url is it had to confirm.
What happens is the server stops the process(s) and the image toolkit (GD or IM) just halts and gallery has no chance to recover or log an error.
The best remedy is to delete the image from the UI and upload it over. or you can try the rebuild items module and do one item at a time.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 37
I am trying to delete the album. It waits 60-120 secs then fails. When I go back into it, it's merely smaller. I have to keep deleting. I don't see any new entries in the logs from it. Maintenance mode is on though. Is that why?
Also, I still have a full assortment of the actual photos in my /var folder. Is it supposed to delete the photos when I delete the album??
Posts: 37
Thanks for the quick reply. That explains the funny formatting I was seeing on the first page of an album, because I guess the theme probably sized itself based on that large image.
But as I tried to create sub-albums, things got really screwed up. The flat file structure did not keep up with the DB changes. I am trying to delete this entire album but it won't let me delete the sub-album that is in the DB (I assume) but not in the flat file structure in /var.
I did turn CloudFlare off as it seems like these long processes make it all nervous so it throws back an error to the browser saying the server is down. When the DNS finishes propagating I will see if that makes at least deleting easier. I don't see why it wouldn't keep deleting in the background, though.
Also remember I had a lot of trouble moving photos in the arrange UI. I had to move small amounts or it would fail. Is that normal?
Sorry for all the questions...I had Gallery 2 for years and years and never this many issues. I like the simplicity of 3 but man it seems like it isn't nearly as robust.
Posts: 37
I disabled CloudFlare. Instead of the CloudFlare error I just got an HTTP 500 after trying to delete for a while.
This is the folder that seems to be giving me so much heartburn: http://www.kevinallenmoore.com/photo/index.php/D600/Ian
I created both of these sub-albums using G3. The 12-17 months sub-album NEVER created itself in /var. And it won't delete now. Is this all hosed?
Posts: 37
Oh yeah, it's hosed. I added an album and it think it's part of another sub-album...the pictures are uploading but aren't in /var and won't load thumbs or fullsize when I click on the album.
Go here: http://www.kevinallenmoore.com/photo/index.php/D600
Then Sam's Halloween Decorations 2013 and you'll see what I mean. It's just so broken.
Posts: 27300
I get a Bad_httpd_conf error
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 37
Sorry, turning CloudFlare off requires DNS changes, so they hadn't finished yet. Please try again!
Posts: 27300
Still the same for me.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 37
Can you try another browser? If that doesn't work, I could try to re-enable CloudFlare as I know that's not the issue here. But that requires another DNS change and however long for it to propagate.
I am going to turn CloudFlare back on, which will require another DNS propagation. Should take a few hours....sorry about this confusion. I appreciate your help though.
Posts: 37
Ok maybe third try's a charm. Please try again?
Posts: 37
I'm just going to delete the installation and DB and re-import from my 2.3.2 install. I have no idea what happened but I believe it is currently beyond repair.
Posts: 37
General question: are there best practices for ensuring the integrity of my Gallery 3 install (including the DB) over the long term?
For example, right now, my Gallery 2 install is importing into my Gallery 3 (some 20,000 photos and IIRC, ~218 total albums/sub-albums). I did set the gallery in maintenance mode to avoid the server having extra load while importing.
I am at work and it would be nice to check on the progress. Normally I'd just browse to my site, log in as the admin, and go to the maintenance tab to check the import progress. But even in maintenance mode, the albums are displayed to the admin. So that will have to load from the server before I can get to the maintenance tab. In general, is it a problem to try to view albums during a process like this (import)? How about any other rebuilding type process?
On moving files, should I be able to move hundreds of photos from one album to another with no issues?
I want to have the right level of expectation this time around...
Posts: 37
Still importing. It is on 14,800 of about 20,000. I did follow the tips G3 gave me and disabled EXIF and Search modules, though that doesn't seem to impact the speed. I also left the thumbnails at the default 200 (meaning it has to resize since my G2 thumbs were 165px). However, it has not failed once. Knock on wood, it has gone straight to 14,800 in one shot. I am encouraged.
EDIT: Stalled at just over 15,000. In the log, it tried to import two unsupported files (AVI and BMP) but I'm not sure if that's where it stopped or that was just the last logged thing. Resumed...
Posts: 37
Ok I am all up to date. Imported, EXIF, search...everything is up to date. Things seem to be working fine though I haven't really pushed it (still in maintenance mode also).
What I would like to do is move about 700 photos that are currently in an album into a sub-album. Is there any reason this shouldn't work through the Organize album interface? Is there a better way to do it?
Posts: 37
So moving 700 photos doesn't work.
Not sure how long it ran, but I came back to my computer and had the message "An error occurred. Consult your system administrator."
Looking in the sub-album I was trying to move 700 photos to, it moved 57 before failing.
I don't see anything in the logs pertaining this incident.
Should I just keep trying in chunks until all 700 are moved?
Posts: 37
Ok now I'm getting really frustrated. I moved all 700 in roughly 50 photo increments. Then I started another sub-album to organize more. It's now acting corrupted again. And when I looked at the first sub-album I just organized, it's actually corrupted, too. Let me turn off maintenance mode so you can finally see what is happening.
This is where I was organizing: http://www.kevinallenmoore.com/photo/index.php/D600/Ian
Specifically, at the end: http://www.kevinallenmoore.com/photo/index.php/D600/Ian?page=48
Now look at a sub-album: http://www.kevinallenmoore.com/photo/index.php/D600/Ian/7-to-11-months
But if you go to the last page (or any page with photos past DSC_0381, which was the last successful move according to what I see in my /Var folder): http://www.kevinallenmoore.com/photo/index.php/D600/Ian/7-to-11-months?page=53
Or try this sub-album: http://www.kevinallenmoore.com/photo/index.php/D600/Ian/12-to-23-months
It even causes corruption across the whole site, on albums I never tried to organize. That's REALLY scary.
Click on a photo. And look what happens to the folder structure at the top (and in the URL). This is the corruption I got last time. The DB and folder structure somehow just completely broke. I suspect this is a G3 bug as I've duplicated this problem twice in a row. Barring a fix (or guidance on how to avoid the issue), I think my only option is to, for the third time, delete the G3 installation, re-import from my G2 and just completely avoid arranging my photos. Or maybe organize in G2 then re-import? But I can't even be sure that will avoid the issue, since I have no understanding of what the issue actually is.
I really need to fix this. My site has been offline for a week now. Please respond.
Posts: 27300
I would suggest staying with G2 if you had no issues moving and rearranging large number of items.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 37
Dave,
After I posted that, I tried moving photos around. I selected a bunch of photos but it didn't give me a destination to select so I couldn't move anything. I had to laugh.
Also, the main thing that drove me off G2 was the Java applet stopped working properly (Gallery Remote) for uploading.
Aside from that, I always found G2 slow.
Are you confirming that I shouldn't move mass amounts of photos on G3, even if I only move 50 at a time? Is there anything else that is ill-advised?
It seemed like things were going fine and once I started moving in smaller batches, I no longer got the error message. But looking at my /Var folder, it is obvious that at some point it just stopped actually copying the JPG to the new folder (but it updated the G3 site in the DB). Could it be something on my hosting service causing the issue? Like inadequate memory for PHP stuff? Or is this a G3 bug?
EDIT: Ah, I couldn't choose a destination because I had renamed the Gallery 2 folder. I'll try doing my mass move in the old G2 site.
Posts: 27300
There is no Java applet for G3 that I know that works.
Could be any of the above.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 37
That's fine about the Java applet. I don't like Java anyway and the flash uploader in G3 is more modern, though I can't figure out how to get rid of the 7MB upload limit. But that's minor.
I just moved hundreds of photos in G2, FWIW (for all G2's slowness while browsing it sure moves photos around quickly!). I'll try to do mass moving in G2 then re-import.
Posts: 8339
BTW G2 has the same uploader http://codex.galleryproject.org/Gallery2:Modules:uploadify
-s
________________________________
All New jQuery Minislideshow for G2/G3
Posts: 37
I did try Uploadify. It didn't work, either: https://iglnma.dm1.livefilestore.com/y2piDQ9EpB1NAtKkMC8aXLl6ElLDmtVIQDkFs0GN2erdQTKQfmQ7QIc7yLjttCWS5s6FKTL78JaB6a8tFd883mY-PeuYtbIoP8_11SlMzFgmKU/G2_Uploadify_failure.png?psid=1
That's when I decided to try G3. I would like G3 to work as it is much faster, which was my #1 gripe with G2. But it also seems I am one working upload method away from staying with my old G2, for all the trouble I'm having with G3.
But as it stands now, I have nothing.
Posts: 8339
I had another user tell me uploadify had stopped working for him as well. So he pushed some updates my way.
I move that version to git https://github.com/suprsidr/uploadify
It didn't work for me. The old version still works for me though.
I have not had time to investigated the differences yet.
-s
________________________________
All New jQuery Minislideshow for G2/G3
Posts: 37
I tried that one. Different error. It did run through the upload (showing percent progressing to 100%) which it did not do on the other version.
https://iglnma.dm2302.livefilestore.com/y2prEtO3yqnuTIHr-bTGWN05fP7pM25FqWK69VvzTErNT_7NS3CUp4cFmTe-TeYUoLGUiaS8HJiawQoiINrQ_Gh3zG9pPTjMtTX7I2MIjzeQko/G2_Uploadify_failure2.png?psid=1
Posts: 8339
That is the error I got with his version too.
try my version and
add this to your .htaccess:
-s
________________________________
All New jQuery Minislideshow for G2/G3
Posts: 37
I did try that in my .htaccess on the other version and I was getting HTTP 500 errors. I am re-importing into G3 (4th try) at the moment so I don't want to mess with either Gallery site. But I will try as soon as I am able. Thanks!
Posts: 37
Dave,
Can you tell me why this is happening?
http://www.kevinallenmoore.com/photo/index.php/D600/Ian/12-23-months
I have not moved ANY photos using G3. I merely imported, and ran the search and exif modules. I added the highlight module. And I uploaded 50 photos to the album I linked to above. According to the /Var folder contents, all the originals, resizes and thumbs have been generated. Yet the album contents display like this. Guests do not have access to view the full size images yet G3 is trying to display them in place of the thumbnails. This behavior is extremely bizarre to me. Do you have any input on what is going on?
Posts: 27300
Your image toolkit fails to create the thumb and does not pass this failure to G3. During this process the image is just copied to the thumbnail from temp. Occasionally the image gets corrupted as well. It happens to me occasionally on a shared host if I push it with lots of images at the same time.
That is one of the reasons I wrote the rebuild items module.
As to why; I don't know. I would guess some resource limit as it seems larger images and lots of them seem to make it more reproducible. But I do know if I don't push it with lots of large items it does not happen.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 37
Dave,
Thank you for the reply. I am not familiar with the rebuild items module. It looks like this is it: http://codex.galleryproject.org/Gallery3:Modules:rebuild_items
I will give it a try on this particular album.
Posts: 37
Yes! That worked nicely. So that is a known work-around for uploading lots of items at a time. It seems there is no work-around for trying to move large numbers of items, and no way to "undo" either.
Going down to the lower priority migration issues, is there a way to auto-rotate based on EXIF?
Posts: 27300
http://codex.galleryproject.org/Gallery3:Modules:autorotate
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 37
Last time I tried Gallerymodules.com the site was parked...seems to work now. Thanks.
Posts: 37
Ugh. More problems. I uploaded like 25 shots (all portrait) to test the auto rotate. Most of the rotates failed and even after I ran the rebuild module, the resizes and thumbs are broken on many. I assume this is the same issue as the other one...the ImageMagick process gets killed on the server for some reason? By the way I did also try manually rotating individual images and that just corrupts the bottom portion of the JPEG (again, IM just fail halfway through processing the image). I tried another one just now and aside from taking like 20 seconds to rotate a single image, it worked. Workflow wise it sucks to have to rotate a JPEG that has already been marked by my camera but I could do it locally before upload.
That sure is limiting...I mean I don't think G3 is going to be viable for me with all these issues, though many of them are relating to ImageMagick. Are there plans to fix any of this stuff in future versions? Is there anything I can ask my web host to do on their end to help with this?
Posts: 37
Which .htaccess am I supposed to change? I added that to the one in my Gallery 2 root and I got the same HTTP 404 errors as before. Maybe it doesn't know where my G2data folder is?
Posts: 37
I asked my web host about the image processing issue. This is the response I received:
Just thought I'd pass this along.
Posts: 27300
So convert is a process that Image Magic and or Graphics Magic (not sure what one you are using but it does not matter) uses to convert the images. Looks to me like it was killed due to too-much ram (memory). We have no control over that so I don't know how the "scripts needing to be optimized.".
I guess what they are saying is that for the traffic and size of items you are using their servers can't keep up or they have the procwatch3 (processor watch) set too low for what you want to do.
I guess you have a few options:
1. You could approach the Image Magic and or Graphics Magic authors to see if they can "optimize". (not a chance)
2. Move hosts. (up to you)
3. Upload smaller items and use less resources on this host. 24MP seems to be a bit large for display on the web. (your choice and my recommendation)
4. Find a developer that can dig deep into the calls we make to the graphics tool-kits we use to see if we have made a mistake. (not me, kohana possibly)
5. Different plan on the same host that allows for more resources to be used due to the size and quantity of items you have. (you and your host)
6. See if there is any community members that can offer suggestions (community)
7. I'm sure i messed some other option(s) as well.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 37
Dave,
I have migrated my Gallery 3 installation to run under a new user and subdomain on my webhost that is not shared with my other sites (or even my Gallery 2 site). I have my camera set to 13MP instead of 24MP to ease general file handling overhead on both my PC and my Gallery 3.
Of course I am still running into problems with Convert (web host says I would need to upgrade to a VPS due to PHP memory/script ceilings and you say it's just JPEG resizing and it is what it is). But my question is about the Rebuild Items module. It works well when resizes or thumbs are too big, as it was intended. But I also have issues where the rebuilds or thumbs may be corrupted but aren't dimensionally illegal. For example, I have resizes that ended up at 200x200. The auto flag won't find them so I have to manually go into the list, find the offender and click Rebuild to mark it (which also reloads the page). This is incredibly tedious.
Where can I leave feedback for the Rebuild Items module?
Posts: 27300
I guess right here for feedback. I don't know how to detect if the 200x200 images are right or not or if those dimensions got added to the DB.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 37
I'm also confused about this convert script situation. Gallery 2 was never known for being speedy but I never had a problem (that I can remember) with image resizing. And I've been on this web host since 2006. So what's fundamentally different about Gallery 3 that causes JPEG resizing to be so memory intensive?
Posts: 27300
I don't know. We did not write the image toolkit code and I bet 24MP files where not extensively tested on 100s of hosts.
If you where happy with G3 way change?
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 37
First, the 24MP images didn't seem to pose a problem on G2. Second, I have switched my camera to 13MP anyway so that should no longer be an issue.
I switched because I wanted something more modern, lightweight, and fast to share my photos with my family and friends.
Posts: 37
Hi again. I've been searching and can't seem to figure this out. Is there a way to rewrite a direct linked Gallery 2 image to Gallery 3 URL compliance? I have hard linked some photos all over the place over the years and going back and looking, the links are all dead. So enabling an .htaccess rewrite would fix that. Obviously Gallery 3 uses the /var/sub storage method, so the links are definitely different. Thanks.