Module: Rebuild items
|
floridave
Joined: 2003-12-22
Posts: 27300 |
Posted: Mon, 2010-07-26 05:03
|
|
http://codex.gallery2.org/Gallery3:Modules:rebuild_items A module that marks items as 'dirty' in the database and lets Gallery rebuild those items, on a per album basis, via the admin section. If you experience issues with gallery not rebuilding items, please start a new thread. Dave |
|
Posts: 7994
Woot! Nice one, Dave.
---
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git
Posts: 30
Dave, I tried using this module today to repair several broken and new touched up images. I uploaded the new original image to the appropriate /var/album directory and then performed the repair on that album. It worked for the thumbnails, but not for the "resized" images.
Color corrected in Picasa ... thumb and large image looks good, but the "resized" image did not change and is still the old dark gray color (new ones have a rich black): http://egrips.com/gallery/promo/sports/UlsterSpecialTredz
Same scenario as above
http://egrips.com/gallery/promo/sports/GoldsGym_Proof2
Same scenario as above
http://egrips.com/gallery/promo/sports/Suns_GoSuns_Proof1
Same scenario as above, but this new image was also cropped much tighter, as the original had a bunch of white space below it, which you can still see in the "resized" version.
http://egrips.com/gallery/promo/sports/Card-Hanshin-Tigersx5
Hopefully I did something wrong, but I checked all the boxes to make all versions "dirty".
I'm hopeful this module takes the place of the G2 "ReUpload" module.
Thanks for your work!
Matt
Posts: 27300
Only the first task in the list that is checked will be performed.
So you have to visit the module again from the album options and then check the re-sizes box not the thumbs box.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 30
The last photo was also cropped to a smaller dimension. It's thumb and re-sized photo look great and reflect all the changes of the "new" original, but the enlarged image is distorted, and still enlarges to the old dimensions of the original file ... stretched in both directions.
http://egrips.com/gallery/promo/sports/Card-Hanshin-Tigersx5
Matt
Posts: 27300
The images look fine to me I bet it is some caching issue.
resize: http://egrips.com/gallery/var/resizes/promo/sports/Card%20Hanshin%20Tigersx5.jpg?m=1281479624
full: http://egrips.com/gallery/var/albums/promo/sports/Card%20Hanshin%20Tigersx5.jpg?m=1281479624
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 30
Not a cache issue ... 3 different cleared out browsers, and 3 different computers, all same result.
The database shows the original size as well, so the Rebuild Items module isn't updating the items table with the new Original image size during the rebuild.
1420 x 699 is the size of new original image
1600 x 1126 is the size in the database, and the size of the stretched-out image space, when you click the "resized" image. Also the enlarged image is below the viewing area and you have to scroll down to see it.
Posts: 27300
Did you change the original item?
If you did then this is not the module for you.
I'm now a bit confused as to what you have done.
A new thread should have been started but since we are down this road already we can carry on.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 30
Yes, as stated in my original post, I updated the original item (via ftp), then ran the module. I don't know what's under the covers, but it's very close to being exactly what I need (and hoped) it would do. There is no other way, that I know of, to update an image without totally deleting it, and all the tags, and descriptions (starting from scratch), in G3 ....... but this is very close!
Do you see what I'm describing when you click the resized image?
Posts: 27300
I now understand what you are after.
A new different module could probably be developed but is not in the scope of this module.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 30
Thanks Dave! It's still a great module and EXTREMELY useful to us, just not when original items have been cropped / dimensions change.
I appreciate your 18,539th post!
Posts: 51
hello dave
its maybe great module for my gallery or this module can fix my problem in here http://gallery.menalto.com/node/96957
or you have suggestion to fix this problem my gallery
Thank you very much,
Regards,
Putra
Posts: 67
it is possible to add a feature to this?
The ability to include all sub-albums would be great.
Thanks
Posts: 21
The rebuild items option is available to those with edit permissions, but I think it only works if you have admin permissions? This results in a "Dang... Something went wrong!" page instead.
Would be good to not show the option if the user is not an admin, or display a page explaining that admin rights are required.
Posts: 27300
sykong,
Thanks for the report, I have fixed the module in the fist post.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 27300
Updated the module to try to detect thumbs/resizes that did not get generated properly during upload.
It compares the thumb/resize size against the size of the item in the DB and you can rebuild/mark dirty those items only.
see first post for new version.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 1857
@floridave: Many thanks for this module!!
I think that the exif module pulls the keywords from the exif data and posts them to the database as tags when initially uploading images. When rebuilding exif using the module, the tags don't get updated. I don't know if anyone else noticed (or cares), but it may be useful to note somewhere that the tags don't get updated.
Also, I noticed that the module only executes the first of the checked boxes; maybe they could just be radio buttons (so you can only select one)?
Posts: 1857
Now I'm seeing that the module picks up new data, but not updated data. (Example: If a caption is added to a photo that previously had one, it will be updated; if a caption is changed/modified for a photo, the new caption is not picked up.)
Is this expected behavior or is something on this end interfering with the module?
Posts: 27300
I wanted to use radio buttons but the Kohana/forge form builder we use does not have radio buttons.... or at least ones that work the way radio buttons are expected to work. If you see a forge/Kohana method let me know. I could change it to a drop-down but I thought that the text would be good enough. But as we all know not everybody reads or is just click happy.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 1857
Wow. I didn't know about the radio limitation. (That also changes what I said at http://gallery.menalto.com/node/99842#comment-371960 )
And I was one of the non-reading, click-happy people when I first used the module. Lol.
I agree that the text is fine as is.
Posts: 27300
Not sure I follow. Can you give a user story? The module was really developed as a last resort for those who have hosts that kill processes during the upload process and we/Kohana don't throw a exception to stopped processes in this case.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 1857
I upload a photos with captions but discover some error that requires I change all of the original captions. I edit the captions offline and replace the original photos in my var/albums directory. I then run the rebuild items module, resetting exif data and going through the maintenance task. I'll find that the original captions are still there; they have not been replaced by the new ones.
Given the original intent, this may be outside the scope of the module. For me, the module's been a big help replacing images with smaller, web-optimized versions. I was also trying to use it to modify all of the exif captions and to update all the associated tags. (I didn't want to do this via "edit photo" for each photo because I want the new captions to be available in the original photo so that it's also searchable locally.)
Hope that makes sense; got a little wordy.
Posts: 693
This may be of interest to me, since I am about to do some replacements. I wonder if the captions and tags were cleared from the DB before the rebuild then perhaps it might pick up the new caption. If this is the case then there should be a query that does this. Perhaps also the logic could be built into the module. If this works for the caption then it should also work for tags.
U-G
Posts: 27300
All this module does is mark dirty the item then the maintenance task is run to fix that.
I don't think that this module should delete the data in the data field of the EXIF table for the item.
It starts to get complex really quick here for my skill set.
A new module could be developed to do this. I will ramble here for a bit....
If you replace a item with different EXIF/IPTC data and want it to be populated then other things have to happen as well....
The EXIF module populates the title and description field as well as the tags. Say you re-upload an item and some of the other fields change..
Should the old tags be removed? Should the title of the item now take the title from the new re-uploaded image what about the description?
I think this is such a small use case that it might not be worth the effort to make sure all the different ramifications of screwing with the the DB even more.
I can help somebody that might use this module as a starting point but don't feel that the extra coding and checking and a good UI to go with is is worth the effort.
Would just deleting the data field be good enough?
Thoughts?
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 693
I agree that it's hard to justify spending time on something that's used only infrequently. However when it's needed then you miss it.
I prefer to manage my images in a Digital Asset Management (DAM) program and upload images with metadata already included and have G3 import this information. Occasionally I find errors in my captioning and tagging and would like to upload the corrected version. But at the moment it seems like it's hard to get G3 to pick up the new information if I simply overwrite the version in the albums folder and run rebuild.
Greydragon has an option to mark items as "dirty" and as well has an option to reload EXIF. Yet it doesn't pick up edited captions and tags either. I might ask Serge about that first if I can't work out why. The advantage of the module is that it provides targeted rebuilding, whereas the Greygragon option is DB-wide, which is ok for me at this stage but isn't an option you'd want to run all that often on larger databases.
I will comment further once I've had time to do a little digging.
U-G
Posts: 27300
If it is such a rare occasion then why not delete the original item and re-add it? Seems to me that writing a module for a rare occasion when a perfectly good work around is available seems like a waste of time. Don't get me wrong, if there was a real need for this then I am all for it, I just don't see a real need for it. If your process of adding meta data is flawed then I don't think that Gallery should be to tool to fix that.
Perhaps another more targeted module is in order?
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 27300
Let's start a new thread with your specific requirements and work flow. So as not to distract from this thread.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 693
Ok, but first I want to correct something I said in this thread. I must have done something wrong in the past, because I just tried it again and it worked. If I re-upload an image with a changed caption, and then in Greydragon I tell it to mark the EXIF as "dirty", and then in maintenance I reimport the EXIF, I get the new caption. Your module is better because I don't have to run it for the entire gallery. It's a really neat module for these situations.
The relevant code (in Greydragon) seems to be:
db::update('exif_records') ->set(array('dirty'=>'1')) ->execute();And I'm wondering if something similar could be done for tags. If I want to pursue this I will start a new thread.
U-G
Posts: 27300
Looks like the same to me:
db::build() ->update("exif_records") ->set("dirty", 1) ->where("item_id", "=", $form_id) ->execute();except I only do it for the specific item
or where I get the items in the album as well:
db::build() ->update("exif_records") ->set("dirty", 1) ->where("item_id", "in", db::build()->select("id")->from("items")->where("parent_id", "=", $album_id)) ->execute();Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 693
Yes, that's right. I didn't make that clear. You are doing the same thing. What I was wondering was whether it's possible to use this sort of code to mark tags as dirty in order to trigger some sort of rebuild. I used Serge's code as an example simply because it is slightly simpler.
U-G
Posts: 27300
undagiga,
can you post a url to 2 images that have the original and then the changes you make to the EXIF/IPTC info.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 27300
We only chnage the caption if the caption is blank. If the caption is not empty we don't change the caption. This prevents the EXIF/IPTC fields overriding the existing data supplied by the User-interface ( edit item ).
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 693
I'll send you a PM with some sample images, but first I want to be sure that I understand what you want to see. You want to see two images, one with that has had the caption changed via an EXIF rebuild, and one that hasn't - is that right? What exactly will this tell you?
U-G
Posts: 27300
I would like
24 images. One that was the one you added to G3 and one that has the edited info that you manually added later.Another pair of images with a title not-in/in the EXIF/IPTC data would be usefull as well.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 693
I'm not sure that we're on the same wavelength, but perhaps we are. I haven't "manually" added any information. All my images have IPTC pre-populated before I upload them, so that Title, Description and Tags are automatically imported. In the case of "Title" this requires a well documented hack as IPTC isn't imported to title in the base code. In a few cases I have subsequently changed the image including its IPTC and uploaded it directly to the albums folder overwriting the existing copy, marked it as "dirty" and also the EXIF as "dirty" and run the maintenance tasks. My recollection is that the caption was updated (but not the tags).
So what I propose to do is create two test images, upload and import them (server add). I will then change the IPTC for one of them and overwrite the image in the albums folder. Is this alright? Do you want me also to run the rebuild tasks on it or not?
All my images have titles, so for the second pair, you only need an additional one without the title? Actually this is going to be trickier, as I have added to the import hack for titles so that G3 merges two IPTC fields for the image title - IPTC Object Name and IPTC Source. I guess I can ensure that both these fields are blank.
I'm happy to help as always, but want to make sure I understand.
U-G
p.s. I'm happy with the current behaviour for title and caption and wouldn't want to see it change. I.e. they are only reimported from EXIF if EXIF is marked as "dirty". This makes it easy for me to maintain metadata outside of G3 and keep it sync via upload and EXIF rebuild. My issue is only with tags - I'd like to be able to do the same with them.
Posts: 27300
What I want is a clear use case so I understand what is happening and what is not.
Then tell me a user story and send me the before and after photos for this.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 27300
https://github.com/gallery/gallery3/commit/99fd65ff02d17cbb3ec936b4bdbf2a8c6b0b73e3
See if that fixes your issue.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 693
First a clarification. I've been doing some systematic testing and I've realised that what I said above is wrong (again). If I upload a replacement image with a different IPTC Caption, then the G3 description is not updated. As you suggest, there is a test whether the caption is blank, and if it it isn't then it isn't replaced.
What happened in my previous testing was that I had changed the "IPTC Object Name" and the G3 title was updated. This occurred because I inserted the hack to import object name into title outside of the test for whether the field is blank or not, and so it's always reimported.
Now if I want to change the behaviour for caption I can see how to do that - looks fairly easy. What I can't see is how to change the behaviour for tags, so that different IPTC Keywords in a replacement image leads to different G3 tags.
The latest release didn't fix the problem. I'll PM you a URL with test images.
U-G
Posts: 27300
I think I understand now. The tags module is on the list to add some features like editing/combining tag names so I am a bit reluctant to do any work on that untill that is done.
I will put it on my list of thing to do. Thanks for the detail in your PM.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 693
I understand why you want to move cautiously, but some pointers as to where I might look to hack the code in the meantime would be helpful.
Thanks.
U-G
Posts: 693
I embarrassed to admit that the release of 3.0.2 passed me by. I don't know how this happened. It looks like it was over Easter, so perhaps I was distracted.
I see that there was: "Added support for splitting and joining tags". Does this mean that rebuilding tags via this module is now feasible? If so, is this something that someone of my coding abilities could attempt?
I ask because with the release of the Tag Albums module we have the chance to create virtual albums. I need to insert new tags and for me it is safest and most efficient to do this offline in my database and then re-upload to var/albums.
Thanks
U-G
Posts: 27300
Don't know never tried. Give it a try and report back.
With virtual albums I have not tried that yet either. There is only so many hours in the day.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 693
Ain't that the truth.
I'd be glad to hear from anyone with any tips on how to do this.
U-G
Posts: 27300
Updated the module to gather tags from the item as well.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 6
Hi,
I've implemented re-scanning all items for new tags in an album by a simple db query and then calling your rebuild_item::get_tags() for each item found. Works for me (tm).
Patch should be attached.
Sincerely,
Roland
Posts: 6
Addendum to my last post: The "items" table contains everything including the albums (I thought it contained only images), so the database query should include
->where("type", "=", "photo")or some similar measure...
Roland
Posts: 27300
Thanks for the patch.
I have updated this module to version 6 with your changes.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 21
I guess this might be more of a feature request, but is there a way to mark all oversized images and thumbnails as dirty? (And automatically rebuild without having to give my photographers admin access?)
Posts: 1857
Not sure what the relationship is between you and the photographers; not sure of the relationship between the photographers and the website. The request sounds a bit complicated and likely not a good idea for the average user (because then anyone could potentially rebuild images at any time: drains system resources and could also cause issues if more than one person attempted at a time on same photos). Maybe I'm misunderstanding?
Why do the individual photographers need to rebuild the images of albums they don't have permissions on? Perhaps there's another solution that works?
Posts: 27300
There is now: http://codex.gallery2.org/Gallery3:Modules:replace
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 27300
sykong
Perhaps you need to find the reason why the items need to be rebuilt. A resource issue perhaps? This is now a module that should be used very often. If you users add items and they don't get built when they are added that is what you should focus on to get fixed.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team