Server Add module enhancement
|
siburny
Joined: 2012-03-27
Posts: 55 |
Posted: Tue, 2012-03-27 01:43
|
|
Hi developers and users! To better suit my needs I am modifying Server Add module, and just want to get a sense if anyone else would use my modifications. As we know Server Add module doesn't verify if images were already imported or not, so I want to add extra checks. For my needs, full path would be sufficient validation. But if someone else is interested, I can add extra MD5 hash verification. I can also make it configurable via Administration panel (by-default, we can disable that). What do you think, guys? |
|
Posts: 27300
go for it. I'm interested.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 55
Excellent. I have a couple of thoughts, so can you please give me your opinion
0) I will add configuration Checkbox which is unchecked by default (current behavior).
1) Duplicate check: in my case, full path is enough, but for some people it might not be. I can add MD5 column to "server_add_entries".
2) If duplicate check is enabled, I will not create duplicate albums.
3) What happens when file is changed? Theoretically, it should update the image.
Posts: 27300
0) I'm sure that you know that forge forms have some weird behavior with radio buttons, so check boxes are the way to go.
1) If you can do a MD5 check that would be cool. Some people might add items in different albums that are the same, some check to see if it exists already and some warning/override wold be cool.
2) Some option here I think would be beneficial. Not sure how a UI wold work. Perhaps later in development.
3) I think this is a rare case and time should not be spent on this until later.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 55
1) Done
so I will look into this.
2) MD5 is done, skipping is done.
3) Must look into this now, because even though the images are skipped, empty albums are still created because folders were scanned.
4) I fall into that rare case
Posts: 55
Dave, I need your advice. I am done with changes and ready to commit them to my fork on GitHub. How do you like me to proceed then? Would you be willing to test it?
Posts: 27300
Commit your changes to git hub, request a pull request from Bharat to gallery-contrib.
I would like to test this so attach a .zip file of your module and I (and others) will give it a whirl.
If you are comfortable with writing some docs you can use your username/passoword on the codex to create a module page. If you need help with that I can help, just let me know.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 55
Let's test it first
Since I have a fork on GitHub I will commit to my fork, and then send you a zip file with that module. Sounds good?
Posts: 55
Dave, here is the ZIP file with just "server_add" module in it. Let me know what you think. I also committed changes to my repo: here is the commit https://github.com/siburny/gallery3/commit/7d1fb3b8961fccb814df01cfd62cf52762007f30
Posts: 183
Hey Siburny,
I'm interested, too! I've been thinking of a similar (albeit less-sophisticated) idea, too. My thought process was to check full file path and file size. If there's nothing in the identical full file path, proceed with add; if there's something there but it's a different file size, proceed with replace; if there's something there with the same file size, skip.
Also, if I might suggest, there are a couple other features I'd like to see added. In fact, I've already hacked my own personal copy to include most of these features...
- Remove the breadcrumbs in the server_add dialog. Their information content is largely duplicated by the title of the dialog, and with my theme if they're too long they get in the way and screw up the file selection UI.
- Add the ability to define in an admin window a few extra preg_replace commands on the title names before they're auto-inserted. This is a big deal for me, as all of my album directories start with a date or date range and I want to auto-add the slashes (e.g. "20120401-20120403 Sample Album" turns into "2012/04/01-2012/04/03 Sample Album").
- Add the ability to include movies in server_add. Currently this isn't possible, but is taken care of by the videos module. The videos module lets you select what file extensions you like, and there are no limitations (for example, I include PDFs). Because the videos module uses a modified version of server_add at its core, it isn't too hard to incorporate. This would allow you to add whole directories with mixed filenames in a single stroke.
In the interim, I've downloaded your file and will take a stab at testing it in the next couple days. Thanks!
Take care,
Shad
Posts: 693
Can I ask a question about this module? It sounds really neat and something that I was looking for. However the reason I often want to replace an image is not the image itself, but its metadata. When I upload an image, its title, description and tags are imported from IPTC. I manage these fields in a DAM offline, and if I find errors I want to correct them offline and then replace the versions in the gallery. without losing comments and view counts. I can more or less manage to do this now, but I can't get the tags updated. I have to do this manually, which is tedious and error-prone.
Does this module help with this? I have a hunch that the answer is no. If not, could it?
U-G
Posts: 55
The answer as it stands right now is no, BUT I will be working on the same feature very soon, because I need to be able to "update" images without losing comments and view counts. Initially, I will just update the image data file (not sure if metadata gets reloaded in that case). Then I can look into metadata updating.
Posts: 55
Also, I think server_add can add "movies" already (it scans for "flv", "mp4", and "m4v"). We can probably expand that list if needed.
The other suggestions looks good too:
1) I use "date" in the folder naming too
2) Will look into breadcrumbs
Posts: 693
I'd be a real happy camper if it could. I haven't had the time to hack this myself.
U-G
Posts: 2
The enhancemnt works perfekt! Thanks.
I`m working now on a solution for start the "add" process by a cronjob. But its not easy to understand the PHP Code. I have a small solution to call the ajax request by simply open an specific url. But js works only in an browser, because its a custom html which calls the "js add function" by "on document loaded". If I would call the url by a php script in a cronjob - nothing happend. Clearly beacause js works only in a browser on clientside.
Im an classical frontend developer with less knowlage in php.
If somebody has a solution yet or can help me for one, it would be perfekt. Also Im willing to support it by an donation.
So far. Detecting Dublicates is really an enhancement. Thanks.
Posts: 55
For those who is interested (and have similar setup as I do), I changed algorithm to fit my needs.
1) I added second Settings parameter called "Process updates?".
2) I do not verify md5 before doing import, I just verify full path (in my case it's always unique)
3) If that file was already imported, then I use md5 checksum to see if it changed (I do not trust last-modified time). If it was indeed updated and updates are allowed, I update existing image record (using set_date_file which re-generates thumbnail and re-reads tags(?))
If updates are not allowed, then that file is skipped.
4) It doesn't create new albums if the same albums were created
Since I changed the way module works, I am thinking of renaming it so it won't be confused with official server_add. I might name it "syncfolder" or something, and then commit to gallery3-contrib where all other extra modules reside. I would have change some stuff around because I don't want to use the same table name, so two module could co-exist. As soon as I do that, I might be able to create a cron version of that module, so the process could be automated.
Any ideas?
Posts: 2
Hey siburny,
if can help me with my cronjob solution, I can make a 200€ donation for that. Only if we can find an solution in the next few days.
Can u help me?
Posts: 55
I sent you a PM a while ago, but you never responded
Anyway, I renamed the plugin into "FolderSync" so I will not interfere with official server_add plugin. Any one interested to test? There is a small warning though: it will not work for already imported images - only NEW images and albums will be tracked
Posts: 2
Hello, where can I download this plugin? Thanks.
Posts: 55
If you the module that this discussion started with, then it's attached here: http://gallery.menalto.com/node/105847#comment-389323
However, since then I created a separate module (folder_sync), which behaves (slightly?) different that how we envisioned in this thread: basically, it tries to MIRROR in the Gallery what you have on your disk.
The code is here: https://github.com/siburny/gallery3/tree/master/modules/folder_sync
1) Admin UI to manually add folders was removed
2) Cron (or command line) is the only way to add folders/images
3) Removed reliance on Task class, since there is no UI
4) Cron job was re-structured (no step1, step2, ...)
5) It processes folders/images in batches (right now 500 per run)
6) Best way is to schedule it every 60 minutes.
7) It's not finished yet; I am "working" on delete capability: if original file was deleted from disk, then Gallery should remove it too.
Posts: 2
Thanks very much. I was looking for your folder_sync module. Thanks again for the module.
Posts: 55
Please be aware though, that code contains bugs and is not ready fully ready yet. I should wrap some extra development soon, so if you can wait, I would
Posts: 27300
siburny,
Thanks for development on this. Let me know when I should create a codex page for you.
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 55
Here is update: been running this module on my installation of Gallery3 with about 30,000 images - works as expected!
Changelog:
* Added Admin option to allow deletion (by default, items are NOT deleted).
* Improved limit handling (batching).
* Cleanup the code a little.
Updated code is on GitHub: https://github.com/siburny/gallery3/tree/master/modules/folder_sync
Posts: 183
Hi Siburny,
Thanks for the work on the module! Certainly, something that fundamentally syncs folders would be a huge help.
One quick (and perhaps stupid) question: for the authorized paths, can you use local paths (e.g. "file:///C:/Photos/etc")?
Thanks,
Shad
Posts: 55
Actually, all paths are local ONLY: this module synchronizes Gallery and local folder(s) on the server which hosts your Gallery.
Not sure about specifying drive letter though: all my folders start with "/" (e.g. /home/siburny/uploaded/)
Posts: 55
I just tried creating a page myself, but got denied
Can you create it and give me permission to change: I just spent 30 minutes writing the text :-D
Posts: 27300
http://codex.gallery2.org/Gallery3:Modules:folder_sync
You will have to login to the codex using your forums username and password. (separate login)
If you get stuck just send a PM with your chnages and I can make it happen.
Cheers!
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team
Posts: 55
Thank you, Dave!
Codex page updated; new thread for bug reports/support requests is created: http://gallery.menalto.com/node/109161