Large image store import
|
Helo
Joined: 2010-02-10
Posts: 2 |
Posted: Wed, 2010-02-10 15:06
|
|
Hello, We have been working with gallery for a few years now with great success. This has led us to consolidating all of our image storage into gallery. Unfortunately the data set we are working with is rather large, 900GB, and the built in import and galleryadd were insufficient for our needs. Does anyone have any suggestions about what direction we should be looking in order to import this data? (We've done some preliminary work with the api but that is still early on) Additionally we would like to separate off the image storage and code into two locations. The image storage will be put on a san to allow for expansion. We are wondering if this is easy to do or will it require code changes? Thanks, |
|
Posts: 4342
The g2data directory needs to be part of the local filesystem within php. If you SAN can do that then there are no code changes necessary.
I can't offer any advice on the other questions.
Posts: 16504
Beg and plead with http://www.care2.com/ to share their modifications?
I have a feeling they have a bit more than 900GB of photos stored under their heavily in-house customized version of G2.
Other than what Alec mentioned, try searching the forums for Amazon S3
http://www.google.com/search?domains=gallery.menalto.com&sitesearch=gallery.menalto.com&q=amazon&op=Search+with+Google
http://gallery.menalto.com/node/77816
http://gallery.menalto.com/node/46771
So there's one option that may or may not work for you or you might be able to modify it to work for your environment.
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here
Posts: 2
Thank you all for taking the time to give me your suggestions. The information was really helpful!
Mark
Posts: 5
Hi,
I have a big amount of pictures on my server (hundreds of GB) and I would like to view them using Gallery3 without letting the import module of the gallery copy them again in a different folder. I would like to point the gallery folder to the folder where I already have the pictures and let the gallery process it in a way that would not duplicate the pictures. Maximum let gallery create some smaller versions of the pictures, and of course, create entries in the DB for the albums/pictures.
Thanks.
Posts: 16504
G3 does not work like this. Perhaps someone will create a symlink option in the future, but it's not going to be a core feature. If you need this feature set, use G2 and use the symlink option when using the Add From Server method.
____________________________________________
Like Gallery? Like the support? Donate now!
Posts: 5
Thanks for your reply.
It's a pity not to have it in the newer versions.
I've found the "symlink" option. Nice idea, but still:
a) it creates a symlink for each file, which is a huge overload. Is there any reason for not making a symlink of the whole 500GB root dir instead of making few tens of thousand symlinks?
b) do you know if the write permission is really needed (755)? In my case I would like to give only read perms (555), to ensure that none of the pics are being changed or anything.
c) where are the small sized pics being written? I did not find them in the "cache" dir of the gallery2.
I thank you for you time.
PS. Is it easy to modify this plugin to allow the same nice feature on gallery3?
Posts: 16504
See Bharat's comment here:
http://sourceforge.net/apps/trac/gallery/ticket/907
For G2, when using the Local Server there are checkboxes on the right of the table of files/directories to choose to import for using the symlink option.
____________________________________________
Like Gallery? Like the support? Donate now!
Posts: 5
You are too fast
You answered while I was editing the post
Here is the update version:
It's a pity not to have it in the newer versions.
I've found the "symlink" option. Nice idea, but still:
a) it creates a symlink for each file, which is a huge overload. Is there any reason for not making a symlink of the whole 500GB root dir instead of making few tens of thousand symlinks?
b) do you know if the write permission is really needed (755)? In my case I would like to give only read perms (555), to ensure that none of the pics are being changed or anything.
c) where are the small sized pics being written? I did not find them in the "cache" dir of the gallery2.
d) creating the cache is horribly slow. I've done a small test with only 1000 pics, and it took 30 minutes... (yes, I have all the default plugins turned on). Is there anything I should do to speed this up?
I thank you for you time.
PS. Is it easy to modify this plugin to allow the same nice feature on gallery3?
Posts: 16504
a) No, each file needs a pointer. Gallery doesn't read directories, it needs to import the photos and do stuff.
b) yes, write permissions by the server are needed, period.
c) which version? In g2, g2data/cache/derivative and they are numbers and renamed and only the database knows what file belongs to what image. In G3 /var/thumbs and /var/resizes
For G2, basically don't go poking around in g2data, if you start mucking around in there beyond just reading and looking, you will most likely break something.
d) If using G2 and you don't have things cached, browsing G2 will be slow. Uninstall unused plugins, but if Gallery is just creating the thumbs and resizes most of those shouldn't be touched. 30 minutes is a long time, get a faster box
Seriously though, that all depends on resources on the box, size of images, etc. Since you keep asking about symlinks, how/where are is your g2data? Is it on a network share? NFS and windows shares are SLOOOOOW
PS, I sort of answered your question above, see Bharat's comment in that ticket. It might be easy, you'll need to dig into the code, or jump onto #gallery on IRC and see if someone is around more familiar with the code base than myself to give some quick pointers. Either way, you're not going to get around points a and b
____________________________________________
Like Gallery? Like the support? Donate now!