File storage

Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Sat, 2009-08-08 13:01

I've started to play with G3B2, and I think there is still one more simplification over G2 that should be done : file storage.

Right now, G3, as G2 and G1, mimics the photo/album organization at the filesystem level.

I must confess, that I really don't care how my pictures are named on the disk, I don't see any reason why I should change/see the names of the pictures and folders as they are on the filesystem. As far as I know, those are not parameters I can chose on Flickr, Picasa web and other Photo galleries.

I would prefer Gallery to manage the file storage:
- set the filename to something that does not cause any problem (special characters and so on).
- organize the files in a B-tree or any other mechanism that will ensure the access to the pictures is fast.
- do not map albums to the disk. Albums are logical views. Pictures could belong to 2 or more albums.
If done that way, we would have an easier UI (go and explain my mum why she has to give a folder name), we would have less problems with files uploads, porting G3 later on to other systems will be easier, and albums could become more powerful.

Well, just my own 2 cents!

floridave's picture

Joined: 2003-12-22
Posts: 27300
Posted: Sat, 2009-08-08 17:34

There was lots of discussion on this at the G3 sprint.
I don't think it is going to chanage now as development has gone down the path it has already.
Since some people use gallery as their photo archive and some users where very adamant about keeping some logical file structure this was the path taken.
I am sure there was a few technical reasons as well.

Entities vs. Maps
* Get rid of locking
* Get rid of serial number
* Basically, remove the separation between the two types.


Blog & G2 || floridave - Gallery Team

Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Mon, 2009-08-10 15:30

Well I would not mind if this choice did not have some functionnal impacts:
- no possibility to have one photo being inserted into several albums.
- display of an obscure parameter for all elements (name of the object on the filesystem).

I understand that some use G2 as a photo archive, however, one must note that even G2 is not a perfect solution for this:
- filenames are modified. They are not modified the same way depending on which interface is used to import the file. One can not rely on G2 to keep the filenames unchanged.
- original files might be modified by Gallery (rotate command for example).

I understand this might be too late to do it now, but I wish you could count my voice for such a modif in G4.

bharat's picture

Joined: 2002-05-21
Posts: 7994
Posted: Mon, 2009-08-10 17:01

Hi, Lapinoo. Thanks for the perspective! On the flip side, if we go that route then it really damages interoperability. Even though we don't get it perfectly right, having a file system hierarchy that mirrors your logical hierarchy makes it much easier for folks to understand when they try to look under the covers. The developer community is an important factor here.

To address your specific points
1) While it's true that when we go this route, albums cannot be logical views we do offer tag albums which effectively allow you to specify arbitrary logical groupings

2) It's unclear to me that you're going to get significantly faster filesystem throughput with a different file structure. Remember that our common user is on shared hosting with wildly different filesystem configurations.

3) We can minimize how much people notice things like the folder and file name. We do this already by automatically filling in the folder name field based on the title. We are going to take this further by detaching the url from the filename:

It's not that it's too late to do this (we could do it now if we chose to). It's more that we actively decided during the design process that we do not want it that way.
Problems? Check gallery3/var/logs
bugs/feature req's | upgrade to the latest code | use git | help! vote!

nivekiam's picture

Joined: 2002-12-10
Posts: 16504
Posted: Mon, 2009-08-10 17:14
- original files might be modified by Gallery (rotate command for example).

This has already been addressed by rWatcher, he wrote a "keeporiginal" module. I don't know if it's only specifically looking for the rotate operation or if it will know if some other operation happens. But it works pretty darn well.
Like Gallery? Like the support? Donate now!!! See G2 live here

rWatcher's picture

Joined: 2005-09-06
Posts: 722
Posted: Mon, 2009-08-10 18:20
nivekiam wrote:
- original files might be modified by Gallery (rotate command for example).

This has already been addressed by rWatcher, he wrote a "keeporiginal" module. I don't know if it's only specifically looking for the rotate operation or if it will know if some other operation happens. But it works pretty darn well.

It's only looking for the rotate operation. As far as I can tell, this is the only time when the "full sized" image is modified.
Should this change in the future it should be relatively easy to expand it to cover other operations though.

nivekiam's picture

Joined: 2002-12-10
Posts: 16504
Posted: Mon, 2009-08-10 19:16

Thanks for the clarification and thanks for the contributions. The ones I've used or played around with work great.
Like Gallery? Like the support? Donate now!!! See G2 live here

Lapinoo's picture

Joined: 2004-05-08
Posts: 378
Posted: Wed, 2009-08-12 16:56

Thanks for all your feedback.

I'm going to provide you with some more information on the way I used G1 and now G2.

I host my pictures and the pictures of a few friends.
My pictures are all managed and processed outside of Gallery. I only use Gallery to display them. So for me, the tool I am looking for is a gallery-like tool in which I can upload (and re-upload if I reprocess a RAW file), which is nice looking (ie. with a few neat themes like some produce for Wordpress), easy to administer and that allow me to make the pictures accessible to friends and visitors in a fast way and simple to navigate. I do not edit pictures within Gallery (whatever toolkit gallery uses, none will be as powerful as Lightroom or Photoshop), I tag pictures on my machine so tags are to be extracted from the EXIF information.
My friends take advantage of the service which is simple enough to use, unlimited, free and comes with dedicated assistance. However, there are not as picky as I am for the quality of the interface. They just throw the pictures they want to make available online.

I like the concept of managing pictures and albums independantly because it gives me a lot of flexibility to organize the pictures and invite people to browse the gallery. Each come for a specific purpose ans is not going to look for a picture that is located in an album not directly related to their visit (they don't search, they are lazy). I could have used links for pictures in G2 but it requires too much time to manage, I in fact limit myself to using album links which is not perfect, but better than nothing.

So, from my perspective, Flickr would be a perfect fit if I could chose a UI that gives a better place to pictures. I like the service, but the UI, remains more community oriented than picture oriented. Zenfolio would be good too, but it would mean additional costs to me (I would still have to maintain my server for other purposes).
No perfect solution yet ;)

This explains why I am in favor of not bidding pictures to albums.

Regarding performance issues, they might be negligeable, but as far as I remember UNIX (inode based FS) directories behave slower and slower when they grow. For instance, if 5 years ago some of my friends came with less than 100 pictures to create an album, they start to import more (the biggest one I have is almost 1000 element big). So maybe in 5 more years, they will try to create 10000+ element albums. For such albums, storing 10000 elements into one directory might not be the most efficient.
Maybe this problem is no longer a problem on newer FS. Still Adobe Lightroom uses some volume balancing method to manage the previews it creates for all imported pictures. So, if they chose to do it, I think there is still a technical reason. Since my LR catalog can be seen as a bigger Gallery, maybe some concepts can be reused there.

I am not challenging your choice. Maybe (I don't have any statistics), most people care more for having a straight organization of pictures in albums, and in that case this point is irrelevant.

My biggest complaints about G2 are:
- UI is not perfect, but still is much better than G1. I wish there were more Pedro Gilberto ;)
- Managing albums and pictures independantly is a real pain. I wish I could do more. It would allow me to reorganize my picture and offer a better navigation of my gallery.
- Limited interface (GR API) : not all properties are available, impossible to re-upload a picture that has been reprocessed without losing the metadatas (views, comments) or having to do it manually from the Web UI.
So to be honest, I wished G3 would addressed the second point (Point 1 seems to be achieved, I still hope the REST API will be there for me to update my Lightroom2Gallery plugin to have a better integration of my Gallery to Lightroom).

Keep up the good work !