Integrating Gallery with Amazon S3
neilhand
Joined: 2006-03-27
Posts: 4 |
![]() |
Hi all, I didn't see this mentioned on the site during my searching so hopefully this is not something already covered. I currently use Gallery to host an online photo album. Since space is becoming an issue i would like to store the images using the S3 web service from Amazon. My first thought was to create a quick and dirty PHP+flash website, but then i started to wonder if this is something that Gallery could be expanded for. So i have a few questions. 1) Is this something already being looked at Any pointer would be appreciated before i start digging through the code. Best Regards |
|
valiant
Joined: 2003-01-04
Posts: 32509 |
![]() |
no, there's nothing available or planned for g2 <-> amazon s3. s3 isn't that special. the general case of remote storage is sure interesting and has been discussed in these forums from time to time. there's no relazted solution there yet, but possible approaches have been discussed. just a few notes without repeating myself too much: |
|
mmonter
Joined: 2006-07-25
Posts: 1 |
![]() |
I would like to see this kind of integration as well. I have well over 20GB of photos I'd like to post online. S3 appears to be an economical solution. Please keep us posted on your progress, Neil. |
|
evanbwhite
Joined: 2006-08-15
Posts: 1 |
![]() |
I too would like to see this kind of integration between Gallery and S3. In my mind, S3 IS something special in that it gives a really affordable remote storage option, much more affordable that most others. I too would like to see any progress on that, Neil, if you are able to tweak gallery to configure it for remote storage. Best, |
|
kzo
Joined: 2004-02-02
Posts: 9 |
![]() |
has there been any work done on this? |
|
neilhand
Joined: 2006-03-27
Posts: 4 |
![]() |
Unfortunately not by me. My baby daughter is keeping me too busy, and any spare time has been used to do programming for my wifes design business. It is however something that i would like to get back to when the opportunity arises. |
|
majortom
Joined: 2006-01-01
Posts: 40 |
![]() |
I think that implementing a more general remote storage interface would be more valuable than one specifically for s3. /carmi |
|
aarora
Joined: 2004-08-05
Posts: 6 |
![]() |
I was thinking about this myself until I came across this thread, just recently. My problem is that I like my hosting provider except that it's not cheap to get more storage. And any hosting provider with cheap storage either does not have the features I want or have enough reviewers posting how crappy the service is. So, Amazon S3 seemed to be a really good choice. One problem with Amazon S3, or any remote storage, is that it costs money for the bandwidth. And any hosting provider also charges for the bandwidth. So, if Gallery handled the data transfers through redirection, it would cost double. So, ideally, the image transfers would interact directly with the remote storage provider than through Gallery. For downloading, a remote storage URL could be generated pretty easily. (Would IE complain about mixed URLs in an SSL environment?) For the tools which use the remote API, another API would have to be created to get the storage URL and then the images downloaded directly. I have to read up on the Amazon S3 protocol to see how easy it would be to use simple URL redirection. However, upload seems like it would be harder to implement. Since every remote storage implementation would probably have a different API for storing the data, it probably would be hard to avoid using Gallery to redirect the image. I would love to volunteer here at implementing this in the next month or so once things settle down during the holidays. I don't have any experience with touching Gallery code, but I'm sure I can push my way through and some up with some prototypes. Any suggestions on how to start? |
|
valiant
Joined: 2003-01-04
Posts: 32509 |
![]() |
i haven't thought seriously about remote storage for gallery yet, but i can give you some starting points. - see my post earlier in this thread: http://gallery.menalto.com/node/46771#comment-177307 - there is an existing module which uses the Coral CDN (http://www.coralcdn.org/) to offload bandwidth / load from the Gallery server to a world-wide content distribution network. but this won't help you with saving disk space. the problems you'll be facing: i haven't thought of a good way to deal with that yet. but as i said. if diskspace is your problem, you probably have the wrong webhost. |
|
aarora
Joined: 2004-08-05
Posts: 6 |
![]() |
I did some research into web hosts that give 50-200 Gb. For everyone that looked reasonably cheap (less than $10 a month), I would easily find a website with all of the issues with that webhost. I even signed up for one only to find out that any of my request tickets took days to get resolved, which was pretty annoying. Second, alot of them didn't have the features that I was looking for. So, needless to say, the right webhost, IMHO, is hard to find. And I got really tired of looking around, especially when I was really happy with the webhost I was already using. I forgot about the thumbnail regeneration issue. That's an interesting problem. But I would assume that it would have to be as you said, Gallery pulling the image to the local server. So, to solve this issue, it would require a cache on the local server of some arbitrary limited size, so that this is not always expensive. I'm guessing that when people generate thumbnails, they deal with a small set of images as they are adjusting the sizes, etc. I would think that this remote storage solution should solve the bandwidth issues as well. Amazon S3 has the ability to use Bittorrent. Thanks for the info. I'll take a look at the GalleryPlatform class. I am guessing that I would have to create a different interface anyways since there are cases where some of the files will need to be stored locally and some remotely. |
|
valiant
Joined: 2003-01-04
Posts: 32509 |
![]() |
tried dreamhost? they give you tons of webspace and i heard mostly good stuff about them. |
|
aarora
Joined: 2004-08-05
Posts: 6 |
![]() |
I looked at them. and then I found http://www.dreamhost-sucks.com/ then i decided to wait. |
|
valiant
Joined: 2003-01-04
Posts: 32509 |
![]() |
Thanks for bringing that site to our attention. We'll look into the issue and have our webhost affiliate manager deal with the situation. EDIT: |
|
tomkerswill
Joined: 2006-07-28
Posts: 2 |
![]() |
One of the good things about S3 is that everything's stored in multiple data centres. You don't really get that if you've got your own server or a regular hosting provider. I like the idea of storing my photos on S3. One thing I thought about was setting up s3infidisk on my local server, to store the data on S3, but have it mounted as a local drive (using fuse fs). This is then symlinked to the gallery data directory. So to gallery, it looks as if all the photo files are local. One disadvantage would be the double-bandwidth issue. However, you could rewrite the URLs to point to S3 easily, but then continue to use the g2 data directory when uploading photos and regenerating thumbs. Tom |
|
skunker
Joined: 2005-02-04
Posts: 344 |
![]() |
Hi Guys, Hope that helps. |
|
tsawicki
Joined: 2007-02-26
Posts: 3 |
![]() |
throw me into the camp that would love to see s3 support in gallery. i don't want to go through the hastle of moving hosting providers when all i need is cheap storage. and s3 is cheap for hosted top tier storage. for work, i have used it before and it's a great solution. s3 support would definitely improved the commecial applications for gallery as well. |
|
anthropocentric
Joined: 2007-01-27
Posts: 4 |
![]() |
Count me in as another person requesting that Gallery support Amazon S3! |
|
anthropocentric
Joined: 2007-01-27
Posts: 4 |
![]() |
If you would like to add Amazon S3 support for Gallery, please vote for this feature here: Click on "Vote by id". The ID is: 1646294 |
|
airjrdn
Joined: 2007-05-01
Posts: 1 |
![]() |
I'd like to see this feature added as well. |
|
hubbard
Joined: 2007-07-16
Posts: 1 |
![]() |
Me too. I work at Amazon and I love Gallery and S3, but I have not yet heard of an integrated system for them both. I will check on this from my end. Tom Hubbard hubbard@amazon.com206-266-3806 |
|
dennylam
Joined: 2007-08-04
Posts: 1 |
![]() |
I use JungleDisk, an disk mount / backup application that uses amazon S3. Is there any way to hook it up to gallery2 ?? |
|
digitalgimpus
Joined: 2003-11-22
Posts: 15 |
![]() |
Not likely using JungleDisk since it uses a somewhat proprietary way of storing data (and it encrypts data). But if gallery2 support s3, no reason why your gallery install couldn't hold things in a temp directory, and upload to s3 (then delete the temp). That way you can use any upload method supported by gallery2. |
|
myobie
Joined: 2006-05-30
Posts: 18 |
![]() |
I would love to see a amazon s3 module. I was thinking about using it as a backup solution. Options: I would love to see any integration w S3 tho! |
|
UpTheToon
Joined: 2006-12-01
Posts: 11 |
![]() |
Any further developments on this front? Looks like a great idea. |
|
mpascal
Joined: 2005-09-27
Posts: 2 |
![]() |
I'd like to see S3 integration too. I have two servers at Datapipe now but they charge $5/GB for backups which is cost prohibitive for large scale media storage. Those cheap webhosts IF they offer any backups they often backup on the same disc. Here is a glimpse to Dreamhost's "backup" procedure. |
|
JustinHoMi
Joined: 2002-08-17
Posts: 42 |
![]() |
I'd also like to see S3 support. It really could cut costs, and give us many more choices for where to host our galleries. Right now we're limited to the companies who oversubscribe their services. |
|
JustinHoMi
Joined: 2002-08-17
Posts: 42 |
![]() |
Here's something interesting... a MySQL storage engine for Amazon S3. |
|
nervusvagus
![]()
Joined: 2008-06-11
Posts: 11 |
![]() |
Gallery2, s3fs and Amazon S3 |
|
pc2s
Joined: 2009-10-12
Posts: 1 |
![]() |
I also want this feature. |
|
nivekiam
![]()
Joined: 2002-12-10
Posts: 16504 |
![]() |
pc2s, Read the full thread and links. nervusvagus wrote:
Gallery2, s3fs and Amazon S3 ____________________________________________ |
|