Module: Amazon S3

Announcing a new module available which allows you to synchronise your content with Amazon Simple Storage Service (S3) for either use as a CDN or backup purposes.

Version: 1
Blog post: http://www.danneh.org/2010/11/amazon-s3-for-gallery-3/
Download: http://www.danneh.org/files/aws_s3.zip
Codex: http://codex.gallery2.org/Gallery3:Modules:aws_s3

Update Jan 11th 2012: It's come to my attention that this thread has been locked for whatever reason. So, with that said, if you've got any comments, feedback or suggestions for this module, please direct them at my original blog post (http://www.danneh.org/2010/11/amazon-s3-for-gallery-3/). I'll keep this particular post up to date with new release info and whatnot, and use it as a public domain, as this forum thread once was, for advice, suggestions and bug reports. -Dan

ah, so spaces in the filename might be a problem with S3. i have read a couple of instances where this is an issue. i think in future versions i'll going to have to sit and re-write filenames that have odd characters/spaces in them. this module simply uses the existing filenames/paths as stored by g3, but for management purposes an extension on the items table or a new table for it might be necessary. i've got to deal with the issues with cloudfront first, but i'll pencil this in the v2 as well.

and lol - i just spellchecked them and there's an h missing. i'll fix that and push it into git later on.

but all in all, i'm glad you had a decent amount of success with it

Dan

mm strange. suggests that the sync didn't go as successfully as it thought. can you send me the log file in var/modules/aws_s3/log/aws_s3-<date>.log. this should have some insight as to why it failed. if you can't pm it, send it to my email: dan at danneh dot org.

Dan

Using Firefox's Firebug addon, I see that all the images get a 403 Forbidden. I went to my bucket and change the Permissions to include Everyone with List and View Permissions but that didn't help.

humm..very strange. if you view the contents of the bucket using aws's console (https://console.aws.amazon.com/s3/home), are the items there? (default under g3/<guid>/[fs|rs|th]/...)

hm, ok. from what i saw with the signature'd one, the request/signature had expired. now, for public galleries/albums, it shouldn't even be putting the signature onto the url as it's not required, so there's one issue to begin with. however, having a signature on the end of a public permissioned url shouldn't affect it operating.

check your server's time and make sure it isn't skewed too much (+/- 60 seconds). run "ntpdate -p4 pool.ntp.org" and check the response. obviously you'll need to be root to do this. i'm leaning towards this being the problem as if the time is skewed too much, the request/signature will have expired before it's been generated (in real-time/aws server time context, anyway).

if you can't set/sync the time on the server, try increasing the expiry time in the aws module's admin to something higher. i'd go in 5 minute (300 second) intervals until it works.

the g3 folder (and the guid folder beneath this, and the fs/th/rs folders beneath that) should have full public read permissions. the module won't create the bucket, but it will create the g3 and guid folders if they don't exist. aws doesn't have a folder heirarchical structure, all files are simply 'objects', and the g3/guid/th/etc.. is simply a prefix to an object. listing occurs by filtering objects with the same prefixes. now, i could possibly forsee an issue if the first item it uploads is a private one, and is given ACL_PRIVATE rights, and this may inherit right up to the root g3 folder. however, if you check (and change, if necessary) the g3, guid, and fs/rs/th folders to ensure they have public/everyone read/view permissions, that should be fine.

the more i think about it though, the time being skewed seems more and more likely.

Dan

When I click on the g3 folder, or the guid folder or the folders under it, I can't change their permissions. The only permission I can change is that of the bucket itself. It has the only permission for my own user to do everything (read, list, write, edit permissions). Should I add Everyone and give it View Permissions? Cuz I tried that and it didn't work.

yes, that's what i meant. view (read). if your content is private, (i.e; not visible by the "everyone" group in g3) then the content is uploaded with a private acl which gives only the bucket owner (you) read permissions. if you're seeing the image when you're logged into aws, then naturally you'll be able to see the images because you're logged in. as far as your bucket contents permissions go, if the content is public or private, only YOU should have list and edit permissions. if the content is public, "everyone" will have view permissions. if not, only you will have view permissions. the signature basically authenticates the client as YOU for a single item for a set amount of time. after which, the signature expires and the client is no longer able to access the resource.

the module is detecting that the content isn't publically visible, hence it appending a signature to the end of the url, which grants the client temporary access to that particular image until expiry time. you'll get a 403 if the signature expires, and this would only happen if either;
1. the signature was generated and somehow the client only requested the image from s3 more than 60 seconds later (default expiry time in the module), or
2. the time on the server is skewed more than 60 seconds behind the current time so that the timestamp that the module/library generates as the signature's expiry time, by the time it's delivered to the client, has already passed.

i stand firmly by the time skewed theory, as i've not had any problems like this during development/testing. i'll attempt to replicate it on my development machine later, but i firmly believe this is the cause. like i said, run "ntpdate -p4 pool.ntp.org" as root on the server. i can practically guarantee you'll get a < -60 second skew.

Dan

It might be early to tell, but I tried to create another bucket using S3 Organizer, and like the initial bucket I created, which contained an underscore (which was created using their AWS Management Console for S3), I tried to create another one and S3 Organizer didn't allow me to create it, saying I can't use an underscore! So I went to Amazon's docs, and noticed that I really can't. I have NO idea why it lets you create a bucket who's name doesn't comply with their own restrictions!

So I created another bucket with a name I want to use, and which followed their restrictions and while it is just at the thumbnails, I can see them already online and using Firebug for Firefox, I can see that it reads them from Amazon!

Maybe put it up on your FAQs?

I wouldn't go as far as adding code to verify the naming convention, but I would put a link, at the text you have were you require the bucket name, to their page:

http://docs.amazonwebservices.com/AmazonS3/latest/index.html?UsingBucket.html

This link would get updated by them, and thus would keep you from needing to keep your code up-to-date with their changes in the future.

OK I got a step further.
Apparently it works now - pending more testing ;-)

The problem was that my php did not have the cUrl extension.

I fixed this by using synaptic to install 'php5-curl'.

I found out by trying the "AWS SDK for PHP":
http://aws.amazon.com/sdkforphp/
From there I downloaded the sdk:
http://pear.amazonwebservices.com/get/sdk-1.2.0.zip

I then placed that in a subdir under my apache.
I copied the file 'config-sample.inc.php' to 'config.inc.php', and filled in most of the required data.

I didn't understand what was meant by 'Constant: AWS_CANONICAL_NAME' as it was not listed on the aws credentials-page.
I found this page, that helped me understand that my 'canonical name' is simply 'jakob'!:
http://www.bucketexplorer.com/awsutility.aspx

I then browsed to an aws-sdk-testpage on my local server:
https://server.net.lan/aws/_compatibility_test/sdk_compatibility_test.php

Where i was told that I could not use the SDK, because i was missing php-cURL (with SSL).

Regards, Jakob Hede Madsen.

that all makes sense, actually.

the call to curl_init() won't error in the httpd logs, but it'll error in the php_error.log (disabled by default). although i think g3 attempts to catch some php errors in it's own log files (var/logs/<date>.log), so there may be something in there along the lines of "call to undefined function: curl_init()".

i'm in the midst of working on version 2, including some more helper functions to get the bucket name syntactically correct, optimise the upload function a bit, and i'll add in a test also that will prevent you from activating the module if curl isn't installed, since the s3 lib isn't my code, and i can't really be bothered writing my own wrapper for it - and even so, i'd still depend on curl being there!

Dan

In the file
./helpers/aws_s3.php
Line 38:
S3::getAuthenticatedURL("danneh-org", $resource, module::get_var("aws_s3", "sig_exp"));

The hardcoded reference to your domain seems odd. What are the implications, and should I 'patch' this?

Regards, Jakob Hede Madsen.

lmfao! omg i can't believe i'm that stupid lol. implications? no private content will work because the bucket name is incorrect.

replace that line with this:

S3::getAuthenticatedURL(module::get_var("aws_s3", "bucket_name"), $resource, module::get_var("aws_s3", "sig_exp"));

I'll patch the git contrib repo and my site now.

Well spotted, thanks!

Dan

Regarding the nonfunctional slideshow.

I fixed the hardcoded reference as per your instructions.
I had hoped that it might fix the slideshow problem, but alas.

My observation:
As you can see in the attached picture, the Flash appears, and it loads the thumbs, but it remains black.
The statusline in the browser flickers constantly between 'transfering' and 'waiting' for s3.

What can I tell you about my AWS_S3 configuration?
It's located in Ireland.

Regards, Jakob Hede Madsen.

Another that you may already be on to:

When I upload images named like this "Photo on 2010-10-25 at 23.21.jpg"
They don't work.
The same images renamed to something like "p1.jpg" works.
The images are uploaded to the local server, but it is suspiciouly fast - probably because the step of sending data out to s3 is omitted.
The images - and their resises do appear under the dir 'var', but their image can't be seen in the album - only their name.

Regards, Jakob Hede Madsen.

danneh3826 is aware of this with this:

Dave

_____________________________________________
Blog & G2 || floridave - Gallery Team

I have just verified the problem with the disappearing images.

Images that are added to an album that has restricted access for 'everybody' to 'view', disappears when the restriction is removed.
Meanwhile other images that are added while the restriction is open, will remain in the album regardless of the state of restriction.

Regards, Jakob Hede Madsen.

jakob;
i have a feeling i know where that problem is coming from, regarding images disappearing. i think it's something to do with a missing hook-in within the s3 module which gets a signal from g3 that it's not attempting to move the items from one location in the bucket to another.

in version 2 i am addressing issues such as this, changing permissions from public to private (and vice versa), moving content around, and i'll take another look into the slideshow in both public and private album contexts to see if i can resolve this. i'll add these as known issues on the codex page, and this will at least remind me to address it in version 2.

if you want to only be able to upload full sized images as opposed to all types of images, i can put this in as a feature request. obviously i'll be prioritising the bugs and general issues first over feature requests, so this may not appear in version 2.

dan.

This is probably not of any help in this particular case, but it might somehow be related:
I (well actually a tester) experienced problems when uploading through the normal flash-upload-interface. Some pictures didn't complete.
I alleviated this problem by reducing the number of concurrent uploads from 5 to 2 or 3.
You can do this in the advanced settings.

Here is what I have observed:

Yes, the pictures are stored both locally and on S3.
This is great because it allows you to switch S3 off again, and your gallery continues without a hiccup.

It does mean that the upload procedure takes a very long time, and that the progress-bars in the upload-interface, completes long before the process is actually finished, which might confuse some users.

The upload-process has three steps:

1. Upload from client to server
2. Processing on server
3. Upload from server to S3

·

The progress bar is proportional to step 1.
Step three is potentially much more time-consuming than step 1 because it has to go out through the narrow connection, which the excersise of using S3 is meant to bypass later.
So you have to pay the penalty for the slow outgoing line from the server once at upload, in order not to pay anymore at all the subsequent views.

It's a thought that step 3 - the upload from server to S3 could be detached from the upload-procedure, and handled asynchronously afterwards. Then the gallery could serve images locally, until they had completed upload to S3.

Regards, Jakob Hede Madsen.

Well, there is nothing in your follow-up, that changes my perception of what you sought to express in your initial post.
Maybe you should try to read what I wrote again? And I'm sorry if I'm not expressing myself so eloquently, I am Danish.

I perfectly well understood that you are in the initial phase - I quoted you saying that - which is why I wrote
"This is probably not of any help in this particular case, but it might somehow be related"

Then you asked:
"What happens when I upload new images?"
And I replied to that question pertaining to the non-initial process of uploading new pictures.

I know I too can be dense, but I don't think the hat fits in this case.
I am not really the person you primarily want an answer from, but I am offering my input to the best of my knowledge.

Allow me to ask:
If you have outgoing bandwidth of 15Mb/s (I assume you meant Mega, not millibits) which I think is a lot, then why do you want to use S3 at all?
Of course if you have that kind of bandwidth from your server to S3, then you wont see the bottleneck I described, but maybe I'm making a wrong assumption when I think that when shifting to S3, it is always the case that one comes from limited outgoing bandwidth?

This also pertains to my earlier questions to Dan - which by the way you haven't really addressed yet Dan - about the bandwidth out of S3. I am only seeing around 300 Kb/s coming out of S3, and while that is better than my dsl-up, it's very far from impressing, and I would be able to get far higher bandwidth coming out of my webprovider servage.net(, who unfortunately have other bottlenecks, that makes it unbearable to run G3 at their servers).

What kind of bandwidth are you seeing CJ00TA?
I assume you're not shifting to S3 in order to see your delivery-bandwidth drop by a factor 50?

Regards, Jakob Hede Madsen.

@CJ00TA; there are issues still with the sync process that i've become aware of at some point or another since v1 was released. i am in the process of addressing these issues, but i have limited time available to me, so unless you want to contribute to the code, you'll just have to wait for me to do it. i have planned a handful of improvements for v2, which include the possibility of moving the initial sync task to a cronjob. (though i've not seen any mention of a cron within g3, but i'm sure there's something somewhere, or a way of running that task into the background). there'll also be the addition of a table to the database for it to log various items statuses as to whether it's been uploaded, and which items have been uploaded (thumb, resize, fullsize, all 3, etc), and the time it was last uploaded. the code will then use this table to determine whether to redirect the request to s3 or locally. this will also address the issue of a "failed task", and preventing the task from emptying the bucket so it would have to start again. this additional table will also allow it to fork the S3 upload task to the background and allow you to see the image(s) that you just uploaded prior to it being uploaded to S3. as an additional feature, the admin will be able to give you an overview of number of items vs number of items uploaded, number of failed uploads, the ability to re-upload single images (failed or not), and the ability to manually flag items as uploaded (thus allowing a manual upload process if need be).

@jakob; i really can't answer your bandwidth questions relating to S3. i'm not an amazon network admin, nor do i have any in-depth knowledge of how their system works. if you're only getting 300kb/s from S3, then i'd do a bit of bandwidth testing on your dsl line and perhaps see if the bottleneck lies in your dsl line as opposed to S3's multi-gigabit/second lines throughout the world. go to www.speedtest.net, choose a mirror that's closest to where your bucket lies (i.e; dublin for an ireland bucket) and run another test using the closest mirror to you and paste the results here. try also running a traceroute to "s3.amazonaws.com" and seeing what sort of response time you get from their border (amazon-gw.ip4.tinet.net, or words to that effect). i'm averaging around 60ms from my less-than-perfect uk based dsl line to their ireland nodes. downloading one of my larger files yields me around 650-700kb/s

HTTP request sent, awaiting response... 200 OK
Length: 3065436 (2.9M) [image/jpeg]
Saving to: `CAM_0142.JPG'

100%[======================================================================================================================================================>] 3,065,436    757K/s   in 4.1s    

2010-12-08 20:19:23 (731 KB/s) - `CAM_0142.JPG' saved [3065436/3065436]

so.... i don't know what else i can say. if you have a real problem with it still, get in touch with your isp's tech support and complain of throughput issues, or amazon themselves (who will most likely tell you where to go).

that's the entire idea of this module, S3, and CDN's in general. i've seen the *exact same* result after using this module on my gallery. since all the static/large content is offloaded to S3, once G3's sent you the HTML page, that's it - it's done with the request. and S3 is *designed* to do just this and to be used in this exact scenario.

glad you're seeing the benefits after using it anyway
i should enquire to amazon and see if i can get a referral payout for all these people i'm sending to AWS lol :D

Dan

jakob; your setup does certainly provide some challenging hurdles, and knowing how your G3 is setup gives me more ammo to go away with and install, setup, add content and test a similar setup to yours (i.e; private content, g3 installed on a home server, access internally on the same network, access externally pulling html from home server and images from s3, uploading content externally, etc).

it's something i'll certainly add into the test cases and see if i can pull together a solution for you to use this module and keep G3 running at a more-than-acceptable level for you, and make this module overall more feature-rich.

thanks,

Dan

Dan, Unfortunately, I'm not a programmer by any stretch of the imagination. I do appreciate what you've done with this module. My questions were in general, not complaints. I was/am looking for experiences with the module and see if there are any solutions that I'm not aware of.

With that being said.. when/if you need help testing the newer versions of this.. I'd be more than happy to help. Maybe I can provide a larger test case for you. If I can help you.. PM me.