Module: Amazon S3

danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-11-28 12:15

Announcing a new module available which allows you to synchronise your content with Amazon Simple Storage Service (S3) for either use as a CDN or backup purposes.

Version: 1
Blog post: http://www.danneh.org/2010/11/amazon-s3-for-gallery-3/
Download: http://www.danneh.org/files/aws_s3.zip
Codex: http://codex.gallery2.org/Gallery3:Modules:aws_s3

Update Jan 11th 2012: It's come to my attention that this thread has been locked for whatever reason. So, with that said, if you've got any comments, feedback or suggestions for this module, please direct them at my original blog post (http://www.danneh.org/2010/11/amazon-s3-for-gallery-3/). I'll keep this particular post up to date with new release info and whatnot, and use it as a public domain, as this forum thread once was, for advice, suggestions and bug reports. -Dan

 
floridave
floridave's picture

Joined: 2003-12-22
Posts: 27300
Posted: Sun, 2010-11-28 20:46

COOL!
When I first set this up ( I have never used Amazon S3 ) it did not work at all so I started over.
Activated: Amazon S3
Settings -> Amazon S3
checked S3 enabled
entered the Access Key ID, Secret Access Key, & Bucket Name from the amazon site under the 'Access Credentials' page.
Saved settings and I was prompted via :
"Your site has not yet been syncronised with your Amazon S3 bucket. Content will not appear correctly until you perform syncronisation. Fix this now"
Modal window poped up "Syncronise with Amazon S3 " and the progress bar finished with:
'Sync task completed successfully'
I clicked close.

I notice that 1 item of the 17 might not have synced properly.
http://www.langleycom.com/gallery3/index.php/ The first item.
I have one item: http://www.langleycom.com/gallery3/index.php/Picture-1-1716921105
that gives a response of:

Quote:
<Error>
<Code>AccessDenied</Code>
<Message>Access Denied</Message>
<RequestId>D0557C60100E11DC</RequestId>

<HostId>
+fK/HySUlAbrQZ266hWhqNDu4XKk6aWP74hABRz2CqMfMBNoogrX7gWi9NolCQQ6
</HostId>
</Error>

----time passes----

I think I found the issue.
http://www.langleycom.com/gallery3/var/albums/Picture%201-1716921105.png is a item with a space in the file name.
Deleting the item and re-adding the item without a space in the file name fixed the issue.

--------more time passes -------
I am no expert in spelling but syncronised and syncronisation don't look right to me.
Sorry to nit pick.

Great module!
Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-11-28 21:05

ah, so spaces in the filename might be a problem with S3. i have read a couple of instances where this is an issue. i think in future versions i'll going to have to sit and re-write filenames that have odd characters/spaces in them. this module simply uses the existing filenames/paths as stored by g3, but for management purposes an extension on the items table or a new table for it might be necessary. i've got to deal with the issues with cloudfront first, but i'll pencil this in the v2 as well.

and lol - i just spellchecked them and there's an h missing. i'll fix that and push it into git later on.

but all in all, i'm glad you had a decent amount of success with it :)

Dan

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Wed, 2010-12-01 22:30

As I said in my other post, thanks! And while I'm at it, I just noticed that when enabled, it won't show any images (not in the Admin Dashboard, nor in the main gallery). No thumbnails, no resized images and no full sized images. Nothing shows up when this module is enabled (SSL is currently disabled and sync went just fine and finished successfully.

See attached screen capture to show what I see.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Wed, 2010-12-01 22:32

mm strange. suggests that the sync didn't go as successfully as it thought. can you send me the log file in var/modules/aws_s3/log/aws_s3-<date>.log. this should have some insight as to why it failed. if you can't pm it, send it to my email: dan at danneh dot org.

Dan

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 05:00

Apart from success lines, and uploading item lines, there isn't anything else in there (like Error or Warning):

Quote:
cat var/modules/aws_s3/log/aws_s3-2010-12-01.log | grep -v success | grep -v "uploading item"
2010-12-01 11:16:17: re-sync task started..
2010-12-01 11:16:17: items to sync: 6889
2010-12-01 11:16:17: emptying bucket contents (any files that may already exist in the bucket/prefix path)
2010-12-01 14:03:26: completing upload task..

Emailed the log to you as well.

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 05:44

Using Firefox's Firebug addon, I see that all the images get a 403 Forbidden. I went to my bucket and change the Permissions to include Everyone with List and View Permissions but that didn't help.

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 09:20

I noticed that if I use Firebug, and take the Entire URL (Copy Location in Firebug) it is trying to access (but gives the 403 Forbidden), and just remove the Signature part (which is the last param after Expires param), it will show me the image in question. This might suggest that the Signature is somehow breaking things?

I PMed you the full URL and the one without the Signature, so you can compare the two.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Thu, 2010-12-02 09:21

humm..very strange. if you view the contents of the bucket using aws's console (https://console.aws.amazon.com/s3/home), are the items there? (default under g3/<guid>/[fs|rs|th]/...)

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 09:27

Yes

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Thu, 2010-12-02 09:38

hm, ok. from what i saw with the signature'd one, the request/signature had expired. now, for public galleries/albums, it shouldn't even be putting the signature onto the url as it's not required, so there's one issue to begin with. however, having a signature on the end of a public permissioned url shouldn't affect it operating.

check your server's time and make sure it isn't skewed too much (+/- 60 seconds). run "ntpdate -p4 pool.ntp.org" and check the response. obviously you'll need to be root to do this. i'm leaning towards this being the problem as if the time is skewed too much, the request/signature will have expired before it's been generated (in real-time/aws server time context, anyway).

if you can't set/sync the time on the server, try increasing the expiry time in the aws module's admin to something higher. i'd go in 5 minute (300 second) intervals until it works.

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 09:39

What kind of permissions do you have on your g3 folder under your bucket? You mention in the module setup page, that one should make sure it exists, and the correct ACL should be setup ... what settings should I have?

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Thu, 2010-12-02 09:46

the g3 folder (and the guid folder beneath this, and the fs/th/rs folders beneath that) should have full public read permissions. the module won't create the bucket, but it will create the g3 and guid folders if they don't exist. aws doesn't have a folder heirarchical structure, all files are simply 'objects', and the g3/guid/th/etc.. is simply a prefix to an object. listing occurs by filtering objects with the same prefixes. now, i could possibly forsee an issue if the first item it uploads is a private one, and is given ACL_PRIVATE rights, and this may inherit right up to the root g3 folder. however, if you check (and change, if necessary) the g3, guid, and fs/rs/th folders to ensure they have public/everyone read/view permissions, that should be fine.

the more i think about it though, the time being skewed seems more and more likely.

Dan

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 11:22

You mean that they should have View Permissions (but not List Permissions), right? Because if List Permissions were to be enabled, that would put my images wide open for the entire world to see... And that would be a serious security issue. In fact, is there no way to only display images via my Gallery3 without people having direct links to my images on S3?

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 11:28

When I click on the g3 folder, or the guid folder or the folders under it, I can't change their permissions. The only permission I can change is that of the bucket itself. It has the only permission for my own user to do everything (read, list, write, edit permissions). Should I add Everyone and give it View Permissions? Cuz I tried that and it didn't work.

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 11:33

Having said that, when I drill down in the albums, all the image files themselves have read permissions and I get a link to each one and when viewed by a browser, the image loads just fine. I don't understand why the modules keep erroring out on permissions when I can get these images to load using a regular browser URL directly to S3.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Thu, 2010-12-02 11:46

yes, that's what i meant. view (read). if your content is private, (i.e; not visible by the "everyone" group in g3) then the content is uploaded with a private acl which gives only the bucket owner (you) read permissions. if you're seeing the image when you're logged into aws, then naturally you'll be able to see the images because you're logged in. as far as your bucket contents permissions go, if the content is public or private, only YOU should have list and edit permissions. if the content is public, "everyone" will have view permissions. if not, only you will have view permissions. the signature basically authenticates the client as YOU for a single item for a set amount of time. after which, the signature expires and the client is no longer able to access the resource.

the module is detecting that the content isn't publically visible, hence it appending a signature to the end of the url, which grants the client temporary access to that particular image until expiry time. you'll get a 403 if the signature expires, and this would only happen if either;
1. the signature was generated and somehow the client only requested the image from s3 more than 60 seconds later (default expiry time in the module), or
2. the time on the server is skewed more than 60 seconds behind the current time so that the timestamp that the module/library generates as the signature's expiry time, by the time it's delivered to the client, has already passed.

i stand firmly by the time skewed theory, as i've not had any problems like this during development/testing. i'll attempt to replicate it on my development machine later, but i firmly believe this is the cause. like i said, run "ntpdate -p4 pool.ntp.org" as root on the server. i can practically guarantee you'll get a < -60 second skew.

Dan

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 13:14
Quote:
# ntpdate pool.ntp.org
2 Dec 05:12:26 ntpdate[4944]: step time server 207.171.7.151 offset 0.516204 sec

Seems to be just a half second behind ... what else might it be?

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 14:46

It might be early to tell, but I tried to create another bucket using S3 Organizer, and like the initial bucket I created, which contained an underscore (which was created using their AWS Management Console for S3), I tried to create another one and S3 Organizer didn't allow me to create it, saying I can't use an underscore! So I went to Amazon's docs, and noticed that I really can't. I have NO idea why it lets you create a bucket who's name doesn't comply with their own restrictions!

So I created another bucket with a name I want to use, and which followed their restrictions and while it is just at the thumbnails, I can see them already online and using Firebug for Firefox, I can see that it reads them from Amazon!

Maybe put it up on your FAQs?

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Thu, 2010-12-02 15:27

how very strange! makes sense, because my bucket's called "danneh-org", so i wouldn't have spotted that problem. i'll verify it with my installation later, but if that is the case, well spotted! i'll certainly mention that in the FAQs, and probably put some sort of bucket name verification on the admin page so ensure that it makes sure you know that you can't use underscores in the bucket name.

well done :)

Dan

 
Shai

Joined: 2003-12-10
Posts: 61
Posted: Thu, 2010-12-02 15:57

I wouldn't go as far as adding code to verify the naming convention, but I would put a link, at the text you have were you require the bucket name, to their page:

http://docs.amazonwebservices.com/AmazonS3/latest/index.html?UsingBucket.html

This link would get updated by them, and thus would keep you from needing to keep your code up-to-date with their changes in the future.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Sun, 2010-12-05 00:31

This sounded sweet. I have my gallery on a server at home behind dsl.
I tried it, but no luck getting it working.

I made an S3 bucket, I copied the required keys from s3 into the module-setup.
When I try to run the 'fix' it never gets further than 'Something went wrong'.

Excerpt from log:
2010-12-04 15:52:50: re-sync task started..
2010-12-04 15:52:50: items to sync: 68
2010-12-04 15:52:50: emptying bucket contents (any files that may already exist in the bucket/prefix path)
2010-12-04 15:52:52: emptying bucket contents (any files that may already exist in the bucket/prefix path)
2010-12-04 15:52:55: emptying bucket contents (any files that may already exist in the bucket/prefix path)
2010-12-04 15:53:00: emptying bucket contents (any files that may already exist in the bucket/prefix path)
2010-12-04 15:53:06: emptying bucket contents (any files that may already exist in the bucket/prefix path)

I tried turning logging on in S3, but that didn't yield any data anywhere as far as could see.

I made my bucket-permissions fully open to everyone.

I tried creating a public folder g3 in the bucket.

I tried creating one more bucket.

Any ideas?
Any more info I can provide?

My bucket is named according to this pattern:

a1234

And it is accessible like so http://a1234.s3.amazonaws.com/

Data I entered into 'Bucket Name' in the module setup was strictly only the 'name' "a1234" - not any http or any other parts of the URL.

Thanks Jakob

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Sun, 2010-12-05 17:09

OK I got a step further.
Apparently it works now - pending more testing ;-)

The problem was that my php did not have the cUrl extension.

I fixed this by using synaptic to install 'php5-curl'.

I found out by trying the "AWS SDK for PHP":
http://aws.amazon.com/sdkforphp/
From there I downloaded the sdk:
http://pear.amazonwebservices.com/get/sdk-1.2.0.zip

I then placed that in a subdir under my apache.
I copied the file 'config-sample.inc.php' to 'config.inc.php', and filled in most of the required data.

I didn't understand what was meant by 'Constant: AWS_CANONICAL_NAME' as it was not listed on the aws credentials-page.
I found this page, that helped me understand that my 'canonical name' is simply 'jakob'!:
http://www.bucketexplorer.com/awsutility.aspx

I then browsed to an aws-sdk-testpage on my local server:
https://server.net.lan/aws/_compatibility_test/sdk_compatibility_test.php

Where i was told that I could not use the SDK, because i was missing php-cURL (with SSL).

Regards, Jakob Hede Madsen.

 
tgpayne

Joined: 2010-12-05
Posts: 1
Posted: Sun, 2010-12-05 17:38

Thanks for a great module! It's perfect for me with a home server with v. limited upload bandwidth.

I thought I'd share a problem I had with getting it to work on Arch Linux, in case it helps anyone. It's currently synchronising so hopefully all is now well. Edit: this seems to be the same problem encountered by Jakob!

The module initialised no problem, but whenever I tried to synchronise it was hanging at the point where it empties the contents of the bucket. There was no error message in the module logs or the web server logs.

I eventually traced the problem to the line "$curl = curl_init();" in /lib/s3.php. As it turned out, I needed to add the curl.so extension in php.ini (which then generated an error in locating curl_init function). I then installed the php-curl module, and hey presto, it works. Took a few hours to get to this point but I learnt a little about SOAP and cURL in the process, so not wasted time.

I don't understand why the call to curl_init didn't raise an error in the httpd logs - I would have expected one, or is it me?! Perhaps the documentation could include a note about curl, for people like me who are not that familiar with the workings.

Thanks again, Tristan

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-12-05 20:51

that all makes sense, actually.

the call to curl_init() won't error in the httpd logs, but it'll error in the php_error.log (disabled by default). although i think g3 attempts to catch some php errors in it's own log files (var/logs/<date>.log), so there may be something in there along the lines of "call to undefined function: curl_init()".

i'm in the midst of working on version 2, including some more helper functions to get the bucket name syntactically correct, optimise the upload function a bit, and i'll add in a test also that will prevent you from activating the module if curl isn't installed, since the s3 lib isn't my code, and i can't really be bothered writing my own wrapper for it - and even so, i'd still depend on curl being there!

Dan

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Sun, 2010-12-05 21:18

Hi Dan.

Thank you for this fine module.
Some questions:

---------
I experience that your module breaks the cooliris slideshow.
Is that also your experience?
Can you suggest a way forward for me?

---------
I experience that the s3-service is faster than my 1Mb dsl-up, but only by a small factor 2-3, and it definitely does not saturate my effective 8 Mb down.
Is that also your experience?
Do I understand correctly that cloudfront is supposed to give better performance - good enough to saturate normal dsl?
You mention cloudfront on the codexpage
http://codex.gallery2.org/Gallery3:Modules:aws_s3

Quote:
Optionally, you can change the URL template below for use with CloudFront.

But I can't follow that 'below' reference.

You write that

Quote:
This module currently does not work with CloudFront and private/permission restricted content.

All my content is private - it's of my kid's school-class, and needs to be private.
Does that mean no go for me on cloudfront, or could there be some kind of (fake temporary) solution that worked with cloudfront - kept the private appearance on the G3 interface, but was actually open on the cloudfront?

-------
Would it be possible to only transfer the fullsize images to cloudfront, and keep everything else on the local server?

Is it conceivable to get the slideshow working with the local resizes?
The raw jpg's in the 5MB range are too big for slideshow anyways.

------

Regards, Jakob Hede Madsen.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Sun, 2010-12-05 21:26

In the file
./helpers/aws_s3.php
Line 38:
S3::getAuthenticatedURL("danneh-org", $resource, module::get_var("aws_s3", "sig_exp"));

The hardcoded reference to your domain seems odd. What are the implications, and should I 'patch' this?

Regards, Jakob Hede Madsen.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-12-05 21:53

Jakob,

I've just checked this on my production gallery and the slideshow appears to work correctly, pulling all the content from S3 as opposed to the G3 server itself. What's your AWS_S3 configuration, and what appears to be happening for you?

Yes, S3 is a high-bandwidth content delivery system, and CloudFront is a similar system. To understand the differences, let me attempt to outline the basics:
- S3 is a storage mechanism with a method to allow you to access your content from the same network segment that you chose when setting up the bucket (i.e, Europe, Asia, North America, etc). In this setup, all content is delivered from that region (Europe = Ireland, etc), irrespective of where you're situated in the globe.
- CloudFront is a content delivery network ONLY. In other words, there's no "bucket" for CF, but you upload your content to S3 and access it via CF. CF distributes your content throughout Amazon's global delivery network, and accesses to the content will be delivered from a segment of their network geographically closest to you.

In either case, DON'T expect it to saturate your bandwidth when downloading. There's many technical reasons behind this, but you WILL see a difference when accessing your content remotely, since S3's ridiculously quick network is much faster than a 1Mb DSL upload.

CloudFront is really only for hardcore sites that need a globally powered CDN. In most cases, using the raw S3 delivery network will be good enough. You CAN CNAME the S3 bucket URL, go here http://docs.amazonwebservices.com/AmazonS3/latest/index.html?VirtualHosting.html for info on how to achieve this. At this present moment, this module will NOT work with private content going via CloudFront due to the differences in the QSA signature between CF and S3. If you have private content, you simply cannot use CF until I can get QSA working. The library I'm using doesn't support URL signing for CloudFront, only S3. This is a known issue (hence the posted statement), and will be resolved in a future version, but in the meantime, for private content, you'll have to use S3's CDN (http://{bucket}.s3.amazonaws.com/g3/{guid}/{resource}).

And lastly, uploading only the full sized versions won't solve any problems, as full sized images are only downloaded upon request. The idea of a CDN is to speed up access to the site by offloading the transmission of static content (in this case, images) to a delivery network, leaving the G3 server only to deal with vending the HTML pages, dealing with permissions and access, etc. If there's enough demand for it, I'll add into a future version checkboxes for the type of content to upload. Until then, it'll upload thumbnails, resizes and full-sized images, since utilising a CDN is the ultimate goal for this module.

Dan

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-12-05 21:56

lmfao! omg i can't believe i'm that stupid lol. implications? no private content will work because the bucket name is incorrect.

replace that line with this:

S3::getAuthenticatedURL(module::get_var("aws_s3", "bucket_name"), $resource, module::get_var("aws_s3", "sig_exp"));

I'll patch the git contrib repo and my site now.

Well spotted, thanks!

Dan

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Sun, 2010-12-05 23:17

Ok so cloudfront is not for me, but should I really only see bandwidth like 200-300 KB/s from s3?

That is certainly not 'ridiculously quick'!

I am simply interested in getting proper performance, and by that I think it should be possible to saturate my relatively meager effective 8 Mb.

How can I get from here to there?

I am thinking of handling the smaller stuff directly from server, and only the large stuff from S3, because 1) my own connection is fast enough for the small stuff, and 2) Then I don't have to pay S3 for delivering stuff, that I can handle myself.

Regards, Jakob Hede Madsen.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Sun, 2010-12-05 23:57

Regarding the nonfunctional slideshow.

I fixed the hardcoded reference as per your instructions.
I had hoped that it might fix the slideshow problem, but alas.

My observation:
As you can see in the attached picture, the Flash appears, and it loads the thumbs, but it remains black.
The statusline in the browser flickers constantly between 'transfering' and 'waiting' for s3.

What can I tell you about my AWS_S3 configuration?
It's located in Ireland.

Regards, Jakob Hede Madsen.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Mon, 2010-12-06 01:33

Update regarding slideshowproblem.

I have made an album with fully open permissions.
In this album the slideshow works in conjunction with s3.
Also RSS works for this album.

All other albums reside inside one single access-restricted 'master-album' because I need to guarantee privacy.
In all these albums rss does not work for me, and s3-slideshow does not work.

I observe that I can only access rss and s3-slideshow in albums where 'everybody' has 'view' access.
I use Vienna for RSS, and apparently Vienna has a feature to authenticate to the server, but entering my credentials does not give me rss-access to restricted albums. Maybe this is an error in G3's rss handling of restricted material, that somehow only surfaces in the slideshow, when it's running on s3?

Hope that is food for thought.

Regards, Jakob Hede Madsen.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Mon, 2010-12-06 02:08

Another that you may already be on to:

When I upload images named like this "Photo on 2010-10-25 at 23.21.jpg"
They don't work.
The same images renamed to something like "p1.jpg" works.
The images are uploaded to the local server, but it is suspiciouly fast - probably because the step of sending data out to s3 is omitted.
The images - and their resises do appear under the dir 'var', but their image can't be seen in the album - only their name.

Regards, Jakob Hede Madsen.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Mon, 2010-12-06 02:45

OK, here's a rather weird one.
It's a bit late and I haven't given this much verification, but I want it out there for your contemplation.

Pictures that were uploaded to an album while it was restricted DISAPPEARS when when the restriction is removed.

Regards, Jakob Hede Madsen.

 
floridave
floridave's picture

Joined: 2003-12-22
Posts: 27300
Posted: Mon, 2010-12-06 03:25
Quote:
When I upload images named like this "Photo on 2010-10-25 at 23.21.jpg"

danneh3826 is aware of this with this:

Quote:
ah, so spaces in the filename might be a problem with S3. i have read a couple of instances where this is an issue.

Dave

_____________________________________________
Blog & G2 || floridave - Gallery Team

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Mon, 2010-12-06 07:35

Hi Dave.

What are your thoughts on the issue with failing RSS on restricted albums, that I mentioned above?
Can you confirm that there is a conflict between rss and access-restriction?
Can you confirm that an rss-client like Vienna should be able to authenticate itself against g3 in order to gain access to rss from restricted albums?
Do you believe - if g3 indeed has a problem with rss from restricted albums - that this is related to the problem with nonfunctional s3-slideshow from restricted albums?

Regards, Jakob Hede Madsen.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Mon, 2010-12-06 08:39

I have just verified the problem with the disappearing images.

Images that are added to an album that has restricted access for 'everybody' to 'view', disappears when the restriction is removed.
Meanwhile other images that are added while the restriction is open, will remain in the album regardless of the state of restriction.

Regards, Jakob Hede Madsen.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Mon, 2010-12-06 08:49

When I use 'Organize Album' to move an album to another location, all images in the moved album disappears.
When I move the album back to its original location, the images reappear.

The same happens when moving a single image instead of a whole album.

Apparently images cannot be moved, without disappearing.

Regards, Jakob Hede Madsen.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Mon, 2010-12-06 09:51

jakob;
i have a feeling i know where that problem is coming from, regarding images disappearing. i think it's something to do with a missing hook-in within the s3 module which gets a signal from g3 that it's not attempting to move the items from one location in the bucket to another.

in version 2 i am addressing issues such as this, changing permissions from public to private (and vice versa), moving content around, and i'll take another look into the slideshow in both public and private album contexts to see if i can resolve this. i'll add these as known issues on the codex page, and this will at least remind me to address it in version 2.

if you want to only be able to upload full sized images as opposed to all types of images, i can put this in as a feature request. obviously i'll be prioritising the bugs and general issues first over feature requests, so this may not appear in version 2.

dan.

 
CJ00TA

Joined: 2010-03-31
Posts: 34
Posted: Wed, 2010-12-08 14:23

IS there a way that I can manually up some of my folders? I'm having timeout issues when the initial sync is running (~72,000 pics, about 800gb)

What happens when I upload new images? Are they stored local, and stored on S3?

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Wed, 2010-12-08 16:08
CJ00TA wrote:
I'm having timeout issues when the initial sync is running

This is probably not of any help in this particular case, but it might somehow be related:
I (well actually a tester) experienced problems when uploading through the normal flash-upload-interface. Some pictures didn't complete.
I alleviated this problem by reducing the number of concurrent uploads from 5 to 2 or 3.
You can do this in the advanced settings.

CJ00TA wrote:
What happens when I upload new images? Are they stored local, and stored on S3?

Here is what I have observed:

Yes, the pictures are stored both locally and on S3.
This is great because it allows you to switch S3 off again, and your gallery continues without a hiccup.

It does mean that the upload procedure takes a very long time, and that the progress-bars in the upload-interface, completes long before the process is actually finished, which might confuse some users.

The upload-process has three steps:

  1. Upload from client to server
  2. Processing on server
  3. Upload from server to S3

·

The progress bar is proportional to step 1.
Step three is potentially much more time-consuming than step 1 because it has to go out through the narrow connection, which the excersise of using S3 is meant to bypass later.
So you have to pay the penalty for the slow outgoing line from the server once at upload, in order not to pay anymore at all the subsequent views.

It's a thought that step 3 - the upload from server to S3 could be detached from the upload-procedure, and handled asynchronously afterwards. Then the gallery could serve images locally, until they had completed upload to S3.

Regards, Jakob Hede Madsen.

 
CJ00TA

Joined: 2010-03-31
Posts: 34
Posted: Wed, 2010-12-08 18:57

I guess I should be a bit more clear...

I'm in the initial sync phase of S3... when I enable it, I get the popup that says I should sync.. I can click the "Fix this now" link, and it's sending images to S3.. and I can see them in the S3 manager.

On my first failed attempt, I uploaded about 11,000 images before it failed. All of these images are already stored on my web server (Dreamhost) and sending to S3 at what appears to be a great clip (~15mb/sec).

I'm not even at the process of uploading *new* images from flash while S3 is enabled.

I'd love to see this module not have to re-upload everything every time there is a problem.. my webhost will hate me if I have to move 800gb each time I have to enable/disable the S3 module.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Wed, 2010-12-08 19:59
CJ00TA wrote:
I guess I should be a bit more clear...

Well, there is nothing in your follow-up, that changes my perception of what you sought to express in your initial post.
Maybe you should try to read what I wrote again? And I'm sorry if I'm not expressing myself so eloquently, I am Danish.

I perfectly well understood that you are in the initial phase - I quoted you saying that - which is why I wrote
"This is probably not of any help in this particular case, but it might somehow be related"

Then you asked:
"What happens when I upload new images?"
And I replied to that question pertaining to the non-initial process of uploading new pictures.

I know I too can be dense, but I don't think the hat fits in this case.
I am not really the person you primarily want an answer from, but I am offering my input to the best of my knowledge.

Allow me to ask:
If you have outgoing bandwidth of 15Mb/s (I assume you meant Mega, not millibits) which I think is a lot, then why do you want to use S3 at all?
Of course if you have that kind of bandwidth from your server to S3, then you wont see the bottleneck I described, but maybe I'm making a wrong assumption when I think that when shifting to S3, it is always the case that one comes from limited outgoing bandwidth?

This also pertains to my earlier questions to Dan - which by the way you haven't really addressed yet Dan - about the bandwidth out of S3. I am only seeing around 300 Kb/s coming out of S3, and while that is better than my dsl-up, it's very far from impressing, and I would be able to get far higher bandwidth coming out of my webprovider servage.net(, who unfortunately have other bottlenecks, that makes it unbearable to run G3 at their servers).

What kind of bandwidth are you seeing CJ00TA?
I assume you're not shifting to S3 in order to see your delivery-bandwidth drop by a factor 50?

Regards, Jakob Hede Madsen.

 
CJ00TA

Joined: 2010-03-31
Posts: 34
Posted: Wed, 2010-12-08 20:11

Jakob, I appreciate your help.

I'm currently using a shared hosting solution.... and the web servers are VERY slow at serving up my images at times. I'm seeing ~800KB/sec from S3

With the limited testing I've done, just by moving my images off of this shared host to S3, my gallery responsiveness is MUCH MUCH faster.

Also, by experimenting with this, I may be able to move my database and G3 installs into my own server and stop paying for this shared host.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Wed, 2010-12-08 20:22

@CJ00TA; there are issues still with the sync process that i've become aware of at some point or another since v1 was released. i am in the process of addressing these issues, but i have limited time available to me, so unless you want to contribute to the code, you'll just have to wait for me to do it. i have planned a handful of improvements for v2, which include the possibility of moving the initial sync task to a cronjob. (though i've not seen any mention of a cron within g3, but i'm sure there's something somewhere, or a way of running that task into the background). there'll also be the addition of a table to the database for it to log various items statuses as to whether it's been uploaded, and which items have been uploaded (thumb, resize, fullsize, all 3, etc), and the time it was last uploaded. the code will then use this table to determine whether to redirect the request to s3 or locally. this will also address the issue of a "failed task", and preventing the task from emptying the bucket so it would have to start again. this additional table will also allow it to fork the S3 upload task to the background and allow you to see the image(s) that you just uploaded prior to it being uploaded to S3. as an additional feature, the admin will be able to give you an overview of number of items vs number of items uploaded, number of failed uploads, the ability to re-upload single images (failed or not), and the ability to manually flag items as uploaded (thus allowing a manual upload process if need be).

@jakob; i really can't answer your bandwidth questions relating to S3. i'm not an amazon network admin, nor do i have any in-depth knowledge of how their system works. if you're only getting 300kb/s from S3, then i'd do a bit of bandwidth testing on your dsl line and perhaps see if the bottleneck lies in your dsl line as opposed to S3's multi-gigabit/second lines throughout the world. go to www.speedtest.net, choose a mirror that's closest to where your bucket lies (i.e; dublin for an ireland bucket) and run another test using the closest mirror to you and paste the results here. try also running a traceroute to "s3.amazonaws.com" and seeing what sort of response time you get from their border (amazon-gw.ip4.tinet.net, or words to that effect). i'm averaging around 60ms from my less-than-perfect uk based dsl line to their ireland nodes. downloading one of my larger files yields me around 650-700kb/s


HTTP request sent, awaiting response... 200 OK
Length: 3065436 (2.9M) [image/jpeg]
Saving to: `CAM_0142.JPG'

100%[======================================================================================================================================================>] 3,065,436    757K/s   in 4.1s    

2010-12-08 20:19:23 (731 KB/s) - `CAM_0142.JPG' saved [3065436/3065436]

so.... i don't know what else i can say. if you have a real problem with it still, get in touch with your isp's tech support and complain of throughput issues, or amazon themselves (who will most likely tell you where to go).

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Wed, 2010-12-08 20:27

I have much sympathy for G3, and I think that the S3-module is a soild contribution.
I am making a humble gallery to use for my oldest daughters class, were we can share pictures of our kids.
I would love to use G3 for this, and I have invested quite some time trying to get it right, however there seem to be persistent issues, that ultimately may force me to choose another solution such as Picasa.

Because these are pictures of children, and there currently is widespread paranoia of the potential misuse of such pictures, I have to guarantee that the pictures are access-restricted. I also need to cater for multiple users from a wide range of computer-abilities.

These two factors may combine to make my use-case a fringe-case?

Most of the problems I have seen seem related to the access-restrictions I have applied.
I can't seem to get rss to work out of access-restricted albums, and this seems to be an inherit limitation of the current state of G3, and since other services such as the slideshow depend upon rss, this problem seem to have wider implications.
I have explicitly tried to get Dave to answer this issue, and I would still want to get your contribution Dave. Even if it was just to tell me that this has already been covered elsewhere (and a kind link).

Some of the problems are related to catering for a broad userbase. It would be temporarily acceptable to have to deal with disappearing images, when moving items, or changing permisions, but I cannot offer a system to ordinary humans with all these caveats.
Also I am afraid that it could potentially confuse a user that the progressbar for an upload finishes, and he then has to wait up to 10 times longer before that picture is actually finished. This is the case in my experience with G3-S3.

So maybe I must just conclude that G3 does not yet cover a wide enough use-case-landscape for me to be able to apply it to the schoolclass-gallery.
If I could run it from my homeserver, and then rather than using S3 as delivery for static content, I could use my hosting-provider, and I could make the process of uploading to the provider separate from the user-experience of uploading, then I could maybe get satisfying performance with G3.

Now I tried to explain my situation to defuse any hostility, and see if some of you feel like you have something to contribute.

Regards, Jakob Hede Madsen.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Wed, 2010-12-08 20:28
CJ00TA wrote:
just by moving my images off of this shared host to S3, my gallery responsiveness is MUCH MUCH faster.

that's the entire idea of this module, S3, and CDN's in general. i've seen the *exact same* result after using this module on my gallery. since all the static/large content is offloaded to S3, once G3's sent you the HTML page, that's it - it's done with the request. and S3 is *designed* to do just this and to be used in this exact scenario.

glad you're seeing the benefits after using it anyway :)
i should enquire to amazon and see if i can get a referral payout for all these people i'm sending to AWS lol :D

Dan

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Wed, 2010-12-08 20:33

Hi Dan, thanks for the suggestions, I will shortly provide some testresults.

Regards, Jakob Hede Madsen.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Wed, 2010-12-08 20:43

jakob; your setup does certainly provide some challenging hurdles, and knowing how your G3 is setup gives me more ammo to go away with and install, setup, add content and test a similar setup to yours (i.e; private content, g3 installed on a home server, access internally on the same network, access externally pulling html from home server and images from s3, uploading content externally, etc).

it's something i'll certainly add into the test cases and see if i can pull together a solution for you to use this module and keep G3 running at a more-than-acceptable level for you, and make this module overall more feature-rich.

thanks,

Dan

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Wed, 2010-12-08 21:24
webjakob wrote:
Hi Dan, thanks for the suggestions, I will shortly provide some testresults.

I just made some tests.
I cannot test actual G3S3 peformance right now, because my G3 is currently switched off S3, and apparently I currently have active outside uploaders, because we have asked for contributions for an event tomorrow.

This also means, that I can't precisely test my connections, because the uploaders are poluting the results.

The upshot of that is that I'm happy to actually see any uploaders! ;-)

Anyway the tests I just performed, showed that I have approximately 7 Mb down and 800 Kb up, regardless of targeting denmark or Dublin. Also simply downloading an item from my S3-bucket seemed to saturate my connection, so that does not support my complaints of weak G3S3 performance.

I will have to get back to that later, when the other parents have gone to bed... ;-)

If you're into esoteric details, you can look at the attached screenshots of my speedtests.
The blue column represents received data, and it consistently had that dip in the middle of the transmission.
I don't know if that dip is clue to anything, or if it is coupled to my specific connection, or is typical of speedtest.net.

Regards, Jakob Hede Madsen.

 
CJ00TA

Joined: 2010-03-31
Posts: 34
Posted: Wed, 2010-12-08 22:33

Dan, Unfortunately, I'm not a programmer by any stretch of the imagination. I do appreciate what you've done with this module. My questions were in general, not complaints. I was/am looking for experiences with the module and see if there are any solutions that I'm not aware of.

With that being said.. when/if you need help testing the newer versions of this.. I'd be more than happy to help. Maybe I can provide a larger test case for you. If I can help you.. PM me.