Module: Amazon S3

webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Thu, 2010-12-09 01:59

Ok got back to the the S3 bandwidth tests.

S3 easily saturates my 800KB/s down.

So either the internetwinds blows better today than they did last I checked, or my checking is better... whatever. ;-)
At least that makes the prospect of ultimately using S3, when the kinks of the module have been ironed out, much more attractive.

But I do need:
- The ability for endusers to upload images that are named like they were from mars. And that includes spaces, weird chars, and especially the *stupid* Danish diacritical chars æøå ÆØÅ. (doh)*
And we can't have images that go disapearing. ;-)
Lots of other small details.

Ultimately I might tug along for now without S3 in the hope that this module will emerge as a butterfly! ;-)

BTW.
One more vote for the ability to switch back into S3, without completely emptying the bucket.

Regards, Jakob Hede Madsen.

 
floridave
floridave's picture

Joined: 2003-12-22
Posts: 27300
Posted: Thu, 2010-12-09 04:47
webjakob wrote:
I have explicitly tried to get Dave to answer this issue, and I would still want to get your contribution Dave. Even if it was just to tell me that this has already been covered elsewhere (and a kind link).

In a thread that is now 2 pages long and I only joined the thread to test this cool new module. It does have a few issues that I have pointed out, and the author is addressing as he has time.
You seem to have enthusiasm and perhaps you can channel some of this enthusiasm by helping out.
I don't know what you are after and I don't want to try to understand it, in a two page thread. I understand that English is not your first language, but your tone to me sounds a bit demanding. We are volenteers here and have no obligation to answer or help out.

I know nothing about RSS, REST, rewrite rules and a lot of other stuff. If I don't answer, it might mean I don't know and am waiting for some other volunteer to help you out.

Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Thu, 2010-12-09 20:02

jakob; i understand there are many improvements that can (and will) be made to this module. this module is only in it's infancy stage, version 1 just gets the job done, quick and dirty. version 2 will bring a lot more features, include some that you've asked for, and fix a handful of identified issues, such as that of spaces in the filenames. i believe that odd characters in filenames (such as the danish character set) are a restriction in s3 itself (and the way xml works), though i can't find any official documentation to back that statement up, i've read a couple of forum threads that suggest utf8 encoded filenames won't work on s3. with that in mind, i'll probably base64 encode the original filename and upload it with that instead so that character encoding isn't an issue. this would also resolve the problem with spaces in the filename also.

but like i've said before, i only have limited time on my hands, so can't kick out bug fixes and module updates as quick as maybe you'd like. but since you're subscribed to this thread now, and as i'll be posting here when updates are made to the code, you'll get an email with that information so you'll stay informed at least.

Dan

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Thu, 2010-12-09 22:19

Hi Dan.

Yes I understand that it is just in its infancy, and I think it's a good start.
The things it does - it appears to do solidly without flakiness. There are lots of things missing, but that is obviously just because it hasn't developed those areas yet.

I've been struggling to get a system ready for today, so yes, of course I would have loved to have version 2 or better of your module yesterday! ;-)

I look forward to see your improvements in good time. Let me know if I can help with the silly characters.

Regards, Jakob Hede Madsen.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Thu, 2010-12-09 22:28

Hi Dave.

Ok I see that there are areas of G3 that you're not familiar with. I made the wrong assumption that you were more globally involved in G3.
And I found it odd, that you invested time in criticizing me for making a repeat mention of a problem that you had already mentioned (even when I had alluded to the fact that it might be a repeat, and that my description could potentially add useful data), instead of constructively commenting on some of the explicit questions.

It's easy to misjudge motives and behavior in a forum like this.
Keep up the great work.

Regards, Jakob Hede Madsen.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Fri, 2010-12-10 16:26

jakob; i'm sorry there was any misunderstanding. i'm not part of the core team, but maybe, one day, i shall be. it's certainly a project that interests me a lot, and one thing i like, as a developer, is having other people appreciate the time and effort you've put into your work and seeing various copies of your stuff splattered around everyone's websites - stuff that people actually find useful. that's my goal :)

i certainly now have enough info and feedback to go away and make version 2 more of what you might expect of it (and i dare say many others would expect of it in the future also).

thanks :)

Dan

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-12-12 13:38
webjakob wrote:
In the file
./helpers/aws_s3.php
Line 38:
S3::getAuthenticatedURL("danneh-org", $resource, module::get_var("aws_s3", "sig_exp"));

The hardcoded reference to your domain seems odd. What are the implications, and should I 'patch' this?

i've patched this in github (pull request pending approval) and on my site - eventually!

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-12-19 00:00

Howdy all

Bit of a manic release night tonight. I been a busy bee!! lol :)

I'm releasing 3 modules tonight. 2 of which are in beta, and one has no need to be a beta because it's so small. Here goes..

AWS_S3 v2 (BETA): Change Log:
- Access Key, Secret Key, and Bucket name validation
- Ability to upload only thumbs, resizes, fullsizes, or any combination of the 3
- Resolved compatibility with crossdomain issue in album organize. Thumbs are now vended from the local server as crossdomain blocks flash from loading these (though CoolIris is not affected, I'm not sure why).
- Ability to (if used with updated Scheduler module (linked below)) sync and upload items to S3 via a cron job rather than "there-and-then". This resolves the upload progress bar stalling/freezing/browser crashing issues while waiting for server to upload to S3.
- Spelling changes
- Updated S3 lib to ignore SSL certificate mismatches. This resolves communicating data to S3 via SSL if required. The warning is still in place on the admin page though, as you may or may not still run into issues using SSL.

Yet to come:
- Use S3 for primary storage (useful for low-storage hosting accounts) - This is pegged for v3.

Task Scheduler v2 (BETA): Change Log:
- Ability to schedule tasks via cron job.
- Updated installer, cleaned up code, added logging facility that I use in my other 2 modules (aws_s3 and transcode - a useful function sometimes)
- Added "one-off" interval

Random Image: Change Log:
- Updated SQL query to pick item at randon without complex joins etc, then load an item model to use the built in functions to get the image details, paths and URL's. The way this module behaved before meant that aws_s3 wasn't able to intercept the thumb_url() call to the item model to redirect the client to S3.
- Cleaned up code, added access check into random image generation function. Will fail if, after 10 iterations, it can't find an item that the client is able to see. I'll judge this by any feedback I might receive as to whether to increase or decrease this figure. This restriction just stops G3 entering an infinite loop if the client isn't logged in and the gallery is 100% private.

You can view all this lot in action (where possible) on my website and my gallery.

Please don't hit me if something doesn't work or breaks. I've only tested it locally and on my production gallery. Experience teaches me that there's so many more different environments and configurations that I can't even think about testing them all thoroughly. If you have time to spare, a spare G3 to run the code on where it doesn't really matter if the data gets screwed up, I'd love to hear from you. Don't forget though - backup your database and var/ directory!

Dan

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-12-19 00:08

Just another note, if you decide to use the Task Scheduler module update to run via cron, use this cron expression:

*/5 * * * * /path/to/wget -O /dev/null -p http://www.yourwebsite.com/gallery3/modules/scheduler/cron.php

Change the interval expression to whatever you feel like. The one I listed runs every 5 minutes. Make sure you have the full path to wget first ("whereis wget") to enter that in the cron expression, as some servers/distros don't like using just "wget", as PATH isn't always defined, so will inherently fail.

Dan

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-12-19 17:19

Also forgot to mention that it now moves content around S3 when you move content around your G3 installation. Images should no longer vanish when you move them from album to album. It also updates the image on S3 when you rotate it. Haven't tested watermarks yet though.

 
webjakob
webjakob's picture

Joined: 2010-11-15
Posts: 21
Posted: Sun, 2010-12-19 18:16
danneh3826 wrote:
Images should no longer vanish when you move them from album to album.

That sounds great.
I will check this module out again, to see how close it is to be robust enough for public consumption.

Any idea how to handle filename-issues?
What about a module that changes filenames as they are imported to the gallery, so that obey S3 naming-limitations?

Regards, Jakob Hede Madsen.

 
floridave
floridave's picture

Joined: 2003-12-22
Posts: 27300
Posted: Sun, 2010-12-19 18:24

danneh3826,
Thanks for your continued contribution!
For your 2 other modules can you create a new forums topic for each. It will be easier to track and keep a thread on topic. I can create some skeletal docs on the codex if you don't have time.

Perhaps somebody else from the community that is using your modules can contribute to the cause of needed documentation? (Hint - Hint for others :-) )

Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-12-19 19:09

@jakob; file naming issues should also now be resolved, so it *should* accept files with spaces and other weird characters in them.

@dave; sure, however them other modules aren't mine :) they've simply been modified/enhanced so i can use some or all of their functionality. if they don't have their own forum topics, sure i'll create one for each. if they do, then when the code seems stable enough i'll update github with this code and provide some basic docs (especially for the scheduler for other developers to use the cron function).

Random Image is a much-improved version of this module to handle permissions better, width restrictions, tidier output, and loading the item model properly as opposed to just grabbing the data out of the db.

Task Scheduler is an enhanced version of what's already in github under "scheduler" (no codex page it seems). whether it has a forum topic, i'm not too sure.

once all the code's been used by a handful of people and all seems to be ok, i'll post the mods to the relevant places, check it into github, and inform their respective authors of improvements to their module's code. i've only posted all 3 here at once because i was working on all 3 at the same time (S3 module testing gets around inside of G3 a fair bit lol)

Dan

 
floridave
floridave's picture

Joined: 2003-12-22
Posts: 27300
Posted: Sun, 2010-12-19 21:38
Quote:
Task Scheduler is an enhanced version of what's already in github under "scheduler" (no codex page it seems). whether it has a forum topic, i'm not too sure.

talmdal wrote it and it is on my list of things to do to document. I have never installed it yet so could not do the docs for it yet. I am sure Tim will take your changes and incorporate them. Feel free to create some docs for it as it is low on my priority list of things to do.

Dave
____________________________________________
Blog & G2 || floridave - Gallery Team

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-12-19 21:53

awesome. will do. do you have a skeleton codex/doc page you've been working with, or should i just write it as i have done for my other modules?

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Mon, 2010-12-20 17:28
danneh3826 wrote:
AWS_S3 v2 (BETA): Change Log:
- Access Key, Secret Key, and Bucket name validation

just found this and instantly had to (finally) create an S3-account to try this out.. installed the module, figured how to get all this keys, copy/pasted them over, but lo and behold

"The Amazon S3 access details provided appear to be incorrect. Please check your values and try again."

it beefs about all three entries (Access Key is invalid, Secret Key is invalid, Bucket name is invalid) although all are definitely correct and i just verfied them by connecting via a desktop-application from my computer.. why could this happen?

tia,
markus.

--
regular git puller.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Mon, 2010-12-20 20:08

in modules/aws_s3/helpers/aws_s3.php around line 257 (function validate_access_details) check it looks like this: (i know i've had the same issue and can't remember if i put it into the release or not now :s

    static function validate_access_details($access_key, $secret_key, $bucket_name) {
        require_once(MODPATH . "aws_s3/lib/s3.php");
        S3::setAuth($access_key, $secret_key);
        S3::$useSSL = false;

        $success_test = S3::putObjectString((string)time(), $bucket_name, self::get_resource_url(".s3_test"));
        if ($success_test)
            S3::deleteObject($bucket_name, self::get_resource_url(".s3_test"));

        return $success_test;
    }

The S3::$useSSL line is crucial to this test. the validation failed basically means that it couldn't create the .s3_test file for whatever reason.

Dan

danneh.org :: Gallery3

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Mon, 2010-12-20 21:03

it's

S3::$useSSL = false;

for the fun of it tried =true .. same result :(

--
regular git puller.

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Tue, 2010-12-21 21:14

any other idea why the .s3_test file could not be written?

fwiw i'm trying to connect to the european region, but i doubt this should matter..

thanks for any help,
markus.

--
regular git puller.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Wed, 2010-12-22 10:15

check in var/logs. if the S3 lib is erroring out for some reason, kohana/gallery should catch the error and dump it in a log file there.

Dan

danneh.org :: Gallery3

 
cradle

Joined: 2010-12-11
Posts: 2
Posted: Wed, 2010-12-22 14:38

I've got the same error (3 errors) as above.

and var/logs/ doesn't show anything, at all, during the failed request cycle.

I'd look further into it, but I need to sleep :P

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Thu, 2010-12-23 00:02

well, nothing in the logs, but i enabled debugging and this is what i got:

ErrorException [ Warning Message ]:
curl_setopt() [function.curl-setopt]: CURLOPT_FOLLOWLOCATION cannot be activated when in safe_mode or an open_basedir is set

just.. safe_mode's definitly disabled for this domain.. gotta investigate..

thanks,
markus.

--
regular git puller.

 
floridave
floridave's picture

Joined: 2003-12-22
Posts: 27300
Posted: Thu, 2010-12-23 00:16

As a FYI we added better checking in the installer for safe_mode.

Dave
____________________________________________
Blog & G2 || floridave - Gallery Team

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Thu, 2010-12-23 01:31

ok, got rid of open_basedir from that plesk-subdomain, now i'm getting a different error:

ErrorException [ Warning Message ]:
S3::putObject(): [MalformedXML] The XML you provided was not well-formed or did not validate against our published schema

http://xchange.photing.com/data/screens/shot00531.png

ideas?

--
regular git puller.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Thu, 2010-12-23 11:44

inside the function "validate_access_details" in helpers/aws_s3.php, enter this on line 258 after the function declaration:

aws_s3::log("validate_access_details: access(" . $access_key . "), secret(" . $secret_key . "), bucket(" . $bucket_name . ")");

then try again. this will log to the module's log files (var/modules/aws_s3/log/aws_s3-<date>.log) what parameters have been passed to the function. a bit of searching around suggests a problem with the S3 lib itself. don't paste the results here, or anywhere, because your secret key's also written. it's there purely so you can see and make sure that all the parameters are being passed and have values. i read that this particular error comes from the bucket_name being empty.

Dan

danneh.org :: Gallery3

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Thu, 2010-12-23 13:33

this looks pefectly fine to me - all parameters are there and correspond to what i filled in..

(and, fwiw, my bucket's name simply consists of 12 lowercase letters.. no funky dashes or anything..)

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Thu, 2010-12-23 20:20

hmm. suggests there's something else odd going on in the request out to S3. the S3 library isn't mine, but i should be able to figure out what's wrong. do you mind giving me filesystem/ssh access to your gallery's location so i can debug this?

Dan

danneh.org :: Gallery3

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Thu, 2010-12-23 23:36
danneh3826 wrote:
do you mind giving me filesystem/ssh access to your gallery's location so i can debug this?

just tried on a fresh install on the same server (but different domain), still the same..

yes, i can give u ftp/ssh-access to this domain.. how to get in touch? got IM either?

markus.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Fri, 2010-12-24 09:40

pm here is good.

Dan

danneh.org :: Gallery3

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Fri, 2010-12-24 15:40

Just posted a quick fix to this, now calling it "beta 2".

Changelog:
- Removed some old comments (not really worth a whole beta to itself, but oh well)
- Fixed an issue where the module is unable to validate Access Key/Secret Key/Bucket Name on first/new install of module due to "Malformed XML" error returned from S3 (thanks to markusk for help fixing this.

Download here (http://www.danneh.org/files/aws_s32.zip)

--
Dan

danneh.org :: Gallery3

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Fri, 2010-12-24 23:25
floridave wrote:
As a FYI we added better checking in the installer for safe_mode.

Dave
____________________________________________
Blog & G2 || floridave - Gallery Team

does this mean that it won't allow gallery to be installed without safe_mode, or is it just a warning? just so as i know whether to enforce the safe_mode = off in the can_activate() function?

--
Dan

danneh.org :: Gallery3

 
cradle

Joined: 2010-12-11
Posts: 2
Posted: Sat, 2010-12-25 10:40

working now for me also - cheers :)

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sat, 2010-12-25 10:42

wicked! :D

--
Dan

danneh.org :: Gallery3

 
floridave
floridave's picture

Joined: 2003-12-22
Posts: 27300
Posted: Sat, 2010-12-25 19:19

We abort the install on safe_mode detection.
https://github.com/gallery/gallery3/commit/6ac82bc6b72e94466787e291e15968f15f33089d

Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Sun, 2010-12-26 01:18

next bug it seems..

while S3-mod is activated i can't seem to use server add, it looks like it stalls after the first image imported (first blamed permission-issues on that new install for that, but just disabled S3-mod and now it's happily importing..)

[i might add.. while s3-mod being activated and no S3credentials entered yet]

markus.

--
regular git puller.)

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Sun, 2010-12-26 01:12

minor cosmetic issue (not sure though if anything can be done about it from module side..) after disabling the module i'm still getting the warning that i haven't synced to S3 yet.. http://xchange.photing.com/data/screens/shot00532.png

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Sun, 2010-12-26 06:01

hmm ok. i'll take a look into it's non-workingness with server add.

as far as the warning goes, are you actually disabling the module within g3, or disabling it from within it's own settings page? the reason i ask is that it's also been built for someone who wants to use s3 purely as a backup method for their g3 images, and not for vending images to web clients. this would still yield a warning that g3 isn't sync'd with s3 if it isn't.

Dan

danneh.org :: Gallery3

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Sun, 2010-12-26 10:43
danneh3826 wrote:
hmm ok. i'll take a look into it's non-workingness with server add.

i just tried again using sever add while your module is enabled and this time properly configured either. it doesn't work :( [to be specific, it again only creates one album and imports one image into it, afterwards it stalls]

Quote:
as far as the warning goes, are you actually disabling the module within g3, or disabling it from within it's own settings page?

i disabled the module within g3. to verify i just tried getting it working by disabling it within the module, but this doesn't help.

markus.

--
regular git puller.

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Sun, 2010-12-26 10:57

oh well. still more to come it seems ;)

as mentioned above, i disabled S3 from within the module itself. i still couldn't use "server add", so afterwards (without changing module-settings) i again disabled the whole module from g3 side. "server add" worked then as expected, but now i can't enable the module again, the tick in it's tickbox doesn't persist after leaving the modules-page (and hitting save of course).

have fun ;)

 
Jonreynolds

Joined: 2008-03-09
Posts: 96
Posted: Tue, 2010-12-28 17:31

Hi,

Thanks for the brilliant module! Just what I have been looking for for a while now. Have it working on my test G3 site now. I was already using S3 to back up my server and I had thought a while ago, that it would be great if I could host my photos on S3 as my VPS server is getting large, due to large amount of photos. (Well large for me, I bet some people have much larger).

So instead of just requesting more and more disk space from my VPS provider, I can keep/host my photos on S3 and just use my VPS to run the software. This means I could greatly reduce the size of my VPS disk space.

My question, which may simply be met with "its coming in V3", is that I still need to upload files to my server, then use 'server add' to add them to G3. I then find the files I have added to G3 still in /gallery3/var/albums... even though all the images are now stored on S3. I have found that I can delete the files saved in albums thought afterwards and the gallery keeps working.

1) is this OK to do for now? (deleting images stored on server disk)
2) is this the feature you are planning for V3 (S3 as primary storage)
2,5) Will the 'use S3 as primary storage' feature mean you still need the disk space to temporarily hold the images you are adding to the gallery (even if they get deleted afterwards)?
2,75) Will it be possible to upload images directly to S3 and then add them to gallery from S3?

Many thanks again for the great module.

Jon Reynolds
www.jcrdevelopments.com

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Tue, 2010-12-28 17:44
Jonreynolds wrote:
I still need to upload files to my server, then use 'server add' to add them to G3.

does 'server add' work for you while this module is enabled?

--
regular git puller.

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Tue, 2010-12-28 17:48

yes, there are plans to use s3 as primary (and only) storage, for people with situations such as yours, or with low quota disk space on their hosting accounts to begin with. to answer your questions:
1) i suppose, in theory, that would work. though once the image files have been deleted from var/albums etc, you can't move them from album to album, rotate, etc, as g3 will fall over not being able to find the file. and then the s3 module will fall over as it tries to move/update s3 with the new file. if you're ok with these restrictions, i suppose that could work.
2) yes, this is planned for v3. it was originally planned for v2, but since the uptake of this module is higher than i ever expected it to be, v2 focuses on making it more robust, allowing various other g3 features to be used (move/rotate, etc).
2.5) yes, you will need the disk space temporarily for g3 to process and extract the image data, generate the thumbnails, and then upload to s3.
2.75) if there's enough demand, i suppose i could put in a basic s3 explorer within the module which would allow you to import/select files directly from s3. the original would still need to be downloaded to the server temporarily so that g3 can extract the data it needs and generate the thumbnail/resize, and upload the results. since in this case though the original is already on s3, it could simply execute a "copy" command to copy the original into the g3s3 directory structure. this could work in a similar way to server-add i suppose, but browsing an s3 bucket/directory rather than a local directory. certainly is food for thought.

Dan

danneh.org :: Gallery3

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Tue, 2010-12-28 18:04

howdy

i've just posted beta 3 of this module. download here

Changelog:
- Resolved issues when moving items from album to album (organize hangs at 0 of x complete)
- Re-structured installer
- (potentially) resolved issues with server add

To install, download and extract to gallery3/modules, overwriting files that already exist. Now go to the database (phpmyadmin, etc). Browse to the "modules" table, find "aws_s3", edit the row, change the version field to "1" and save. Then drop the aws_s3_meta table. Re-visit Admin -> Modules on G3, it'll ask you to upgrade Amazon S3 to v2 again. Allow it to do so. (This is necessary due to a change in the aws_s3_meta table and a slight change into the way it populates the data in this table).

Errors are found in the usual places. Bug reports on a postcard :)

Dan

danneh.org :: Gallery3

 
Jonreynolds

Joined: 2008-03-09
Posts: 96
Posted: Tue, 2010-12-28 18:29
markusk wrote:
Jonreynolds wrote:
I still need to upload files to my server, then use 'server add' to add them to G3.

does 'server add' work for you while this module is enabled?

--
regular git puller.

Ahh, alas, it does not. Sorry I think I got a bit confused as to what I normally do on G2. There is no 'server add' option available at the mo in my G3, due to I guess the enabled S3 module.

Danneh, thanks for the answers, looking forward to that release using S3 as primary.

For now then, I need enough space for all the originals then? All the resized and thumbs are all on S3? Or are they stored on my server too?

Jon Reynolds
www.jcrdevelopments.com

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Tue, 2010-12-28 18:33

you need enough space for originals, resizes and thumbs. this module, currently, only copies everything that g3 generates to s3. so everything is stored on both s3 AND your server. that is what "primary storage" will change, but that's not implemented yet.

Dan

danneh.org :: Gallery3

 
Jonreynolds

Joined: 2008-03-09
Posts: 96
Posted: Tue, 2010-12-28 18:38

Danneh,

Oky dokey. Many thanks for clarifying things. Look forward to next release :)

Jon Reynolds
www.jcrdevelopments.com

 
Jonreynolds

Joined: 2008-03-09
Posts: 96
Posted: Tue, 2010-12-28 19:03

Just tried the new version. I unzipped B3 over the top of my existing aws_s3, replacing files.

I then went into mysql, found the modules table, found the aws_s3 row, but its version was already set to '1'. I went to site admin, modules, got prompted to upgrade, did so, saw it was aws that needed upgrading. Did so.

I could not find the aws_s3_meta table? Where should that be?

Now I have server add option.

Question:

The URLs for pictures I have added appear thus: http://www.jcrdevelopments.com/gallery3/var/albums/tests3/ASSY3D1.jpg?m=1293562295

previously, they seemed to reference aws.amazon.... directly. Is this right? Also the newly added images are not appearing in my s3 bucket...

Also, what do I need to do to the CDN URL? http://{bucket}.s3.amazonaws.com/g3/{guid}/{resource}

Do I leave it alone or do I change the bits in parentheses?

Cheers again!

Jon Reynolds
www.jcrdevelopments.com

 
Jonreynolds

Joined: 2008-03-09
Posts: 96
Posted: Tue, 2010-12-28 19:06

Hi ...again :)

Also, If I go into Maintenance and try and run sync with S3, I get an error: Unable to initialize schedule

Cheers

Jon Reynolds
www.jcrdevelopments.com

 
danneh3826
danneh3826's picture

Joined: 2007-08-18
Posts: 290
Posted: Tue, 2010-12-28 19:11

do you have the scheduler module installed also (the version which i released a week or two ago which has cron functions)? if so, have you enabled crontab on your server? if not, then i guess i need to look at this running without the scheduler. with scheduler installed, it defers upload to s3 for a cronjob to complete. it allows the upload process (to g3) to complete much quicker, especially for those running g3's on slow uplinks (i.e; home servers, etc).

i'll have a look at the resync.

Dan

danneh.org :: Gallery3

 
markusk

Joined: 2006-04-20
Posts: 60
Posted: Tue, 2010-12-28 22:19

ok.. bad updater me, didn't read the instructions (just replaced files and activated, only to see it wasn't working later on).. followed instructions afterwards, but seems it didn't work, told me there were modules to upgrade, but:
http://xchange.photing.com/data/screens/shot00533.png
now i'm in the state again where i can't enable the module..

think it'd be best to start over, is there anything else i gotta remove besides the entry in the module table? and guess i can ditch anything in the S3-bucket then, right?

or do u want to have a look at it on my server once more, dan? [credentials are still the same fwiw]

markus.

--
regular git puller.