Storage Loacation?

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Wed, 2009-05-06 20:01

In the security guide for 2.3 it mentions that the storage directory for your images should NOT be in the document-root. In G3, the default location is in "<installdir>/var/albums". Did the security implications of locating the storage directory in the document-root change with the new arch?

Looks good so far. I'm running Alpha4 and I'm very pleased. It's working quickly. I've also tested server-add and G2 import. The import has stalled a couple times, but a pause/restart gets it going again. For alpha, that's pretty good IMO.

The site is just for testing, but I wanted to see how the G2 importer was working. I like the ability to pause and restart it. I could live with it being faster, but I think that's got more to do with my webhost than anything else. I'll fool around with it for a while. Looking forward to the integration features. I'd love to get a better embedded experience with Drupal.

Thanks for the excellent software.

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Thu, 2009-05-07 00:19

Yes, we decided that it was impractical and difficult to enforce that restriction. It made the product much more complicated, required us to have an image firewall, etc. Instead we're going for the approach of making the var dir secure. This makes the resulting product much faster and lighter. It makes the installer easier, etc.

The bulk of the time spent in the importer is rebuilding your image thumbnail and resizes. I suspect that we can make the import *way* faster if we just copy over those images, but this comes at the cost of some code complexity so we haven't tackled it yet. Because G2 is so configurable, it'd be hard to get it right for everybody so we're taking the easy (but slow) approach for now. Glad to hear you're enjoying it so far.

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Thu, 2009-05-07 00:37

Sounds good. The new installer is certainly simpler. I really liked how easy it was to set up.

I thought that the importer was just copying all the images, that would explain the slowdown. If it's needed to make it reliable, that's the way to go.

Hopefully I can find a bug or something to contribute.

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Sat, 2009-05-09 19:11

I just committed a fix to copy over the G2 images where possible and this makes the photo import phase 2-3x faster. Try it out in a nightly release or from svn!

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Sat, 2009-05-09 20:02

Using svn I installed a fresh instance of G3 with a clean database. I get this error when I attempt to access the settings for the import module after activating it in the admin interface.

modules/g2_import/controllers/admin_g2_import.php [31]:

Undefined variable: g2_stats

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Sat, 2009-05-09 20:18

Oops, I just introduced that bug an hour ago. It is fixed in r20708

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Sat, 2009-05-09 20:29

That got me in. When I enter the path and click save, it gives me this error now:

modules/g2_import/helpers/g2_import.php [352]:

Undefined index: r

Hitting back shows a message that it saved the path though.

I am able to start the importer, and it is significantly faster. Thanks for the great work. I'll let it run and play with some albums and such when it's done. Let me know if I can help test for the new error. I'm fine with wiping it out and starting fresh to get a good test setup.

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Sat, 2009-05-09 20:35

I can't reproduce that error. What's on line 352 of g2_import.php for you?

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Sat, 2009-05-09 20:46

if ($row[0] > $sizes[$type["count"]]) {

Doesn't look like anything is wrong to me. I'll try again with a fresh install to see if I can reproduce here.

Hmmm... I paused the importer and went to check the settings page again and it's giving me the same error there now. Weird.

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Sat, 2009-05-09 20:52

Well, it's fast to get going from scratch again.

I get the same error on a fresh install.

I purged the database (drop all tables) and deleted the gallery3 directory.
Did a fresh checkout (svn co https://gallery.svn.sourceforge.net/svnroot/gallery/gallery3/trunk/ gallery3)
Ran the installer and gave it the database info
Installed the import module
Ran the settings for the import module and entered the path to embed.php
Now I get the same error whenever I try to access the settings page for the importer.

That's all I did. I can try creating a fresh database, but that takes a little longer.

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Sat, 2009-05-09 21:12

Doh, my bad! I fixed it in r20709 .. try another svn up!

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Sun, 2009-05-10 03:30

Now that's service! All is well. Thanks for the quick updates. I'm on svn now, so I'll be able to update as I go along. I'll post again if I find anything to report.

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Sun, 2009-05-10 03:42

OK, I just can't stay away.

Error in the log when running the importer. Looks like I'm missing a file in my old gallery, but the importer should probably catch the error and continue, shouldn't it? I did verify that the file does not exist. I have a 36 and a 38, no 37.

The importer freezes and doesn't seem to be doing anything at all. So I checked the log and saw this:

2009-05-09 20:34:47 -07:00 --- error: Uncaught PHP Error: copy(/home/kb7sei/tabbal.net_gallery_g2data/cache/derivative/0/3/37.dat)
[<a href='function.copy'>function.copy</a>]: failed to open stream: No such file or directory in file modules/g2_import/helpers/g2_import.php on line 316

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Sun, 2009-05-10 04:05

Nice! I just checked in a fix to make that code more robust. Try it again and let me know if it still fails.

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Sun, 2009-05-10 13:18

Great! It got past that bit and is importing photos now. It's running quite quickly. I like that it lets me know that the sizes are set differently in G2 so I can change them to make the importer run faster.

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Sun, 2009-05-10 13:30

Hmmm... it worked well till about 382. Then it stalled. I don't have any logs in the var dir, so I'm afraid this bug report isn't very useful. Perhaps if there were some extra logging added? Or is there a debug log setting?

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Sun, 2009-05-10 20:07

The new tasks infrastructure relies on the browser to keep sending requests.. sometimes that gets interrupted. We haven't made it robust enough yet to keep retrying so possibly the connection dropped and a request got lost. Go to Admin -> Maintenance and try resuming the task.. does it pick up where it left off?

kb7sei

Joined: 2003-07-29
Posts: 16

Posted: Mon, 2009-05-11 04:13

Hmmm... odd. It didn't seem to before, but it's working now. Perhaps I just wasn't patient enough before. Thanks again.

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Wed, 2009-05-06 20:01
	In the security guide for 2.3 it mentions that the storage directory for your images should NOT be in the document-root. In G3, the default location is in "<installdir>/var/albums". Did the security implications of locating the storage directory in the document-root change with the new arch? Looks good so far. I'm running Alpha4 and I'm very pleased. It's working quickly. I've also tested server-add and G2 import. The import has stalled a couple times, but a pause/restart gets it going again. For alpha, that's pretty good IMO. The site is just for testing, but I wanted to see how the G2 importer was working. I like the ability to pause and restart it. I could live with it being faster, but I think that's got more to do with my webhost than anything else. I'll fool around with it for a while. Looking forward to the integration features. I'd love to get a better embedded experience with Drupal. Thanks for the excellent software.
	XUser login Username: * Password: * Create new account Request new password
bharat Joined: 2002-05-21 Posts: 7994	Posted: Thu, 2009-05-07 00:19
	Yes, we decided that it was impractical and difficult to enforce that restriction. It made the product much more complicated, required us to have an image firewall, etc. Instead we're going for the approach of making the var dir secure. This makes the resulting product much faster and lighter. It makes the installer easier, etc. The bulk of the time spent in the importer is rebuilding your image thumbnail and resizes. I suspect that we can make the import way faster if we just copy over those images, but this comes at the cost of some code complexity so we haven't tackled it yet. Because G2 is so configurable, it'd be hard to get it right for everybody so we're taking the easy (but slow) approach for now. Glad to hear you're enjoying it so far.

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Thu, 2009-05-07 00:37
	Sounds good. The new installer is certainly simpler. I really liked how easy it was to set up. I thought that the importer was just copying all the images, that would explain the slowdown. If it's needed to make it reliable, that's the way to go. Hopefully I can find a bug or something to contribute.

bharat Joined: 2002-05-21 Posts: 7994	Posted: Sat, 2009-05-09 19:11
	I just committed a fix to copy over the G2 images where possible and this makes the photo import phase 2-3x faster. Try it out in a nightly release or from svn!

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Sat, 2009-05-09 20:02
	Using svn I installed a fresh instance of G3 with a clean database. I get this error when I attempt to access the settings for the import module after activating it in the admin interface. modules/g2_import/controllers/admin_g2_import.php [31]: Undefined variable: g2_stats

bharat Joined: 2002-05-21 Posts: 7994	Posted: Sat, 2009-05-09 20:18
	Oops, I just introduced that bug an hour ago. It is fixed in r20708

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Sat, 2009-05-09 20:29
	That got me in. When I enter the path and click save, it gives me this error now: modules/g2_import/helpers/g2_import.php [352]: Undefined index: r Hitting back shows a message that it saved the path though. I am able to start the importer, and it is significantly faster. Thanks for the great work. I'll let it run and play with some albums and such when it's done. Let me know if I can help test for the new error. I'm fine with wiping it out and starting fresh to get a good test setup.

bharat Joined: 2002-05-21 Posts: 7994	Posted: Sat, 2009-05-09 20:35
	I can't reproduce that error. What's on line 352 of g2_import.php for you?

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Sat, 2009-05-09 20:46
	if ($row[0] > $sizes[$type["count"]]) { Doesn't look like anything is wrong to me. I'll try again with a fresh install to see if I can reproduce here. Hmmm... I paused the importer and went to check the settings page again and it's giving me the same error there now. Weird.

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Sat, 2009-05-09 20:52
	Well, it's fast to get going from scratch again. I get the same error on a fresh install. I purged the database (drop all tables) and deleted the gallery3 directory. Did a fresh checkout (svn co https://gallery.svn.sourceforge.net/svnroot/gallery/gallery3/trunk/ gallery3) Ran the installer and gave it the database info Installed the import module Ran the settings for the import module and entered the path to embed.php Now I get the same error whenever I try to access the settings page for the importer. That's all I did. I can try creating a fresh database, but that takes a little longer.

bharat Joined: 2002-05-21 Posts: 7994	Posted: Sat, 2009-05-09 21:12
	Doh, my bad! I fixed it in r20709 .. try another svn up!

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Sun, 2009-05-10 03:30
	Now that's service! All is well. Thanks for the quick updates. I'm on svn now, so I'll be able to update as I go along. I'll post again if I find anything to report.

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Sun, 2009-05-10 03:42
	OK, I just can't stay away. Error in the log when running the importer. Looks like I'm missing a file in my old gallery, but the importer should probably catch the error and continue, shouldn't it? I did verify that the file does not exist. I have a 36 and a 38, no 37. The importer freezes and doesn't seem to be doing anything at all. So I checked the log and saw this: 2009-05-09 20:34:47 -07:00 --- error: Uncaught PHP Error: copy(/home/kb7sei/tabbal.net_gallery_g2data/cache/derivative/0/3/37.dat) [<a href='function.copy'>function.copy</a>]: failed to open stream: No such file or directory in file modules/g2_import/helpers/g2_import.php on line 316

bharat Joined: 2002-05-21 Posts: 7994	Posted: Sun, 2009-05-10 04:05
	Nice! I just checked in a fix to make that code more robust. Try it again and let me know if it still fails.

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Sun, 2009-05-10 13:18
	Great! It got past that bit and is importing photos now. It's running quite quickly. I like that it lets me know that the sizes are set differently in G2 so I can change them to make the importer run faster.

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Sun, 2009-05-10 13:30
	Hmmm... it worked well till about 382. Then it stalled. I don't have any logs in the var dir, so I'm afraid this bug report isn't very useful. Perhaps if there were some extra logging added? Or is there a debug log setting?

bharat Joined: 2002-05-21 Posts: 7994	Posted: Sun, 2009-05-10 20:07
	The new tasks infrastructure relies on the browser to keep sending requests.. sometimes that gets interrupted. We haven't made it robust enough yet to keep retrying so possibly the connection dropped and a request got lost. Go to Admin -> Maintenance and try resuming the task.. does it pick up where it left off?

kb7sei Joined: 2003-07-29 Posts: 16	Posted: Mon, 2009-05-11 04:13
	Hmmm... odd. It didn't seem to before, but it's working now. Perhaps I just wasn't patient enough before. Thanks again.

Storage Loacation?

XUser login