Is background image resizing possible?

asc99c

Joined: 2005-01-03
Posts: 10

Posted: Mon, 2005-01-03 10:15

I like the speed of uploading images to gallery 2. I'm currently a little unhappy with the time taken to resize images on the first view. Even if I tick the box saying create / recreate sized images, I seem to have a 75 second wait the first time I view an added image while it gets resized. The thumbnails do create correctly.

What I would like is to set something off in the background which will go and call triggers to resize every image in the database. Has anyone got anything to do this sort of action? I presume it could be written as a script such as resize_all.php that can be set off at midnight and left running through the night.

----

Gallery URL (optional): http://andycunningham.homeip.net/gallery2
Gallery version: 2.0 alpha 4
Webserver (with version): Apache 2.0.46
Datatabase (with version): MySQL 3.23.51
PHP version (eg 4.2.1): 4.3.4
phpinfo URL (optional):
Graphics Toolkit(s): ImageMagick
Operating system: Windows XP Build 2600 SP2
Web browser/version: N/A

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Tue, 2005-01-04 05:37

75 seconds! Holy cow, something is seriously wrong there. On my P3 700Mhz box using ImageMagick it resizes a 4 megapixel image image down to 640x480 in about 4.5 seconds. That's a little slow but it's an order of magnitude faster than what you're seeing.

$ identify 2004_11_25_046.jpg 
2004_11_25_046.jpg JPEG 2304x1728 DirectClass 2.2mb 0.000u 0:01
$ time convert -geometry 640x480 2004_11_25_046.jpg resized.jpg 

real    0m4.562s
user    0m4.312s
sys     0m0.252s
$ identify resized.jpg 
resized.jpg JPEG 640x480 DirectClass 239kb 0.010u 0:01

Can you perform the same test on your box and see what the speed is like? I agree, we should have an option that lets you pre-generate all of the derivatives at upload time. We could easily add another checkbox like we have for thumbnails and do the same type of processing...

asc99c

Joined: 2005-01-03
Posts: 10

Posted: Tue, 2005-01-04 16:19

NB Sorry, the original of this post had me confused with what used to happen running NetPBM.

Just timed this action. Unfortunately I'm on Windows so I can't use time but I can see that it takes around 4 seconds. Part of the problem may be that I have used three resized image sizes (1600x1200, 1024x768, 640x480), all of which are created in one go. There does seem also to be a pause between each convert action where the CPU usage drops back to 0.

Another issue I have come across is probably just down to memory requirements. This machine has only 256MB RAM and due to all the stuff I have running in the background (Apache / MySQL / 3 VPN clients / VNC server / Skype / Google Desktop Search...) I'm usually using about 400MB of RAM so it pages badly. Convert looks to use about 60MB plus 80MB of virtual memory - dunno if this means 140MB total.

Turning off most of my applications I got the conversion time down to 25 seconds, so I guess I'm memory limited. Anyway, I've got upgrades ordered already. Just waiting for the nForce4 motherboards to come into stock (in the UK) and I'll be up to 1GB RAM.

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Tue, 2005-01-04 20:37

G2 will only create the images as it needs them, so if you look at the 1600x1200 then the the 1024x768, it'll only create them as you view them. Of course, if you have many people browsing your gallery at once it could be creating one image per browser...

There's also a bug with the combination of older versions of NetPBM and older versions of Windows that causes NetPBM to hang. We've never figured out what's going wrong here (in truth, it's a NetPBM bug not a Gallery bug). So if you're getting *really* long response times you should consider using ImageMagick instead..

asc99c

Joined: 2005-01-03
Posts: 10

Posted: Tue, 2005-01-04 22:52

Quote:

G2 will only create the images as it needs them

This isn't what's happening on my machine at the moment. The first time I access a newly uploaded image, I see the convert program run three separate times, and with three different convert commands in the form of:

"C:\ImageMagick\convert" "-quality" "90" "-size" "640x640" "-geometry" "640x640" "D:\httpdocs\gallery2\g2data\albums\christmas-2004\DSC01435.JPG" "D:\httpdocs\gallery2\g2data\tmp\img8F5.tmp"

with the other two others using size/geometry 1024 and then 1600. NB I got these command lines from Process Explorer.

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Wed, 2005-01-05 09:33

That's surprising. It sounds like a bug, but I can't reproduce it. Let's see if we can narrow down what's going on. Try this. Deactivate the random image block module, if it's active. Add a new image and stay on the confirmation page (where it gives you a link to the image). Then delete everything under your g2data/cache directory. Then click the link and wait for it to be done.

Now look in your g2data/cache/derivative tree. There should be directories with numbers, and in them should be two files: ##.dat and ##-meta.inc (where ## is the id of the derivative that just got generated). When I do this, I see only one .dat and one -meta.inc file. What do you see?

asc99c

Joined: 2005-01-03
Posts: 10

Posted: Wed, 2005-01-05 23:10

I have just done this set of actions and have got three pairs of files created in derivative\6\5 - 653, 655 and 656 - going by filesizes I assume 653 is the 640x480 image, 655 is 1024 and 656 is 1600.

A little more experimentation reveals that this same action happens when using NetPBM also. ImageMagick appears substantially quicker on my machine though. Looking at the processes in Process Explorer I see the NetPBM bug you described a couple of posts ago with the processes hung using no CPU time.

Just realised something that may be important - I have made some small changes to the code (quicker than reading docs to find out how to do it properly!). If you can't reproduce the bug there it may simply be something I did here and worth no more time tracking down. One change is simply adding a custom bit of HTML to the top of each page, which I doubt matters. The other I am not sure about. Gallery kept generating links going through http://andycunningham.homeip.net/php/php.exe/gallery2/ ... which just popped up a PHP security warning. I couldn't fix it by changing config settings so I delved into some URL generation code and just ran a str_replace to cut out the /php/php.exe part of the URL after which it seemed to work OK.

Anyway, I've partly resolved the problem. I've done a rather dumb standalone script which just makes the calls to the webserver asking to see each item in the gallery (well, just goes through all g2_itemIds including invalid ones). Running that overnight triggers the generation of any images which aren't already created.

bharat

Joined: 2002-05-21
Posts: 7994

Posted: Thu, 2005-01-06 09:02

Whoa. This is unusual. I'll keep trying to reproduce it, but if it's still happening in the next release would you file a bug against it please? Thanks!

Miss Universum

Joined: 2008-06-16
Posts: 40

Posted: Mon, 2009-02-02 10:43

Hm, I know this is quite an old thread... but have you found a solution for it? I'm experiencing problems when really big images are first called from the browser. It always runs into a timeout and images are created only the third or fourth time I reload the page. Is there any chance those resizes can be processed automatically (e.g. during times of low server load)?
ok, I could write a script calling the page, but that does not seem to be very elegant....

thanks,
MU

asc99c Joined: 2005-01-03 Posts: 10	Posted: Mon, 2005-01-03 10:15
	I like the speed of uploading images to gallery 2. I'm currently a little unhappy with the time taken to resize images on the first view. Even if I tick the box saying create / recreate sized images, I seem to have a 75 second wait the first time I view an added image while it gets resized. The thumbnails do create correctly. What I would like is to set something off in the background which will go and call triggers to resize every image in the database. Has anyone got anything to do this sort of action? I presume it could be written as a script such as resize_all.php that can be set off at midnight and left running through the night. ---- Gallery URL (optional): http://andycunningham.homeip.net/gallery2 Gallery version: 2.0 alpha 4 Webserver (with version): Apache 2.0.46 Datatabase (with version): MySQL 3.23.51 PHP version (eg 4.2.1): 4.3.4 phpinfo URL (optional): Graphics Toolkit(s): ImageMagick Operating system: Windows XP Build 2600 SP2 Web browser/version: N/A
	XUser login Username: * Password: * Create new account Request new password
bharat Joined: 2002-05-21 Posts: 7994	Posted: Tue, 2005-01-04 05:37
	75 seconds! Holy cow, something is seriously wrong there. On my P3 700Mhz box using ImageMagick it resizes a 4 megapixel image image down to 640x480 in about 4.5 seconds. That's a little slow but it's an order of magnitude faster than what you're seeing. $ identify 2004_11_25_046.jpg 2004_11_25_046.jpg JPEG 2304x1728 DirectClass 2.2mb 0.000u 0:01 $ time convert -geometry 640x480 2004_11_25_046.jpg resized.jpg real 0m4.562s user 0m4.312s sys 0m0.252s $ identify resized.jpg resized.jpg JPEG 640x480 DirectClass 239kb 0.010u 0:01 Can you perform the same test on your box and see what the speed is like? I agree, we should have an option that lets you pre-generate all of the derivatives at upload time. We could easily add another checkbox like we have for thumbnails and do the same type of processing...

asc99c Joined: 2005-01-03 Posts: 10	Posted: Tue, 2005-01-04 16:19
	NB Sorry, the original of this post had me confused with what used to happen running NetPBM. Just timed this action. Unfortunately I'm on Windows so I can't use time but I can see that it takes around 4 seconds. Part of the problem may be that I have used three resized image sizes (1600x1200, 1024x768, 640x480), all of which are created in one go. There does seem also to be a pause between each convert action where the CPU usage drops back to 0. Another issue I have come across is probably just down to memory requirements. This machine has only 256MB RAM and due to all the stuff I have running in the background (Apache / MySQL / 3 VPN clients / VNC server / Skype / Google Desktop Search...) I'm usually using about 400MB of RAM so it pages badly. Convert looks to use about 60MB plus 80MB of virtual memory - dunno if this means 140MB total. Turning off most of my applications I got the conversion time down to 25 seconds, so I guess I'm memory limited. Anyway, I've got upgrades ordered already. Just waiting for the nForce4 motherboards to come into stock (in the UK) and I'll be up to 1GB RAM.

bharat Joined: 2002-05-21 Posts: 7994	Posted: Tue, 2005-01-04 20:37
	G2 will only create the images as it needs them, so if you look at the 1600x1200 then the the 1024x768, it'll only create them as you view them. Of course, if you have many people browsing your gallery at once it could be creating one image per browser... There's also a bug with the combination of older versions of NetPBM and older versions of Windows that causes NetPBM to hang. We've never figured out what's going wrong here (in truth, it's a NetPBM bug not a Gallery bug). So if you're getting really long response times you should consider using ImageMagick instead..

asc99c Joined: 2005-01-03 Posts: 10	Posted: Tue, 2005-01-04 22:52
	Quote: G2 will only create the images as it needs them This isn't what's happening on my machine at the moment. The first time I access a newly uploaded image, I see the convert program run three separate times, and with three different convert commands in the form of: "C:\ImageMagick\convert" "-quality" "90" "-size" "640x640" "-geometry" "640x640" "D:\httpdocs\gallery2\g2data\albums\christmas-2004\DSC01435.JPG" "D:\httpdocs\gallery2\g2data\tmp\img8F5.tmp" with the other two others using size/geometry 1024 and then 1600. NB I got these command lines from Process Explorer.

bharat Joined: 2002-05-21 Posts: 7994	Posted: Wed, 2005-01-05 09:33
	That's surprising. It sounds like a bug, but I can't reproduce it. Let's see if we can narrow down what's going on. Try this. Deactivate the random image block module, if it's active. Add a new image and stay on the confirmation page (where it gives you a link to the image). Then delete everything under your g2data/cache directory. Then click the link and wait for it to be done. Now look in your g2data/cache/derivative tree. There should be directories with numbers, and in them should be two files: ##.dat and ##-meta.inc (where ## is the id of the derivative that just got generated). When I do this, I see only one .dat and one -meta.inc file. What do you see?

asc99c Joined: 2005-01-03 Posts: 10	Posted: Wed, 2005-01-05 23:10
	I have just done this set of actions and have got three pairs of files created in derivative\6\5 - 653, 655 and 656 - going by filesizes I assume 653 is the 640x480 image, 655 is 1024 and 656 is 1600. A little more experimentation reveals that this same action happens when using NetPBM also. ImageMagick appears substantially quicker on my machine though. Looking at the processes in Process Explorer I see the NetPBM bug you described a couple of posts ago with the processes hung using no CPU time. Just realised something that may be important - I have made some small changes to the code (quicker than reading docs to find out how to do it properly!). If you can't reproduce the bug there it may simply be something I did here and worth no more time tracking down. One change is simply adding a custom bit of HTML to the top of each page, which I doubt matters. The other I am not sure about. Gallery kept generating links going through http://andycunningham.homeip.net/php/php.exe/gallery2/ ... which just popped up a PHP security warning. I couldn't fix it by changing config settings so I delved into some URL generation code and just ran a str_replace to cut out the /php/php.exe part of the URL after which it seemed to work OK. Anyway, I've partly resolved the problem. I've done a rather dumb standalone script which just makes the calls to the webserver asking to see each item in the gallery (well, just goes through all g2_itemIds including invalid ones). Running that overnight triggers the generation of any images which aren't already created.

bharat Joined: 2002-05-21 Posts: 7994	Posted: Thu, 2005-01-06 09:02
	Whoa. This is unusual. I'll keep trying to reproduce it, but if it's still happening in the next release would you file a bug against it please? Thanks!

Miss Universum Joined: 2008-06-16 Posts: 40	Posted: Mon, 2009-02-02 10:43
	Hm, I know this is quite an old thread... but have you found a solution for it? I'm experiencing problems when really big images are first called from the browser. It always runs into a timeout and images are created only the third or fourth time I reload the page. Is there any chance those resizes can be processed automatically (e.g. during times of low server load)? ok, I could write a script calling the page, but that does not seem to be very elegant.... thanks, MU

Is background image resizing possible?

XUser login