fcgi timeout because of Gallery2 intensive tasks

amazeika

Joined: 2008-06-08
Posts: 33
Posted: Wed, 2008-10-08 19:57

Gallery version : 2.2.4
PHP version : 5.2.6 cgi-fcgi
Webserver : Apache/2.2.9 (Unix)
Database : mysql (5.0.51a-community)
Activated toolkits : ImageMagick 6.0.7
Operating system : Linux
PHP memory_limit: 128 MB

I just recompiled apache with fcgi support for PHP, since I'm trying out xcache to speed things up in gallery2. Everything seems to work fine except for a timeout problem when a heavy gallery2 administration task is requested. I recently posted this thread about scripts dying before completing their tasks when a lot of items/albums are involved:

http://gallery.menalto.com/node/82241

Now fcgi just kick me out with with Internal Server Error because of a timeout when the task exceeds 40 secs. This is my apache error_log:

[Wed Oct 08 14:29:03 2008] [warn] mod_fcgid: read data timeout in 40 seconds
[Wed Oct 08 14:29:03 2008] [warn] (110)Connection timed out: mod_fcgid: ap_pass_brigade failed in handle_request function
[Wed Oct 08 14:29:03 2008] [error] [client xxx.xxx.xxx.xxx] File does not exist: /home2/amazeika/public_html/mifondo/gallery2/500.shtml, referer: http://gallery.mifondo.net/main.php
[Wed Oct 08 14:29:13 2008] [warn] mod_fcgid: process 15242 graceful kill fail, sending SIGKILL

I saw that this timeout could be increased with these in httpd.conf:

IPCConnectTimeout
IPCCommTimeout

, but is this the right solution for this problem?. Should I add unlimited for gallery2 to finish its task?
Less intensive tasks, like changing the album config for an album with one picture works fine. This problem is the same as the one I already posted, the only difference now is that fcgi kicks me out. At least now I have more information to deal with the problem ;D.
Please, I need your help guys.
Thx again.

 
nivekiam
nivekiam's picture

Joined: 2002-12-10
Posts: 16504
Posted: Wed, 2008-10-08 20:46

What's your timeout setting for PHP?
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Wed, 2008-10-08 21:46

max_execution_time 120
Tested with 0 -> unlimited, same result.

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Thu, 2008-10-09 21:17

I spent almost the whole day trying to overcome this problem without success. Tried about every possible tweak in php.ini regarding timeouts for PHP and mysql. I also tried about any combination with fcgid timeouts and with the timeout option in httpd.conf. Even tried set_time_limit(0) in main.php. No success. With enormous timeouts in each possible paramter I had something slightly different in my php error log:

[Thu Oct 09 15:48:20 2008] [warn] mod_fcgid: process 28922 graceful kill fail, sending SIGKILL
[Thu Oct 09 15:48:20 2008] [warn] (104)Connection reset by peer: mod_fcgid: read data from fastcgi server error.
[Thu Oct 09 15:48:20 2008] [warn] (104)Connection reset by peer: mod_fcgid: ap_pass_brigade failed in handle_request function
[Thu Oct 09 15:48:20 2008] [error] [client xxx.xxx.xxx.xxx] File does not exist: /home2/amazeika/public_html/mifondo/gallery2/500.shtml, referer: http://gallery.mifondo.net/main.php?g2_view=core.ItemAdmin&g2_subView=core.ItemEdit&g2_itemId=121100&g2_return=%2Fv%2Fmifondo%2F&g2_returnName=%C3%A1lbum

No timeout but a direct kill from mod_fcgid at about 4 minutes after launch. Am I the only one that is having this problem?. Could someone try to reproduce this. All it takes is a big album and try to make a change for all its elements in the admin interface. I read about HTTP timeouts, of about 5 minutes, that kills scripts that are still running regardless of the timeouts parameters. With all the tweaks (max timeouts in all the parameters) the script lives approximately for 4 minutes. The problem is that I have no idea of how to change this HTTP timeouts.

Perhaps I forgot to specify that I'm not able to see the progress bar in the browser. Gallery2 logfiles don't show any problem at all, just normal database processing.

I'm in a desperate position here. Please. I can provide any log you may require, but please help me. I need to be able to make changes on my albums, perhaps launching the scripts directly from shell, I don't know if this is possible.

 
nivekiam
nivekiam's picture

Joined: 2002-12-10
Posts: 16504
Posted: Thu, 2008-10-09 21:37

You said you're on a dedicated server, right. Can you try increasing available memory for PHP to 256MB and let us know if that changes anything?

____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Thu, 2008-10-09 22:08

Yes nivekiam, I'm on a dedi. Just retried with 256M, no success. Script died 4 minutes after execution. Didn't get the progress bar neither. Perhaps is this bar what keeps the whole thing alive?, by constantly sending info to the browser. I’m all ears.

 
nivekiam
nivekiam's picture

Joined: 2002-12-10
Posts: 16504
Posted: Thu, 2008-10-09 22:18

Yes, that is one of the main purposes of the progress bar, well it's also a usability feature. IIRC, there is some configs where that progress bar won't work.

Can you post your system info from Site Admin > Maintenance and a link to phpinfo?

If I can't think of something, I'll see if I can get someone with more experience to take a look in on this thread.
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Fri, 2008-10-10 08:48
Quote:
If I can't think of something, I'll see if I can get someone with more experience to take a look in on this thread.

And for that I'm very very grateful. I have dedicated a considerable amount of time on this site, and the fact of being unable to administrate it is driving me crazy. Thank you again nivekiam.

As requested, the system info (it's in spanish, I can get it in english but I think the info you need is clearly visible):

Versión de Gallery = 2.2.4 núcleo 1.2.0.6
Versión de PHP = 5.2.6 cgi-fcgi
Servidor Web = Apache/2.2.9 (Unix) mod_ssl/2.2.9 OpenSSL/0.9.7a mod_auth_passthrough/2.1 mod_bwlimited/1.4 FrontPage/5.0.2.2635
Base de Datos = mysqli 5.0.51a-community, lock.system=flock
Herramientas = ArchiveUpload, Exif, Getid3, ImageMagick
Aceleración = none, none
Sistema Operativo = Linux host.gamersassembly.net 2.6.9-78.0.1.EL #1 Tue Aug 5 10:49:42 EDT 2008 i686
Tema por defecto = carbon
gettext = habilitado (which means on)
Fichero de idioma = es_ES
Buscador = Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_5; fr-fr) AppleWebKit/525.18 (KHTML, like Gecko) Version/3.1.2 Safari/525.20.1
Filas en la tabla GalleryAccessMap = 56
Filas en la tabla GalleryAccessSubscriberMap = 101946
Filas en la tabla GalleryUser = 109
Filas en la tabla GalleryItem = 101944
Filas en la tabla GalleryAlbumItem = 4226
Filas en la tabla GalleryCacheMap = 0

For the phpinfo, I'm sending you the link via private message.

Thank you nivekiam. I really appreciate your help on this.

Best regards.

 
nivekiam
nivekiam's picture

Joined: 2002-12-10
Posts: 16504
Posted: Fri, 2008-10-10 15:08

Your memory_limit for PHP is 64MB, not 128 or 256. I can't be certain, but with a Gallery your size I'd really suspect php might be dieing because of that. I don't know for sure, but when things just die without giving a reason, I try to rule out as much stuff as possible.

Also, when you get a chance, I'd upgrade to 2.2.6 as there are several security fixes between 2.2.4 and 2.2.6 that should be applied.
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Fri, 2008-10-10 15:27
Quote:
Your memory_limit for PHP is 64MB, not 128 or 256

Hi nivekiam, I changed it yesterday to 64MB before going to bed. I tried with 128M and 256M pretty much the same result, it dies 4 minutes after. I think that perhaps, if memory is the problem, the logic should be, if I double the memory I should also double the execution time, and thats not the result I'm having. I don't have an Out of memory on my logs neither that would point to a memory error.

Do you think that there is something else that I could try out?.

 
nivekiam
nivekiam's picture

Joined: 2002-12-10
Posts: 16504
Posted: Fri, 2008-10-10 16:07

This is testing my skills ;)

Have you looked in the php error log? Since error_log is set to "error_log" I'm not sure where that would end up. Maybe nowhere? Maybe try changing the path for error_log for PHP errors to get logged some place.

http://www.cyberciti.biz/faq/error_log-defines-file-where-script-errors-logged/

If that doesn't get any more information, I'll try to get a dev looking at this thread. Is it o.k. with you to share the phpinfo link you sent me with the G2 devs?
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Fri, 2008-10-10 18:06
Quote:
This is testing my skills ;)

You've been doing great so far and I thank you for that ;D.

Quote:
Have you looked in the php error log? Since error_log is set to "error_log" I'm not sure where that would end up. Maybe nowhere? Maybe try changing the path for error_log for PHP errors to get logged some place.

I see what you mean. I'll try to specify a path and see if I can get more info. The outputs that I’ve been showing to you are from here: /usr/local/apache/logs/error_log which seems like the good location to me. I'll try a complete path in php.ini tomorrow since my wife is coming back home tonight, and computer (the competition from her point of view) will have to wait until tomorrow ;D.

Quote:
If that doesn't get any more information, I'll try to get a dev looking at this thread. Is it o.k. with you to share the phpinfo link you sent me with the G2 devs?

Of course it is ok nivekiam.

 
nivekiam
nivekiam's picture

Joined: 2002-12-10
Posts: 16504
Posted: Fri, 2008-10-10 18:40

I might use something like /usr/local/apache/logs/php.errors

http://us.php.net/manual/en/ref.errorfunc.php#49438

I'd also dig into how to log the most information from PHP so we can try to figure out exactly why it's dieing:
http://www.julian-bez.de/blog/2006/02/19/how-to-set-up-error-logging-with-php/
http://www.w3schools.com/PHP/php_ref_error.asp
http://www.php.net/errorfunc
http://www.google.com/search?q=php+error+logging

____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here

 
floridave
floridave's picture

Joined: 2003-12-22
Posts: 27300
Posted: Sat, 2008-10-11 04:07

I have been following this thread and can't offer any more than nivkiam has suggested.
Perhaps a dev will look at this thread.

Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team

 
nivekiam
nivekiam's picture

Joined: 2002-12-10
Posts: 16504
Posted: Sat, 2008-10-11 04:33

I hope the php errors give some clue.
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Sun, 2008-10-12 08:52

Hello nivekiam and dave.

I took a look to the links you provided. Regarding the path of error_log in php.ini, when is set to error_log, it will create an error_log file in the directory where the script generating the errors is located. In our case this is the main gallery2 directory.

The global path for error is not going to work correctly with my config, since I'm suexing (execution through suExec) fcgid, the wrapper that starts the php process. This means that php processes are executed as processes owned by the virtual host file owner, not root or nobody (apache user). The php.errors suggested by the first link should be writable by the php script owner, and if I run more virtual hosts it will then have to be writable by all of them too. I think the error_log approach in each directory is not bad after all for my config. I got my dedi a week ago, so I’m learning a bunch of new stuff here ;D.

Anyways, the bad news, there is nothing that points to php problem when the script dies. I tested php logging by making wrong php calls in scripts, and php logging is working correctly. error_reporting is set to E_ALL in php.ini. I don’t think we could get more info from here.

I'll dig a little bit more into apache logging. This is the one that at least is giving some information about the dying script. The logs that I provided in the previous posts are from apache log, not php as I stated. Perhaps I'll be able to get more information with proper settings in httpd.conf, but I'm not sure.

I'll keep you informed. Thx guys.

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Sun, 2008-10-12 20:11

Hello guys, I have GOOD news. I managed to keep the script alive !!! woohoo ;D. I finally found a configuration for mod_fcgid, my php handler, that allows php processes to run as long as desired. Tomorrow I'll give you the details. I really think this might interest others.

 
nivekiam
nivekiam's picture

Joined: 2002-12-10
Posts: 16504
Posted: Sun, 2008-10-12 20:23

Great! Yes that information will most likely be helpful to others.
____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here

 
floridave
floridave's picture

Joined: 2003-12-22
Posts: 27300
Posted: Mon, 2008-10-13 03:30

amazeika, looking forward to you findings.
perhaps you can add to:
http://codex.gallery2.org/Gallery2:Performance_Tips
when you find a chance.

Dave
_____________________________________________
Blog & G2 || floridave - Gallery Team

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Tue, 2008-10-14 19:51

As promised, this is how I managed to keep intensive gallery2 tasks running in the background. This will solve several timeout problems regarding intensive administrative tasks such as thumbnail generation, adding a considerable amount of images, changing the configuration of albums with lots of items, among others.

In my current PHP configuration, I’m using the latest fastcgi module (mod_fcgid) as my PHP handler, so this configuration will only be valid if you are using this module. Perhaps there is a similar configuration for its predecessors: mod_fcgi and mod_cgi.

You must be able to change your apache configuration by having root access into your server. There is little chance that you will be able to make this work in a shared server. Your hosting will certainly not like the idea of having php processes running in the background without any restraint.

I’m using a global configuration file for the parameterization of mod_fcgid. In my httpd.conf there is an include to a file named php.conf (where the mod_fcgid is loaded and configured). In this last file and after loading the module, I added :

#Gallery2 Admin Config Begin
BusyTimeout 10800
ProcessLifeTime 10800
IPCConnectTimeout 20
IPCCommTimeout 10800
MaxRequestsPerProcess -1
#Gallery2 Admin Config End

You should notice that you could also tune these parameters for each virtual host separately by simple adding these configuration parameters for each virtual host.

This is a configuration example for keeping the script alive for 10800 seconds, which is equivalent to 3 hours. Perhaps the most determinant values are BusyTimeout, IPCCommTimeout and MaxRequestsPerProcess. I think that ProcessLifeTime could be redundant; I have not yet made the test without it.

BusyTimeout and IPCCommTimeout are both going to determine the execution duration of the task. The one with the lower value will trigger first. BusyTimeout will terminate the script if a single request takes longer than busy timeout. IPCCommTimeout is the communication timeout to a fastcgi application. IPCConnectTimeout will handle the connection timeout only; the execution time does not depend on it.

MaxRequestsPerProcess is the last parameter that got my attention, and the one that finally made things work. This parameter was the responsible for this kill:

[Thu Oct 09 15:48:20 2008] [warn] mod_fcgid: process 28922 graceful kill fail, sending SIGKILL

[Thu Oct 09 15:48:20 2008] [warn] (104)Connection reset by peer: mod_fcgid: read data from fastcgi server error.

[Thu Oct 09 15:48:20 2008] [warn] (104)Connection reset by peer: mod_fcgid: ap_pass_brigade failed in handle_request function

[Thu Oct 09 15:48:20 2008] [error] [client xxx.xxx.xxx.xxx] File does not exist: /home2/amazeika/public_html/mifondo/gallery2/500.shtml, referer: http://gallery.mifondo.net/main.php?g2_view=core.ItemAdmin&g2_subView=core.ItemEdit&g2_itemId=121100&g2_return=%2Fv%2Fmifondo%2F&g2_returnName=%C3%A1lbum

By default this parameter is usually set to 500 for a good reason (http://fastcgi.coremail.cn/doc.htm). However, this also means that after 500 requests, the PHP process was condemned to die taking with him my precious gallery2 task. MaxRequestsPerProcess -1 means fastcgi process will not exit no matter how many requests it has handled, which is exactly what we are looking for. However, be aware that PHP errors may arise for new incoming requests when the PHP cleanup is performed, which is by default after 500 requests. This is a known problem caused by a race condition when running PHP in fastcgi mode through mod_fcgi under heavy server loads (http://fastcgi.coremail.cn/doc.htm for more information). This was corrected in mod_fcgid by adding MaxRequestsPerProcess, which by default is set to 500 to match PHP cleanup. This will kill the process after 500 requests, not good for us though. Perhaps changing the default PHP cleanup value to match MaxRequestsPerProcess (big enough) will ensure total stability while using this config.

Anyways, my final advice to you regarding this configuration is to only use it when performing heavy administrative and maintenance tasks. By default, MaxRequestsPerProcess 500 will ensure proper functioning of PHP in fastcgi mode at all times. It is also wiser to limit the execution time of scripts.

Notice that in my case, I always lost connection with the browser. However, tasks keep running in the background and get the job done.

I hope this thread will help some of you, as for me, it represents the solution to the only problem that I wasn’t able to deal with. I thank the developers for this magnificent piece of art. Excellent work !!!.

Thank you nivekiam and floridave for all your support and for following this thread with interest. Thank you guys.

If someone find that any of the information in this post is incorrect, please do not hesitate to point it out. As I said earlier, this is all new stuff for me, and it's my interpretation from all that I have learned this last week.

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Mon, 2008-10-13 12:38
floridave wrote:
perhaps you can add to:
http://codex.gallery2.org/Gallery2:Performance_Tips
when you find a chance.

No problem, I'll be glad to, but how?

 
amazeika

Joined: 2008-06-08
Posts: 33
Posted: Tue, 2008-10-14 18:47

Today I got some 500 errors because of timeouts again. Same config though. It appears that global config settings could be overrided by default settings if running virtual hosts, which is my case.

http://jay.vox.com/library/post/mod_fcgid-ignoring-fastcgi-config-settings.html

Still testing, I will specify the config directly for each virtual host and see what happens. I also think that this is the optimal config since I will be forcing timeouts for the gallery2 subdomain only.