"too many open files" while importing images

gohrner

Joined: 2005-07-11
Posts: 1
Posted: Mon, 2005-07-11 09:17

Hi!

Finally I got g2 to work reliably, using PostgreSQL instead of MySQL as the backend db server.

However, if I import a lot of image files which are available on the server, gallery2 is busy for some time and then I the following error message:

Warning: fopen(/usr/local/share/gallery2-images/locks/1/4/14003): failed to open stream: Too many open files in /usr/share/gallery2/modules/core/classes/GalleryPlatform.class on line 347

Error Detail -
Error (ERROR_PLATFORM_FAILURE) : /usr/local/share/gallery2-images/locks/1/4/14003
in modules/core/classes/FlockLockSystem.class at line 255 (gallerystatus::error)
in modules/core/classes/FlockLockSystem.class at line 98 (flocklocksystem::_acquirelock)
in modules/core/classes/helpers/GalleryLockHelper_simple.class at line 183 (flocklocksystem::acquirewritelock)
in modules/core/classes/GalleryCoreApi.class at line 2095 (gallerylockhelper_simple::acquirewritelock)
in modules/exif/ExifDescriptionOption.inc at line 69 (gallerycoreapi::acquirewritelock)
in modules/core/ItemAdd.inc at line 143 (exifdescriptionoption::handlerequestafteradd)
in main.php at line 183 (itemaddcontroller::handlerequest)
in main.php at line 87
in main.php at line 78

Looks as if g2 opeans each image file and never closes the stream.

Greetings,

Gunter
----

Gallery URL (optional): --
Gallery version: 2 snapshot 20050708
Webserver (with version): Apache 2.0.54
Datatabase (with version): PostgreSQL 8.0.3
PHP version (eg 4.2.1): 4.3.10
phpinfo URL (optional): --
Graphics Toolkit(s): GD; ImageMagick, NetPBM
Operating system: Linux 2.6 x86_64
Web browser/version: Konqqqueror 3.4.0
G1 version (for migration bugs): --
 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Mon, 2005-07-11 09:35

Thanks for reporting, I've looked at the code for a few seconds and you may be right. Don't have time right now for a real inspection.
I filed a bug:
http://sourceforge.net/tracker/index.php?func=detail&aid=1235921&group_id=7130&atid=107130
 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Mon, 2005-07-11 11:47

gohrner, " if I import a lot of image files which are available on the server", what add item method did you choose and how many items did you add at once?
a recursive item add from local server?
 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Mon, 2005-07-11 11:57

i've looked at the code and it releases the locks when it should.

bottom line:
if you tried to add hundreds or thousands of files at once, this may explain it.
then you should either add items in smaller chunks, fix your open files maximum, or change to database based locking in site admin -> general at the bottom of the page.
 
valiant

Joined: 2003-01-04
Posts: 32509
Posted: Mon, 2005-07-11 14:34

i've fixed a small bug in FlockLockSystem.class, but this but is unrelated to your problem.
 
wsloand

Joined: 2004-07-12
Posts: 17
Posted: Mon, 2006-05-01 15:26

I just had the same error with the current (2.1.1-1) debian package (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=365442). Since this error exists and database locking should fix the problem, could database locking be made the default and file locking be a suggested performance enhancement (with the warning about file locking).
 
Roland B Roberts

Joined: 2006-03-27
Posts: 10
Posted: Fri, 2006-05-19 19:54

You need to increase ulimit -n for the webserver. I just ran into this problem while recreating all the resized images on my site. The soft limit is probably much lower than the hard limit, perhaps as low as 1024. That sounds big, but if you are loading a couple hundred files with several images sizes and thumbnails, you may run out of file handles. I strongly suspect something in Gallery or PHP is failing to close some of those file handles when it should (perhaps they are just going out of scope and not being closed right away?).

In any event, on my Fedora Core 5 host, I added "ulimit -n 65535" to /etc/sysconfig/httpd and was then able to resize all of my images.
 
eye1marie

Joined: 2006-05-21
Posts: 13
Posted: Sun, 2006-05-21 20:10
wsloand wrote:
I just had the same error with the current (2.1.1-1) debian package (http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=365442). Since this error exists and database locking should fix the problem, could database locking be made the default and file locking be a suggestedd performance enhancement (with the warning about file locking).

I had thesame problem, but indeed its fixed now. Thanks!
 
cww

Joined: 2006-08-30
Posts: 2
Posted: Wed, 2006-09-06 04:17

I had the same problem when trying to import about 4,000 images from Gallery v1.5.4. It stalled somewhere around 15% with a really long list of errors that started with the thing about having too many open files.

Quote:
if you tried to add hundreds or thousands of files at once, this may explain it.
then you should either add items in smaller chunks, fix your open files maximum, or change to database based locking in site admin -> general at the bottom of the page.

The database locking solution worked on PostgreSQL 8.1.4. Thanks!

Colin
 
relegated

Joined: 2009-06-29
Posts: 11
Posted: Tue, 2009-06-30 02:18

The database locking also fixed my issue posted here http://gallery.menalto.com/node/88684
 
annorax
annorax's picture

Joined: 2002-07-24
Posts: 6
Posted: Sun, 2010-10-24 16:08
Roland B Roberts wrote:
You need to increase ulimit -n for the webserver. I just ran into this problem while recreating all the resized images on my site. The soft limit is probably much lower than the hard limit, perhaps as low as 1024.
.
.
.

In any event, on my Fedora Core 5 host, I added "ulimit -n 65535" to /etc/sysconfig/httpd and was then able to resize all of my images.

I confirmed that this works on Fedora Core 13 as well; however, I just bumped up the number to 2048 instead of MAXINT. I don't want anything running away on my web server and bringing it resoundingly to it knees. I'll just keep nudging up the number as needed.