Is there a max # of files gallery hand handle adding at a time?
|
DannyITR
Joined: 2003-12-17
Posts: 67 |
Posted: Mon, 2005-11-14 17:20
|
|
Gallery version (not just "2"): 2.1 I'm trying to add via the webpage option about 300 photos to a gallery. During the process the page times out and I do not get a succesfull confirmation of the added photos. The end result is either a gallery that has doubled the amount of images from anywhere to 500-600 or a gallery that only contains half the images in the folder I linked to. I included the trailing "/" at the end and its a brand new gallery installation with no hacks. What could be the problem? Thanks |
|
Posts: 32509
are you using phpnuke/g2?
g2 duplicating your operations, e.g. creating 2 albums instead of 1 etc is only known fro phpnuke/g2.
are you using add items -> from webpage? (not sure because some people mean add items from browser when they say from webpage).
from webpage:
- we don't have set_time_limit calls there, so we could actually time out, it seems
- we don't have a progressbar, so we also could timeout because of apache.
please file a feature request or bug report on http://sf.net/projects/gallery/
Posts: 67
Man this is bad. I dont think I can use gallery anymore if this persists. I can't even upload 40 files now at a time without it screwing up. I'll file the report.
Posts: 31
I'm having a similar issue. I'm trying to populate a gallery with 30000 images. Obviously this isn't going to work very well all in one album so I've split them into 2000 image zip files, uploaded them and am trying to add them each into their own album. Sometimes it works perfectly but mostly it adds way too many images. I haven't tracked down exactly what the extras are, but I know how many images should be in each folder and they tend to be ending up with almost double what they have.
I can't seem to get this to happen consistantly. I've tried in Firefox and Opera. Opera seems to pretty consistantly get a "connection closed by remote server" error, FF seems fine, but the duplicates are still showing.
Gallery version = 2.0.1 core 1.0.0.1
PHP version = 4.4.1 apache
Webserver = Apache
Database = mysql 4.1.14-log
Toolkits = ArchiveUpload, Exif, ImageMagick, NetPBM, Gd
Operating system = FreeBSD bsd3.qnetau.com 5.4-RELEASE-p4 FreeBSD 5.4-RELEASE-p4 #2: Sat Jul 16 13:52:26 UTC 2005
:/usr/obj/usr/src/sys/SERVER i386
Browser = Opera/8.50 (Windows NT 5.1; U; en)
I'm going to keep trying different things and see if I can figure anything further out.
Posts: 32509
"connection closed by remote server" -> most probably you reached the servers or PHPs upload limit. this limit is configurable but not in g2. it's a apache/php config issue.
duplicates? no idea how you get them. instead of adding zips to g2, i'd try to copy the images via ftp onto some folder on the server and then use add item -> from local server.
Posts: 31
Definitely not the server's upload limit. I'm FTPing the zips and going Add Item from the local server.
I've cut them down to 1000 image batches and it seems to be working better, although I did get one run which caused duplicates. It seems that somehow the browser session may be timing out and resubmitting the add item command or something.. Sometimes it manages to go through and shows the "Images Added Successfully" page, but sometimes if I leave it and wait until it gets to that stage it causes duplicates, so I've found it safer to monitor the number of images in the Album in another browser window and stop the Add Item window soon after the Album reaches the correct number of items.
Could it be something to do with php timing out and causing the browser to resubmit?
Posts: 32509
ftp and apache are two different things. i meant the apache server's limits. or php's.
a php timeout causing the browser to resubmit? never seen such a thing and doesn't sound very reasonable / possible. of course if you hit refresh or so, then it's possible.
Posts: 31
I don't think the apache or php upload limits are coming into it as the files are already sitting on the server (and I think I've got about 50mb upload limit set there anyway), it's just unzipping them to tmp and adding from there (or so it seems, not certain of the exact process).
From the different things I've been trying it seems to be something to do with the operation taking too long. If I use a smaller batch of files (up to around 1500) it works fine, much larger and it adds almost double the amount that's supposed to be in there. I haven't had the time to check whether they were duplicates or other files from the tmp directory though. I'm trying to get this upload finished, but if it will help I can try to figure out exactly what's going on
Posts: 32509
so you're using item add from local server and not add item from browser with archiveupload, that confused my a little. ok.
so it's either the php or the apache timeout. or something else
probably the apache timeout since we're taking care of the php timeout.
we're planning to add a progress bar to the add item views but haven't had time to do it yet.
Posts: 31
Yep, add item from server, sorry I wasn't clear.
If apache times out it should cause some kind of browser error shouldn't it? I wasn't getting any errors, but I wasn't letting it run through either. As soon as I saw that an album was getting too many items I stopped the add item window and started deleting the album so I could try again.
Tomorrow I'll do some tests to figure out exactly what's going on from my end. I remember reading something about an http header browser plugin and putting gallery into debug mode. Anything else I should do to maximise the useful information I get?
Posts: 32509
"stopped the add item window"
that's not possible. you may stop the browser from loading, but you can't stop the g2 process.
and if you start deleting while it's still adding in the background, you'll end up with funny issues maybe. actually, it shouldn't allow you to start deleting when it is still adding in another process.
Posts: 31
As the window is still sitting there "loading" it's possible to stop it. I did notice that the add item process continued on (watching the processes in phpmyadmin). That aside, I was able to start deleting several times while the add item process was still going, however this was when the add had "gone wrong" and was adding duplicates. Take from that what you will.
I'll do some tests today to see if I can replicate this issue on another different gallery install on a different host.
Posts: 32509
thanks for testing
Posts: 31
Ok, it just happened again. This time with only 974 images (added from a zip file). It ended up with 1948 images in the album, 2 of each.
Posts: 32509
and your g2 is an unmodified, standalone g2 (not embedded)?
steps to reproduce?
1. create zip file of a few images (e.g. 10 images)
2. activate archiveupload module
3. upload zip
expected results:
added 10 images
actual results:
added 20 images, all images twice
is that correct? then i'll check what i get.
Posts: 31
This is using Firefox 1.0.4. If I use Opera (my main browser) it ends up with the "connection closed by remote server" error, which I'm guessing is related to a php timeout. When this happens the operation (it just occured when I was deleting the album with 1948 images) stops.
I've been adding images exclusively with FF though.
Posts: 32509
should i be able to reproduce the issue with the above steps?
Posts: 31
Yes, unmodified install. Standalone g2.
I doubt it will work with a small number of images though, it only seems to happen if the Add Items operation takes a particularly long time. So perhaps try with 2000 images rather than just 10? I've had a couple work with up to 1500 images, but they're quite small, the zip files are only 10-20mb.
When it does happen it seems to add the images in the same order twice, so it goes
4999.jpg
4000.jpg
4001.jpg
...
4999.jpg
4001.jpg
4002.jpg
Posts: 32509
arg, i was again confused. of course this has nothing to do with archiveupload / zip files.
you're just using zip to transmit data from your pc to the server.
then you extract the zip.
then you use add items -> from server to add the items.
correct?
i did that already several times myself, with 1000+ items etc. and never had such issues.
Posts: 31
I'm not extracting the zip manually. Process:
Upload zip
Add new album
"Add a photo"
From Local Server > find files
Zip files show up
Check correct zip file and hit "Add Files"
Sometimes it works perfectly, sometimes it has the duplicates.
Posts: 67
I've been trying unsuccefully to upload files for a few days now using all three methods in Firefox and Internet Explorer. Any ideas?
Posts: 32509
DannyITR
i had a few question in my first response to your initial inquiry but you didn't answer them...
and in what way does
"add items" -> "from browser" fail?
Posts: 32509
jaymis:
just created a zip of 298 files and added it via local server, it extracted and added 298 files as it should have.
as for larger numbers. i don't see why it should break.
Posts: 31
Hi Valiant, I'm uploading a zip file to a different server to test if it's specific to one setup.
What I've been describing is quite consistant though, and it's consistantly more likely to happen with a larger number of images.
I'll do some tests when this upload's complete and see how it goes.
j
Posts: 31
I've been doing some testing with an alternate server (http://jaymis.com/gallery/) and it's exhibiting similar behaviour. Jaymis.com seems to be processing the images much faster though, so it takes more images before this behaviour appears. As it's processing I've got a second browser open showing the main gallery page and I refresh it to watch how many images are in the album. On the initial gallery I was working with it only processes 5-10 images/second. On jaymis.com it's processing almost 50 images/second.
An import of 950 files works perfectly (although it didn't show the "items successfully added" page), but an import of 3000 files failed and did the duplicate import thing as previously described. All of these imports are zip files using the technique described 5 comments up this thread.
Posts: 31
I also just went to delete this Album (which had around 4500 images by the time the add operation had stalled) and it's come up with an error:
Error (ERROR_LOCK_TIMEOUT) : /home/jaymis/g2data/locks/5/7/572
* in modules/core/classes/FlockLockSystem.class at line 309 (gallerystatus::error)
* in modules/core/classes/FlockLockSystem.class at line 98 (flocklocksystem::_acquirelock)
* in modules/core/classes/helpers/GalleryLockHelper_simple.class at line 185 (flocklocksystem::acquirewritelock)
* in modules/core/classes/GalleryCoreApi.class at line 2113 (gallerylockhelper_simple::acquirewritelock)
* in modules/core/classes/helpers/GalleryEntityHelper_medium.class at line 52 (gallerycoreapi::acquirewritelock)
* in modules/core/classes/GalleryCoreApi.class at line 2198 (galleryentityhelper_medium::deleteentitybyid)
* in modules/core/ItemDelete.inc at line 86 (gallerycoreapi::deleteentitybyid)
* in main.php at line 174 (itemdeletecontroller::handlerequest)
* in main.php at line 87
* in main.php at line 80
System Information
Gallery version 2.0
PHP version 4.4.0 cgi
Webserver Apache/1.3.34 (Unix) mod_fastcgi/2.4.2 mod_auth_passthrough/1.8 mod_log_bytes/1.2 mod_bwlimited/1.4 FrontPage/5.0.2.2635 mod_ssl/2.8.25 OpenSSL/0.9.7a PHP-CGI/0.1b
Database mysql 4.0.25-standard
Toolkits SquareThumb, ImageMagick, NetPBM, Gd
Operating system Linux aphelion.webserversystems.com 2.6.9-11.ELsmp #1 SMP Wed Jun 8 17:54:20 CDT 2005 i686
Browser Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.7.8) Gecko/20050511 Firefox/1.0.4
Posts: 32509
yeah, since the other operation just failed probably with a apache timeout or a php fatal error, it probably still had a lock unreleased. normal g2 error handling releases all locks even when an error occurs, but we can't do that if it suddenly dies without proper shutdown.
i guess most of the issues will go away if we had a progressbar with periodic checkpoints for itemAdd.
Posts: 32509
i've added a task such that we don't forget.
http://sourceforge.net/pm/task.php?func=detailtask&project_task_id=121371&group_id=7130&group_project_id=14056
we are aware of the problem since a while, but i've never seen such symptoms.
Posts: 67
oops, sorry didn't see that.
From webpage and from local server they fail either with timeout or error page (but it still uploads a varying number of photos jsut with no success page).
From browser I uploaded a zip file with 50 images and it times out as in the other methods. If I choose the file individually from my cimputer then it works no problem.
Posts: 67
Gallery 2.1
PHP: 4.3.11
Apache: 1.3.34
MySQL: 4.0.25-standard
PHPinfo link
GD: yes
NetPBM: yes
Imagemagik: no
Browser: FF
Posts: 32509
i can only recommend using GalleryRemote or the other upload methods. maybe don't use the zipupload at all and try to upload only as many items as possible in your case...
sooner or later we'll add this progress bar etc and i hope that will fix these issues.
Posts: 67
Any word on this progress bar? I cannot add more than 40 files at a time. The problem has been with me over 3 server changes and with both G1 and all versions of G2.
Posts: 32509
no, no update on that.
Posts: 67
Its been quite a while now that this bug has been plaging Gallery. I know the software is free but do you guys think you could mabye get on this prob as its quite serious and hinders my ability to use your product.
Posts: 67
Any updates? I can only add 49 files at a time. Any chance I can give you access to my site to find out why its happening?
Posts: 1
not sure if this is the problem, but i just recently found that FireFox will resubmit a request if it gets no response after about 15 seconds. Haven't found any setting to change this. This is the only post I've found so far that mentioned possible resubmits using Firefox.
Posts: 67
I'm still having this problem now even with 2.2, in both IE 6 and 7 and FF. How come this isn't top on the priority list? Not being able to upload more than a few photos at a time is serious I would think.
Posts: 32509
DannyITR
workaround:
switch to database locking. site admin -> general: at the bottom, switch to database locking, save.
btw:
you could also ask your webhost to raise the ulimit for open filehandles.
--------------
Documentation: Support / Troubleshooting | Installation, Upgrade, Configuration and Usage
Posts: 32509
Update: We've just added a progress-bar and many improvements under the hood to make batch-adding items more reliable and user-friendly.
Upgrade to the next nightly snapshot of G2 including the itemadd module or wait for G2.3 (still months away) to get the improvements.
--------------
Documentation: Support / Troubleshooting | Installation, Upgrade, Configuration and Usage
Posts: 67
Thank you for addressing this issue. Pardon my ignorance but what is a nightly snapshot and how do I get this upgrade? I looked around the site and couldn't find anything.
Posts: 32509
please read more about upgrades and nightly snapshots at:
FAQ: How to upgrade Gallery2?
--------------
Documentation: Support / Troubleshooting | Installation, Upgrade, Configuration and Usage