Declining performance and database corruption

I have been having serious problems working with and viewing my main Gallery 2 database. It sometimes takes 5 seconds to bring up the database front page, and other times it takes more than 90 seconds.

What happens in the worst case is that a file directory is moved,
whether that directory contains 5 files or 500 files, and the database
isn't updated with the new location. If I move the directory back
manually, the database almost always catches up and completes the move.
Other times, the database won't even let me start the move process, as
the screen that shows the album tree is replaced by a blank white
screen. If I can get past this point, I may still get a blank white
screen but the files and the database album are moved almost immediately.

On a couple of occasions, I saw albums with around 500 files moved
almost instantly, but not a minute later I couldn't access the database
again at all. Other move or delete operations with 5 or 10 files would
fail miserably and repeatedly. Even changing the text descriptions of
albums was failing. When I do the same file operations over FTP or the ISP's online File Manager, even at the same time that the database is claiming a read/write failure, the change is instantaneous. I don't understand how everything can be so fast and reliable one day for a few hours and then seconds later it feels like the server is running on a 14.4 kbps modem. Wait half an hour, and operations start zooming really fast, and then screech to a halt again inexplicably.

Two days ago, there were some slowness problems, but not like today. I've been effectively locked out of my Gallery database for more than 36 hours, and I'm hitting Error 503:

Service Temporarily Unavailable
The server is temporarily unable to service your request due to maintenance downtime or capacity problems. Please try again later.

The ISP says that it's working fine, that the size of the database may be causing some slowness, but that if I want more performance I need to upgrade from a shared server to a dedicated server. Does it really take that many resources to run Gallery 2?

I've made a listing of the errors that I've been getting on this page:

http://www.alexanderrogge.net/errorsgallery.html

ERROR_STORAGE_FAILURE indicates an incident where there was no response
from the server during a file or directory move, or a path rename operation.

ERROR_LOCK_TIMEOUT is indicates an incident where a file or directory was being moved, and had been moved successfully, but still resulted in this error message.

ERROR_BAD_PATH indicates an attempt to move an album directory that had
already been moved, but the database had not been updated with the new
path after more than five minutes.

I tried changing from File Locking to Database Locking, but that made the problem worse. What can I do to fix this slowness problem?

System Information
Gallery version 2.3
PHP version 5.2.8 cgi-fcgi
Webserver Apache
Database mysql 5.0.91-log
Operating system Linux p3nlh068.shr.prod.phx3.secureserver.net 2.6.18-164.15.1.el5PAE #1 SMP Wed Mar 17 12:14:29 EDT 2010 i686

> Copy'n'paste your G2 system information from "Site Admin -> Maintenance -> System information"

I can't get in. The database is totally unresponsive right now. All I'm getting is a blank white screen.

I tried to blame GoDaddy, but I haven't found many other ISPs that will support a site this large for shared hosting, and I want to make sure that the problem isn't something like a configuration error on my part before I go and jump to another ISP. It does work fine for a few minutes sometimes, but overall the performance has been getting worse to the point that it is now.

According to the ISP, my Gallery 2 database was compressed to 80 MB from 110 MB. It's not a high-traffic site at all. Oddly enough, when the traffic does increase, my site becomes more responsive. When nobody is using it, it starts doing this dead response nonsense.

Try putting your G2 install into debug mode and see if you can get debug info:
FAQ: How to set/use Gallery in debug mode?

If you can either copy it into a text file and attach to this thread, or post that it's in debug mode and post with a link to your G2 install and I'll take a look.

Comments, commentary and suggestions follow:

That's not a big DB at all (the DB server shouldn't be choking, yet it is). Well actually for what G2 stores, it tells me you either have a TON of images/albums or a TON of users or a TON of comments or something like that. So for G2, that could be a pretty good sized install. Still the DB server shouldn't be choking but GoDaddy WAY oversells their servers. Your are sharing your webserver (hopefully the DB is on a different server) with 3,200 sites:
http://whois.domaintools.com/alexanderrogge.net
(note some of that may be old info and it's not really 3,200, could be less, could be way more, I've seen that number hit over 10,000 for GoDaddy hosted sites that were amazingly sluggish...)

But what I meant by size of your G2 install, was how many images & albums, how many users, etc. Do you have lots of special, unique permissions setup, are you using plugins like EXIF, GD, etc. Are you utilizing Gallery's caching mechanism?

I can only suggest 2 things (well 2-1/2)

Run from GoDaddy, fast and far
- the 1/2 note, don't use 1&1 either they are just as bad, IMO

Here's the other thing I can suggest:
If you stick with this low-quality host, if/when you ever get access to your site, follow these guides:
http://codex.gallery2.org/Gallery2:Performance_Tips
http://codex.gallery2.org/Gallery2:ACL_Performance

Quick summary (but do read those guides)
- Use ImageMagick, don't even have GD installed
- Enable caching (Site Admin > Performance)
- Uninstall as many plugins as you can
- DO NOT use EXIF plugin for a big site
- DO NOT have lots of unique permissions setup for users (user albums, plugin probably wouldn't be good on a big site)

____________________________________________
Like Gallery? Like the support? Donate now!!! See G2 live here

The gallery is somewhat large, around 40 GB of image data across 80,000 image files. No, I don't think that this is the problem. I have another Gallery database that I admin on the same server, but which doesn't belong to me, and it's only around 3,000 image files. It's having the same problems. In fact, it was stalling like my main database is now, but that happened weeks ago. Image thumbnails previously-generated were coming up one every 10 seconds or so. It was ridiculous, and I couldn't figure it out. My WordPress blog has also been having response problems, and it's text-only and very small. GoDaddy blames WordPress, but it's a bunch of small text entries! It can't possibly be using too many resources, can it?

There are no other users in my Gallery. I am using Gd because it seemed to be giving me the best quality when I started. I am also using the EXIF plugin. Other than password-protected albums, I don't have any special user permissions. I am using the caching, but that doesn't seem to make any difference. Even the Site Admin screens aren't working now.

I think I'm past a performance complaint at this point. I've been trying to look at a bunch of albums and all I'm getting are blank white screens. The ISP says that the only thing I can do is optimize the database, but that doesn't make sense. If the little Gallery database and my WordPress database are having the same problems, compressing this database won't help.

So, run to the alternative ISP that I found? I'm at wits-end!

When it's not a blank white screen, it's been saying this:

Error
Error (ERROR_STORAGE_FAILURE)

* in modules/core/classes/GalleryStorage.class at line 641 (GalleryCoreApi::error)
* in modules/core/classes/GalleryCoreApi.class at line 2912 (GalleryStorage::getMapEntry)
* in modules/core/classes/helpers/GalleryUserHelper_simple.class at line 91 (GalleryCoreApi::getMapEntry)
* in modules/core/classes/GalleryCoreApi.class at line 566 (GalleryUserHelper_simple::hasItemPermission)
* in modules/core/classes/GalleryView.class at line 362 (GalleryCoreApi::hasItemPermission)
* in modules/core/ShowItem.inc at line 106 (GalleryView::getItem)
* in modules/core/classes/GalleryView.class at line 396 (ShowItemView::getItem)
* in modules/core/classes/GalleryView.class at line 260 (GalleryView::loadThemeAndParameters)
* in main.php at line 465 (GalleryView::doLoadTemplate)
* in main.php at line 104
* in main.php at line 88

I'll look into those plugin issues. I did try ImageMagick once and it didn't seem to work for me. It kept wanting to apply JPEG compression, for one thing, and I didn't want that. I didn't know about the thumbnail failures, which could explain why it looks like they're being rebuilt sometimes.

What happens if I just do an FTP transfer to a new host? Should I be expecting compatibility problems? The good news is that this image directory structure is stored locally, which should mean that it won't have to be reconstructed one album at a time, which is what I went through when the PBase.com database failed, and it's is still having problems.

I deactivated the EXIF plugin and now I'm getting Error 500 Internal Server Error. It's never been this bad before.

Don't they see this problem going on over at the ISP? What I don't understand, and why I'm asking if there's something I've missed, is that if the service is this bad, what about those thousands of other customers who are sharing the server? I have one of the least busiest sites on the Web, as I redirect most traffic to another server. This database is mainly a personal gallery that only a few, and I mean fewer than 100, people use per month. I should barely be using any processing resources at all, and I can't even get in right now. What about the people who are running complex sites all day long, with thousands of visitors per hour? Why aren't they having the same problems and leaving in droves? I wondered if I was being given fewer resources because I'm not demanding those resources all day long. Is it possible that they're disconnecting my site somehow and giving preference to the more-demanding customers?

I can't continue like this, and to be told that there's nothing wrong with the servers by the ISP. There is definitely something wrong on the server end! Why do they keep blaming my browser and network connection? I don't know what else to do except move.

[irony]Maybe there's another G2 user on the same server who's just been uploading 10,000 large jpgs and creating thumbnails with GD....[/irony]

I wasn't referring to you, I was making a joke that some *other* user of G2 on the same server was being a resource hog.

GD is incredibly resource intensive and inefficient in both memory and processor time as it runs as a compiled-in part of the PHP environment, compared to either ImageMagick or NetPBM which both run as native binaries on the server.

As suggested, I deactivated some plugins that I didn't need and switched from Gd to ImageMagick. Although the site did seem to be working faster, it wasn't long before I started hitting ERROR_STORAGE_FAILURE again. Clients would try to click on a thumbnail to download the image and would get nothing but a delay and a message at the bottom of the screen saying "There was an error interacting with the database." I have hit Error 503 again, with no domain access at all. I tried to use the ISP's interface to back up my database, but got a message back saying that the backup had failed and that I needed to contact customer support. This isn't about a big database. There's something much more serious going on with these servers. I put in my final complaint, and then I'm going to move the site somewhere else, as I have no choice.

What I don't understand is if my quiet site is having so many problems, what are all those thousands of other customers doing with their sites, which are probably demanding many more resources? Why aren't thousands of customers leaving every week, unable to keep their sites in business? If it's this bad, I'd think that I should be one of the only customers left, someone who doesn't rely on this site to be working under high-capacity demands every day.

I was hoping that I could get to Gallery 3 before having to migrate somewhere else. I sure wish that I'd known about "No, Daddy!" before I signed up. I actually didn't sign up with them. I paid a local telco, which apparently subcontracted with "No, Daddy!" I was not aware of this fact until I tried to get support with my first Error 500, and noticed that the phone number was based in Arizona. Sneaky, but why can't they keep their service in good condition? I'm even paying three times the amount per month than the hosting provider that I'm looking at moving to, so it's not like I was trying to be cheap to excess. I paid good money and I expect good service, and I even told them that if there was a problem with using "unlimited" storage to excess, ask for a different price now before I invest. I did offer more money, but I wanted unlimited storage because I didn't want to have to worry about quotas. This change was much cheaper than the hosting provider I had before, which is still having problems. Running a Website shouldn't be this frustrating!

The "No, Daddy!" response to my final complaint is to mention the knowledge base answer to the keyword "Error 503": "You probably exceeded the concurrent connection limit for this server." Unless my site is being DoSed, that's impossible. I have a bandwidth requirement that would be barely noticeable in 1995. It's meant to be quiet, and be accessible through a 28.8 kbps modem or a non-broadband phone. If a corporate hosting provider can't keep up with that sort of bandwidth requirement, something is really wrong.

How much server power does it take to run Gallery 2, anyway? Assuming that there are an arbitrary 100,000 images in 1,000 albums, and the total size of all images is 100 GB, what kind of computer would I need to serve this image gallery?

I was thinking of for one user, and scale the requirements from there. How would these requirements compare to running an HTML-based gallery with no database?

I have 87,000 images in probably 200 albums. Each image is 600x400 only. It runs on a dedicated single-core 2GHz rack mount server (cheapest dedicated server I could rent, costs roughly $60 per month) with 160Gb of storage (about 30% full) and 1Gb of RAM. It runs to a standstill when tens or hundreds of people are trying to look at and purchase photos at the same time but it doesn't get hit with that load very often.

Maybe that information helps. As far as how much slower than a purely static gallery, my guess would be G2 is tens or hundreds of times slower.