Performance problem in G2
|
shivathegr8
Joined: 2006-09-12
Posts: 21 |
Posted: Tue, 2006-09-12 12:27
|
|
I have a high traffic portal website and I have upgraded from Gallery 1 to Gallery 2 for display all of our photos. The gallery 1 was working fine and we never had an issue, But we are having major performance problems after upgraded to Gallery 2 and it is unusable. I am using Apache 2.2.2 and PHP 5.4 with 500 apache concurrent connections. The server is turned well and I'm not sure what the problem. It is a dual core xeon processor high end option server with 4 gig of ram. Before upgrading to gallery 2 the apache was taking 300 connections and after upgrading it has been increased to 500.. I am not sure why it is taking extra apache connections. The gallery will work great for a while, maybe 30 minutes or so and is very fast, but after that, apache starts filling up and the load will shoot up to 500. My solution is to kill apache or restart it. I have tried everything I can think of but nothing fixes it. |
|
Posts: 32509
- there's no php 5.4. do you mean php 5.0.4? (a lot of bugs have been fixed since php 5.0.x)
- the extra connections might stem from, in average, longer connections, thus leading to a higher concurrency level
you should employ eaccelerator, it will help considerably.
so what happens after 30 minutes? does it start swapping like crazy because there's no more memory? where has the memory gone?
Posts: 21
Sorry.. I am using php-4.4.4 and now I installed eaccelerator.. Looks like, it is working fine..
Thanks
Posts: 21
I have eAccelerator installed with PHP and while uploading the image all of a sudden it will through an error and the gallery will not work..
-----------------------
Security Violation
The action you attempted is not permitted.
Back to the Gallery
Error Detail -
Error (ERROR_BAD_PARAMETER)
in modules/core/classes/helpers/GalleryChildEntityHelper_simple.class at line 219 (gallerycoreapi::error)
in modules/core/classes/helpers/GalleryChildEntityHelper_simple.class at line 117 (gallerychildentityhelper_simple::_fetchchilditemids)
in modules/core/classes/GalleryCoreApi.class at line 2477 (gallerychildentityhelper_simple::fetchchilditemids)
in modules/core/classes/GalleryTheme.class at line 875 (gallerycoreapi::fetchchilditemids)
in modules/core/classes/GalleryView.class at line 325 (matrixtheme::loadtemplate)
in main.php at line 386 (showitemview::doloadtemplate)
in main.php at line 87
in main.php at line 80
System Information
Gallery version 2.1.2
PHP version 4.4.4 apache2handler
Webserver Apache/2.2.3 (Unix) PHP/4.4.4
Database mysql 5.0.24a
Toolkits NetPBM, Gd
Operating system Linux localhost 2.6.11-1.1369_FC4smp #1 SMP Thu Jun 2 23:16:33 EDT 2005 x86_64
Browser Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; iOpus-I-M; .NET CLR 1.1.4322)
-----------------------
Posts: 32509
- what eaccelerator version are you using?
- without eaccelerator, there's no such error, right?
- also, clear your database cache in site admin -> maintenance
- note that after each gallery upgrade, you need to clear the eaccelerator cache. else you can get such errors.
- before line 219 in in modules/core/classes/helpers/GalleryChildEntityHelper_simple.class, could you please add var_dump($item); to find out what this item is?
also, in the database table g2_Entity, what entityType does the row with the id of this $item have?
Posts: 21
I am using eaccelerator-svn260
I am not getting any errors without eAccelerator.
I have cleared the database cache and I have done database optimization from site admin -> maintenance. After that I have disabled eAccelerator and the gallery is working fine.. I am continually monitoring, has it increasing httpd request or the server load.. After 3 hours also there is no changes.. I think the clearing the database cache and database optimization has helped a lot.
If I use eAccelerator, sometime random images and the thumbnails are not getting displayed.
Posts: 32509
i heard that the latest development version of eaccelerator is for php 5.2.0. maybe that's not true, just something another forum user said.
however, ensure that the eaccelerator version you're using is the latest stable version for your php version.
Posts: 21
The version of eaccelerator is not giving any error.. After optimizing the database, it worked fine for 4 hours and at the peek time the CPU and Memory usage shoot up drastically..
I have enabled the eaccelerator again with some changes.. I am caching only the php files and it is working fine from 2 days..
;;eAccelerator
zend_extension="/usr/local/lib/php/eaccelerator.so"
eaccelerator.shm_size="0"
eaccelerator.cache_dir="tmp/eaccelerator"
eaccelerator.enable="1"
eaccelerator.optimizer="1"
eaccelerator.check_mtime="1"
eaccelerator.debug="0"
eaccelerator.filter="*.php !*.jpg !*.gif !*.html"
eaccelerator.shm_max="0"
eaccelerator.shm_ttl="0"
eaccelerator.shm_prune_period="0"
eaccelerator.shm_only="0"
eaccelerator.compress="1"
eaccelerator.compress_level="9"
Posts: 7985
Your load shoots up to 500 because you're getting some form of blocking. Load average is a rough measurement of the number of concurrent processes, so most likely what's happening is that your database is getting table locks that are causing your various mysql queries to block. When that happens, what queries are running? And are you using MyISAM tables or InnoDB? Switch to InnoDB if you can, it's got a better locking model.
Posts: 22
Database Server:
Intel Xeon 3Ghz Dual CPU
2 GB RAM
73 GB RAID 5 Configured
Fedora Core 4 (32 Bit)
mysql-5.0.24a
Web Server
Intel Xeon 3Ghz Dual CPU
4 GB RAM
73 GB RAID 5 Configured
Fedora Core 4 (64 Bit)
Apache 2.2.2
PHP 4.4.4
eAccelerator-svn260
Tables are not getting locked. We are using MyISAM. Our web server and database servers are separate. Database connections are not crossing the 150 connection limit, but web server is reaching the maximum number of connections (500)
Most of the queries will be,
Query 1
SELECT g2_CacheMap.g_value FROM g2_CacheMap WHERE g2_C
Query 2
SELECT g2_CustomFieldMap.g_itemId, g2_CustomFieldMap.g_field, g2_CustomFieldMap.g_value FROM g2_Cus
Posts: 7985
What's the url to your Gallery?
Can you provide a full dump from http://example.com/server-status and a full dump of your mysql queries when you hit that point? What's your io load at that point? (ie, output from iostat). The last time I saw this happen when the Gallery site got slashdotted, we were hitting our bandwidth cap so even though Gallery (and on our site, Drupal) was processing requests just fine it couldn't get the bits out to the client so the requests were stalling. How's your bandwidth? When you have 500 apache processes, can you strace a sampling of them for 5-10 secs each and report that also?
Posts: 556
- There might be too many idle connections too..
(hitting the limit of 500)
You should consider raising that limit/lower idle timeout
- Generally I recommend using eaccelerator.compress_level="1" instead of "9"
Posts: 22
In apache/conf/extra/httpd-mpm.conf I find,
<IfModule mpm_netware_module>
ThreadStackSize 65536
StartThreads 250
MinSpareThreads 25
MaxSpareThreads 250
MaxThreads 1000
MaxRequestsPerChild 0
MaxMemFree 100
</IfModule>
Should I be changing these values?
Also we have modified the compress_level to 1 as per your recommendation.
regards,
Mahesh
Posts: 556
-You wrote the max connection limit is 500. Did u change it?
You could try something like: MaxClients 1024
-Changing timeout:
httpd.apache.org/docs/2.2/mod/core.html#keepalivetimeout
(KeepAlive On)
KeepAliveTimeout 5
-There might me a memory leakage..In that case the next line should help
MaxRequestsPerChild 2048
Other ideas:
-You could try switching to ImageMagick instead of NetPBM/Gd. IM uses less memory..
-Do you have all the thumbs/resizes (pre)generated? You can ensure this by running a maintenance task in site admin.
-Check that you are not being flooded. (e.g. look for lots of concurrent connections from same ip in apache log, sometimes ppl just start massgrabbing of websites/pictures..)
Let me know if any of these helped
Posts: 556
btw could we see your site?
Posts: 22
hi
We are converting to InnoDB. It will take few hours for sure. In the mean time the URLs are
http://entertainment.oneindia.in/gallery
http://thatstamil.oneindia.in/gallery
http://living.oneindia.in/gallery/main.php
http://thatsmalayalam.oneindia.in/gallery
http://thatskannada.oneindia.in/gallery
-- Mahesh
Posts: 22
What's the url to your Gallery?
Can you provide a full dump from http://example.com/server-status and a full dump of your mysql queries when you hit that point? What's your io load at that point? (ie, output from iostat). The last time I saw this happen when the Gallery site got slashdotted, we were hitting our bandwidth cap so even though Gallery (and on our site, Drupal) was processing requests just fine it couldn't get the bits out to the client so the requests were stalling. How's your bandwidth? When you have 500 apache processes, can you strace a sampling of them for 5-10 secs each and report that also?
---
Dear Bharat
I am converting the DB locking to Innodb now. Then I will respond to each of your queries on the forum. Prefer to do everything you all are asking and then respond
-- Mahesh
Posts: 7985
Performance seems pretty good on those sites the last couple of times I checked. Let me know if it's still bad for you.
Posts: 22
+ What's the url to your Gallery?
http://entertainment.oneindia.in/gallery2/main.php
http://entertainment.oneindia.in/wallpapers2/main.php
http://thatstamil.oneindia.in/gallery2/main.php
http://thatstelugu.oneindia.in/gallery2/main.php
http://thatsmalayalam.oneindia.in/gallery2/main.php
http://thatskannada.oneindia.in/gallery2/main.php
http://thatscricket.oneindia.in/gallery2/main.php
http://living.oneindia.in/gallery/main.php
http://news.oneindia.in/gallery2/main.php
http://www.holidaymakers.in/gallery2/main.php
+ Can you provide a full dump from http://example.com/server-status
Seems to be fine now.
+ and a full dump of your mysql queries when you hit that point?
It is not crossing more than 5 connections now.
------------
mysql> SHOW FULL PROCESSLIST;
+--------+---------+-----------+---------+---------+------+----------------------+------------------------------------------------------------------
| Id | User | Host | db | Command | Time | State | Info & amp; nbsp; & nbsp; |
+--------+---------+-----------+---------+---------+------+----------------------+-----------------------------------------------------------------
| 686173 | gallery | localhost | gallery | Sleep | 13 | | NULL & amp; nbsp; & nbsp; |
| 686195 | root | localhost | NULL | Query | 0 | NULL | SHOW FULL PROCESSLIST ; |
| 686204 | gallery | localhost | gallery | Query | 1 | Copying to tmp table | SELECT g2_entImageBlockCacheMap.g_entitemId FROM g2_entImageBlockCacheMap LEFT JOIN g2_entImageBlockDisabledMap ON g2_entImageBlockCacheMap.g_entitemId=g2_entImageBlockDisabledMap.g_entitemId WHERE g2_entImageBlockCacheMap.g_entuserId = 5 AND g2_entImageBlockCacheMap.g_entitemType = 1 AND g2_entImageBlockDisabledMap.g_entitemId IS NULL ORDER BY RAND() LIMIT 1 |
| 686205 | gallery | localhost | gallery | Query | 1 | Sending data | SELECT COUNT(*) FROM g2_entImageBlockCacheMap WHERE g2_entImageBlockCacheMap.g_entuserId = 5 & ;nbs p; |
+--------+---------+-----------+---------+---------+------+----------------------+-------------------------------------------------------------------
+ What's your io load at that point? (ie, output from iostat). The last time I saw this
+ happen when the Gallery site got slashdotted, we were hitting our
+ bandwidth cap so even though Gallery (and on our site, Drupal) was
+ processing requests just fine it couldn't get the bits out to the client
+ so the requests were stalling. How's your bandwidth? When you have 500
+ apache processes, can you strace a sampling of them for 5-10 secs each and
+ report that also?
Now it is normal and not crossing 250 connections. So these are the things we did to solve the problem,
- Moved to InnoDB locking (after moving we felt the number of MySQL connections increased drastically but now it has come back to normal. Not sure if this is normal behavior)
- Using eaccelerator.compress_level="1" instead of "9"
I have one more question: Why are the images being stored in the database "g2_CacheMap" as well as gallery data folder? Cant we store images only in the gallery data folder? Because of this the database becomes very huge.
We appreciate all of you for helping us.
regards,
Mahesh
Posts: 32509
Images are not stored in the database.
Images are stored in your storage folder (g2data/). albums/ for the originals, cache/derivative/ for thumbnails and resizes. they are generated once and stored there forever.
the db table g2_CacheMap is for compressed HTML data. each generated page is stored in CacheMap. if the page cache hasn't expired yet, all requests are served from the cached page instead of generating all HTML again.
g2_CacheMap in G2.1 grows very large. we hope to prevent it doing from that with the current development version of G2. please wait for Gallery 2.2 for a cleaner solution.
Posts: 22
Hi
After making these changes the counter for each of the photos is not getting incremented
Also, approx by when will G2.2 be released?
-- Mahesh
Posts: 556
-it's getting incremented, but you don't see it immediately, only after the cached pages expire. it's a feature not a bug ;>
-rc1 in some weeks