Question : Web server is very slow (MySQL+Apache+PHP) - need help with upgrade/tuning

Our server is getting to slow. Especially MySQL service ( i'm getting to many alerts about lost connections from PHP scripts).
Trying to tune it, but can't get any visible performance improvement. This is my first server so i really not sure if this is becoming hardware problem or not.

Here are the details:

Linux 2.6.9-42.0.2.EL AMD Opteron(tm) Processor 2GHz

meminfo:

MemTotal:      2075976 kB    MemFree:        371676 kB
Buffers:        169380 kB     Cached:         551564 kB
SwapCached:      10632 kB    Active:        1224108 kB
Inactive:       271584 kB    HighTotal:     1179584 kB
HighFree:        68928 kB    LowTotal:       896392 kB
LowFree:        302748 kB    SwapTotal:     1052248 kB
SwapFree:      1019880 kB    Dirty:             944 kB
Writeback:           0 kB    Mapped:         815416 kB
Slab:           190356 kB    CommitLimit:   2090236 kB
Committed_AS:  1800012 kB   PageTables:       5972 kB
VmallocTotal:   106488 kB   VmallocUsed:      2112 kB
VmallocChunk:   104136 kB    HugePages_Total:     0
HugePages_Free:      0    Hugepagesize:     4096 kB


MySQL info:


Uptime = 0 days 14 hrs 41 min 21 sec
Avg. qps = 237
Total Questions = 12535971
Threads Connected = 43

SLOW QUERIES
Current long_query_time = 5 sec.
You have 774 out of 12516157 that take longer than 5 sec. to complete

MEMORY USAGE
Max Memory Ever Allocated : 918 M
Configured Max Per-thread Buffers : 1 G
Configured Max Global Buffers : 250 M
Configured Max Memory Limit : 2 G
Total System Memory : 2.98 G

KEY BUFFER
Current MyISAM index space = 440 M
Current key_buffer_size = 192 M
Key cache miss rate is 1 : 3375
Key buffer fill ratio = 100.00 %

QUERY CACHE
Query cache is enabled
Current query_cache_size = 48 M
Current query_cache_used = 21 M
Current query_cach_limit = 1 M
Current Query cache fill ratio = 44.78 %

JOINS
Current join_buffer_size = 4.00 M
You have had 3084 queries where a join could not use an index properly

TEMP TABLES
Current max_heap_table_size = 39 M
Current tmp_table_size = 40 M
Of 17442 temp tables, 13% were created on disk

TABLE LOCKING
Current Lock Wait ratio = 1 : 927


In the slow query log there are requests to the same few tables, one of them is temporary (MEMORY), all other are not to too big tables (~200K rows, ~15Mb), but uses inner join several time in the same query + select distinct.

The application, that i think does the main load - makes many small queries per request (up to 200).

We have ~6000 unique visitors per day

So the question is: should i try to squeeze something from this hardware by tuning MySQL server (or queries, table optimization) or this is really looks like maximum this machine can give me?

Code Snippet:

1: 2: 3: 4: 5: 6: 7: 8: 9: 10: 11: 12: 13: 14: 15: 16: 17: 18: 19: 20: 21: 22: 23: 24: 25: 26: 27: 28: 29: 30: 31: 32: 33: 34: 35: 36: 37: 38: 39: 40: 41: 42:

TOP: top - 10:20:29 up 322 days, 5:36, 1 user, load average: 1.21, 1.09, 1.11 Tasks: 169 total, 1 running, 168 sleeping, 0 stopped, 0 zombie Cpu(s): 48.9% us, 8.0% sy, 0.0% ni, 42.7% id, 0.4% wa, 0.0% hi, 0.0% si Mem: 2075976k total, 1716372k used, 359604k free, 169764k buffers Swap: 1052248k total, 32368k used, 1019880k free, 553708k cached PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 31790 mysql 15 0 515m 410m 4592 S 27.0 20.3 118:17.62 mysqld 3193 apache 17 0 57840 35m 6960 S 8.6 1.7 1:44.86 httpd 7616 apache 15 0 54684 31m 6240 S 6.2 1.6 0:46.57 httpd 2873 apache 15 0 54260 31m 6776 S 1.6 1.6 1:46.00 httpd 21979 apache 15 0 54740 31m 5800 S 1.6 1.5 0:05.49 httpd 3208 apache 15 0 54880 32m 6824 S 1.0 1.6 1:40.54 httpd 3224 apache 15 0 56240 33m 6876 S 1.0 1.7 1:41.84 httpd 14194 apache 15 0 54616 31m 6332 S 1.0 1.6 0:22.04 httpd 14201 apache 16 0 54440 31m 5908 S 1.0 1.5 0:20.64 httpd 3292 apache 15 0 56104 33m 6864 S 0.8 1.7 1:46.01 httpd 3509 apache 15 0 54780 32m 6932 S 0.8 1.6 1:44.39 httpd 7518 apache 15 0 54076 31m 6652 S 0.8 1.5 0:45.61 httpd 12968 apache 15 0 55220 32m 6240 S 0.8 1.6 0:29.82 httpd 21962 apache 15 0 54164 30m 5984 S 0.8 1.5 0:06.47 httpd 20093 apache 15 0 55496 32m 6600 S 0.4 1.6 0:09.63 httpd 24839 root 16 0 1988 1044 768 R 0.2 0.1 0:00.03 top 1 root 15 0 2912 508 432 S 0.0 0.0 1:16.12 init 2 root 34 19 0 0 0 S 0.0 0.0 0:58.75 ksoftirqd/0 3 root 5 -10 0 0 0 S 0.0 0.0 0:15.68 events/0 4 root 5 -10 0 0 0 S 0.0 0.0 0:29.94 khelper 5 root 15 -10 0 0 0 S 0.0 0.0 0:00.00 kacpid 29 root 5 -10 0 0 0 S 0.0 0.0 0:00.00 kblockd/0 50 root 13 -10 0 0 0 S 0.0 0.0 0:00.00 aio/0 30 root 15 0 0 0 0 S 0.0 0.0 0:00.00 khubd 49 root 16 0 0 0 0 S 0.0 0.0 29:40.61 kswapd0 194 root 25 0 0 0 0 S 0.0 0.0 0:00.00 kseriod 296 root 22 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_0 309 root 15 0 0 0 0 S 0.0 0.0 94:39.60 kjournald 1603 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kauditd 1644 root 6 -10 0 0 0 S 0.0 0.0 0:00.00 kmirrord 1668 root 15 0 0 0 0 S 0.0 0.0 0:00.02 kjournald 1669 root 15 0 0 0 0 S 0.0 0.0 2:17.73 kjournald 2495 rpc 16 0 2528 628 596 S 0.0 0.0 0:00.00 portma

Open in New Window Select All

Answer : Web server is very slow (MySQL+Apache+PHP) - need help with upgrade/tuning

  I have a few suggestions.

  The first is, if you're having load problems split the database and web server.   For a development environment, it's fine to have them both on the same machine, but once you go into production with it, the database really should have it's own machine.  Otherwise, yes, you'll run into problems at high load times. When you split MySQL off to it's own server, stop every unnecessary service, so the database is the only thing that will take up any time.  No web, ftp, nfs, etc, etc.  Leave SSH so you can log in. :)

  The second is, you may be short on memory.   You have MySQL configured to use up to 2Gb of ram, and that's all the machine has.  This is (almost) ok if it's your dedicated database server, but not sharing it with the web server.  

  The third is, pay careful attention to your slow query log, and listen to what it's telling you.  I won't say that 5 seconds is the worst I've seen (I've seen over 1000 seconds), but it's still not what you really wanted.  Improved indexing does wonders here.  

  I'm curious to why you're getting the delayed inserts locking the database.  

  I had to do a large import (> 9 million records) for a customer on a shared database machine.  It's 2.4Ghz with 1.5Gb RAM, so close to your specs.  The inserts hammered on it, and slowed down the response on the console, but requests from the web server (a separate machine) still came through perfectly.  There are a few very touchy customers who complain if a page takes 1 full second to load, and we had no problems through the entire insert cycle.

  Once you've improved the database situation, you may still run into problems with Apache preforking.  In a high load environment, you may need to prefork quite a few more.  Watch your server-status page to ensure there are plenty of waiting children, so there's always someone to answer.   If your load is high enough, put up multiple servers with DNS round robin, so you don't run into contention issues with the web side.  

  With that server, you should be able to satisfy quite a few web servers, and an awful lot of traffic.  Taking care of the above points should do you very well.  Beyond that, you're looking at more of enterprise design issues with replicated databases to handle the load, but I don't think you're anywhere near that.