Elevati carichi di Linux con basso utilizzo di CPU / memoria

12

Ho una situazione piuttosto strana, in cui i carichi del mio box CentOS 5.5 sono alti, ma la CPU e la memoria utilizzate sono piuttosto basse:

top - 20:41:38 up 42 days,  6:14,  2 users,  load average: 19.79, 21.25, 18.87
Tasks: 254 total,   1 running, 253 sleeping,   0 stopped,   0 zombie
Cpu(s):  3.8%us,  0.3%sy,  0.1%ni, 95.0%id,  0.6%wa,  0.0%hi,  0.1%si,  0.0%st
Mem:   4035284k total,  4008084k used,    27200k free,    38748k buffers
Swap:  4208928k total,   242576k used,  3966352k free,  1465008k cached

free -mt
             total       used       free     shared    buffers     cached
Mem:          3940       3910         29          0         37       1427
-/+ buffers/cache:       2445       1495
Swap:         4110        236       3873
Total:        8050       4147       3903

Iostat mostra anche buoni risultati:

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
           3.83    0.13    0.41    0.58    0.00   95.05

Ecco l'uscita ps aux:

USER       PID %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
root         1  0.0  0.0  10348    80 ?        Ss    2010   2:11 init [3]                                           
root         2  0.0  0.0      0     0 ?        S<    2010   0:00 [migration/0]
root         3  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/0]
root         4  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/0]
root         5  0.0  0.0      0     0 ?        S<    2010   0:02 [migration/1]
root         6  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/1]
root         7  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/1]
root         8  0.0  0.0      0     0 ?        S<    2010   0:02 [migration/2]
root         9  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/2]
root        10  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/2]
root        11  0.0  0.0      0     0 ?        S<    2010   0:02 [migration/3]
root        12  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/3]
root        13  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/3]
root        14  0.0  0.0      0     0 ?        S<    2010   0:03 [migration/4]
root        15  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/4]
root        16  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/4]
root        17  0.0  0.0      0     0 ?        S<    2010   0:01 [migration/5]
root        18  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/5]
root        19  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/5]
root        20  0.0  0.0      0     0 ?        S<    2010   0:11 [migration/6]
root        21  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/6]
root        22  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/6]
root        23  0.0  0.0      0     0 ?        S<    2010   0:01 [migration/7]
root        24  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/7]
root        25  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/7]
root        26  0.0  0.0      0     0 ?        S<    2010   0:00 [migration/8]
root        27  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/8]
root        28  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/8]
root        29  0.0  0.0      0     0 ?        S<    2010   0:00 [migration/9]
root        30  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/9]
root        31  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/9]
root        32  0.0  0.0      0     0 ?        S<    2010   0:08 [migration/10]
root        33  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/10]
root        34  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/10]
root        35  0.0  0.0      0     0 ?        S<    2010   0:05 [migration/11]
root        36  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/11]
root        37  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/11]
root        38  0.0  0.0      0     0 ?        S<    2010   0:02 [migration/12]
root        39  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/12]
root        40  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/12]
root        41  0.0  0.0      0     0 ?        S<    2010   0:14 [migration/13]
root        42  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/13]
root        43  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/13]
root        44  0.0  0.0      0     0 ?        S<    2010   0:04 [migration/14]
root        45  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/14]
root        46  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/14]
root        47  0.0  0.0      0     0 ?        S<    2010   0:01 [migration/15]
root        48  0.0  0.0      0     0 ?        SN    2010   0:00 [ksoftirqd/15]
root        49  0.0  0.0      0     0 ?        S<    2010   0:00 [watchdog/15]
root        50  0.0  0.0      0     0 ?        S<    2010   0:00 [events/0]
root        51  0.0  0.0      0     0 ?        S<    2010   0:00 [events/1]
root        52  0.0  0.0      0     0 ?        S<    2010   0:00 [events/2]
root        53  0.0  0.0      0     0 ?        S<    2010   0:00 [events/3]
root        54  0.0  0.0      0     0 ?        S<    2010   0:00 [events/4]
root        55  0.0  0.0      0     0 ?        S<    2010   0:00 [events/5]
root        56  0.0  0.0      0     0 ?        S<    2010   0:00 [events/6]
root        57  0.0  0.0      0     0 ?        S<    2010   0:00 [events/7]
root        58  0.0  0.0      0     0 ?        S<    2010   0:00 [events/8]
root        59  0.0  0.0      0     0 ?        S<    2010   0:00 [events/9]
root        60  0.0  0.0      0     0 ?        S<    2010   0:00 [events/10]
root        61  0.0  0.0      0     0 ?        S<    2010   0:00 [events/11]
root        62  0.0  0.0      0     0 ?        S<    2010   0:00 [events/12]
root        63  0.0  0.0      0     0 ?        S<    2010   0:00 [events/13]
root        64  0.0  0.0      0     0 ?        S<    2010   0:00 [events/14]
root        65  0.0  0.0      0     0 ?        S<    2010   0:00 [events/15]
root        66  0.0  0.0      0     0 ?        S<    2010   0:00 [khelper]
root       107  0.0  0.0      0     0 ?        S<    2010   0:00 [kthread]
root       126  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/0]
root       127  0.0  0.0      0     0 ?        S<    2010   0:03 [kblockd/1]
root       128  0.0  0.0      0     0 ?        S<    2010   0:01 [kblockd/2]
root       129  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/3]
root       130  0.0  0.0      0     0 ?        S<    2010   0:05 [kblockd/4]
root       131  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/5]
root       132  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/6]
root       133  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/7]
root       134  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/8]
root       135  0.0  0.0      0     0 ?        S<    2010   0:02 [kblockd/9]
root       136  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/10]
root       137  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/11]
root       138  0.0  0.0      0     0 ?        S<    2010   0:04 [kblockd/12]
root       139  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/13]
root       140  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/14]
root       141  0.0  0.0      0     0 ?        S<    2010   0:00 [kblockd/15]
root       142  0.0  0.0      0     0 ?        S<    2010   0:00 [kacpid]
root       281  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/0]
root       282  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/1]
root       283  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/2]
root       284  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/3]
root       285  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/4]
root       286  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/5]
root       287  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/6]
root       288  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/7]
root       289  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/8]
root       290  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/9]
root       291  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/10]
root       292  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/11]
root       293  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/12]
root       294  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/13]
root       295  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/14]
root       296  0.0  0.0      0     0 ?        S<    2010   0:00 [cqueue/15]
root       299  0.0  0.0      0     0 ?        S<    2010   0:00 [khubd]
root       301  0.0  0.0      0     0 ?        S<    2010   0:00 [kseriod]
root       490  0.0  0.0      0     0 ?        S     2010   0:00 [khungtaskd]
root       493  0.1  0.0      0     0 ?        S<    2010  94:48 [kswapd1]
root       494  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/0]
root       495  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/1]
root       496  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/2]
root       497  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/3]
root       498  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/4]
root       499  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/5]
root       500  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/6]
root       501  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/7]
root       502  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/8]
root       503  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/9]
root       504  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/10]
root       505  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/11]
root       506  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/12]
root       507  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/13]
root       508  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/14]
root       509  0.0  0.0      0     0 ?        S<    2010   0:00 [aio/15]
root       665  0.0  0.0      0     0 ?        S<    2010   0:00 [kpsmoused]
root       808  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/0]
root       809  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/1]
root       810  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/2]
root       811  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/3]
root       812  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/4]
root       813  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/5]
root       814  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/6]
root       815  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/7]
root       816  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/8]
root       817  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/9]
root       818  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/10]
root       819  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/11]
root       820  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/12]
root       821  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/13]
root       822  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/14]
root       823  0.0  0.0      0     0 ?        S<    2010   0:00 [ata/15]
root       824  0.0  0.0      0     0 ?        S<    2010   0:00 [ata_aux]
root       842  0.0  0.0      0     0 ?        S<    2010   0:00 [scsi_eh_0]
root       843  0.0  0.0      0     0 ?        S<    2010   0:00 [scsi_eh_1]
root       844  0.0  0.0      0     0 ?        S<    2010   0:00 [scsi_eh_2]
root       845  0.0  0.0      0     0 ?        S<    2010   0:00 [scsi_eh_3]
root       846  0.0  0.0      0     0 ?        S<    2010   0:00 [scsi_eh_4]
root       847  0.0  0.0      0     0 ?        S<    2010   0:00 [scsi_eh_5]
root       882  0.0  0.0      0     0 ?        S<    2010   0:00 [kstriped]
root       951  0.0  0.0      0     0 ?        S<    2010   4:24 [kjournald]
root       976  0.0  0.0      0     0 ?        S<    2010   0:00 [kauditd]
postfix    990  0.0  0.0  54208  2284 ?        S    21:19   0:00 pickup -l -t fifo -u
root      1013  0.0  0.0  12676     8 ?        S<s   2010   0:00 /sbin/udevd -d
root      1326  0.0  0.0  90900  3400 ?        Ss   14:53   0:00 sshd: root@notty 
root      1410  0.0  0.0  53972  2108 ?        Ss   14:53   0:00 /usr/libexec/openssh/sftp-server
root      2690  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/0]
root      2691  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/1]
root      2692  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/2]
root      2693  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/3]
root      2694  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/4]
root      2695  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/5]
root      2696  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/6]
root      2697  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/7]
root      2698  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/8]
root      2699  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/9]
root      2700  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/10]
root      2701  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/11]
root      2702  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/12]
root      2703  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/13]
root      2704  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/14]
root      2705  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpathd/15]
root      2706  0.0  0.0      0     0 ?        S<    2010   0:00 [kmpath_handlerd]
root      2755  0.0  0.0      0     0 ?        S<    2010   4:35 [kjournald]
root      2757  0.0  0.0      0     0 ?        S<    2010   3:38 [kjournald]
root      2759  0.0  0.0      0     0 ?        S<    2010   4:10 [kjournald]
root      2761  0.0  0.0      0     0 ?        S<    2010   4:26 [kjournald]
root      2763  0.0  0.0      0     0 ?        S<    2010   3:15 [kjournald]
root      2765  0.0  0.0      0     0 ?        S<    2010   3:04 [kjournald]
root      2767  0.0  0.0      0     0 ?        S<    2010   3:02 [kjournald]
root      2769  0.0  0.0      0     0 ?        S<    2010   2:58 [kjournald]
root      2771  0.0  0.0      0     0 ?        S<    2010   0:00 [kjournald]
root      3340  0.0  0.0   5908   356 ?        Ss    2010   2:48 syslogd -m 0
root      3343  0.0  0.0   3804   212 ?        Ss    2010   0:03 klogd -x
root      3430  0.0  0.0      0     0 ?        S<    2010   0:50 [kondemand/0]
root      3431  0.0  0.0      0     0 ?        S<    2010   0:54 [kondemand/1]
root      3432  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/2]
root      3433  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/3]
root      3434  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/4]
root      3435  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/5]
root      3436  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/6]
root      3437  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/7]
root      3438  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/8]
root      3439  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/9]
root      3440  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/10]
root      3441  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/11]
root      3442  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/12]
root      3443  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/13]
root      3444  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/14]
root      3445  0.0  0.0      0     0 ?        S<    2010   0:00 [kondemand/15]
root      3461  0.0  0.0  10760   284 ?        Ss    2010   3:44 irqbalance
rpc       3481  0.0  0.0   8052     4 ?        Ss    2010   0:00 portmap
root      3526  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/0]
root      3527  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/1]
root      3528  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/2]
root      3529  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/3]
root      3530  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/4]
root      3531  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/5]
root      3532  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/6]
root      3533  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/7]
root      3534  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/8]
root      3535  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/9]
root      3536  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/10]
root      3537  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/11]
root      3538  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/12]
root      3539  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/13]
root      3540  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/14]
root      3541  0.0  0.0      0     0 ?        S<    2010   0:00 [rpciod/15]
root      3563  0.0  0.0  10160     8 ?        Ss    2010   0:00 rpc.statd
root      3595  0.0  0.0  55180     4 ?        Ss    2010   0:00 rpc.idmapd
dbus      3618  0.0  0.0  21256    28 ?        Ss    2010   0:00 dbus-daemon --system
root      3649  0.2  0.4 563084 18796 ?        S<sl  2010 179:03 mfsmount /mnt/mfs -o rw,mfsmaster=web1.ovs.local
root      3702  0.0  0.0   3800     8 ?        Ss    2010   0:00 /usr/sbin/acpid
68        3715  0.0  0.0  31312   816 ?        Ss    2010   3:14 hald
root      3716  0.0  0.0  21692    28 ?        S     2010   0:00 hald-runner
68        3726  0.0  0.0  12324     8 ?        S     2010   0:00 hald-addon-acpi: listening on acpid socket /var/run/acpid.socket
68        3730  0.0  0.0  12324     8 ?        S     2010   0:00 hald-addon-keyboard: listening on /dev/input/event0
root      3773  0.0  0.0  62608   332 ?        Ss    2010   0:00 /usr/sbin/sshd
ganglia   3786  0.0  0.0  24704   988 ?        Ss    2010  14:26 /usr/sbin/gmond
root      3843  0.0  0.0  54144   300 ?        Ss    2010   1:49 /usr/libexec/postfix/master
postfix   3855  0.0  0.0  54860  1060 ?        S     2010   0:22 qmgr -l -t fifo -u
root      3877  0.0  0.0  74828   708 ?        Ss    2010   1:15 crond
root      3891  1.4  1.9 326960 77704 ?        S<l   2010 896:59 mfschunkserver
root      4122  0.0  0.0  18732   176 ?        Ss    2010   0:10 /usr/sbin/atd
root      4193  0.0  0.8 129180 35984 ?        Ssl   2010  11:04 /usr/bin/ruby /usr/sbin/puppetd
root      4223  0.0  0.0  18416   172 ?        S     2010   0:10 /usr/sbin/smartd -q never
root      4227  0.0  0.0   3792     8 tty1     Ss+   2010   0:00 /sbin/mingetty tty1
root      4230  0.0  0.0   3792     8 tty2     Ss+   2010   0:00 /sbin/mingetty tty2
root      4231  0.0  0.0   3792     8 tty3     Ss+   2010   0:00 /sbin/mingetty tty3
root      4233  0.0  0.0   3792     8 tty4     Ss+   2010   0:00 /sbin/mingetty tty4
root      4234  0.0  0.0   3792     8 tty5     Ss+   2010   0:00 /sbin/mingetty tty5
root      4236  0.0  0.0   3792     8 tty6     Ss+   2010   0:00 /sbin/mingetty tty6
root      5596  0.0  0.0  19368    20 ?        Ss    2010   0:00 DarwinStreamingServer
qtss      5597  0.8  0.9 166572 37408 ?        Sl    2010 523:02 DarwinStreamingServer
root      8714  0.0  0.0      0     0 ?        S    Jan31   0:33 [pdflush]
root      9914  0.0  0.0  65612   968 pts/1    R+   21:49   0:00 ps aux
root     10765  0.0  0.0  76792  1080 ?        Ss   Jan24   0:58 SCREEN
root     10766  0.0  0.0  66212   872 pts/3    Ss   Jan24   0:00 /bin/bash
root     11833  0.0  0.0  63852  1060 pts/3    S+   17:17   0:00 /bin/sh ./launch.sh
root     11834  437 42.9 4126884 1733348 pts/3 Sl+  17:17 1190:50 /usr/bin/java -Xms128m -Xmx512m -XX:+UseConcMarkSweepGC -jar /JavaCore/JavaCore.jar
root     13127  4.7  1.1 110564 46876 ?        Ssl  17:18  12:55 /JavaCore/fetcher.bin
root     19392  0.0  0.0  90108  3336 ?        Rs   20:35   0:00 sshd: root@pts/1 
root     19401  0.0  0.0  66216  1640 pts/1    Ss   20:35   0:00 -bash
root     20567  0.0  0.0  90108   412 ?        Ss   Jan16   1:58 sshd: root@pts/0 
root     20569  0.0  0.0  66084   912 pts/0    Ss   Jan16   0:00 -bash
root     21053  0.0  0.0  63856    28 ?        S    Jan30   0:00 /bin/sh /usr/bin/WowzaMediaServerd /usr/local/WowzaMediaServer/bin/setenv.sh /var/run/WowzaM
root     21054  2.9 10.3 2252652 418468 ?      Sl   Jan30 314:25 java -Xmx1200M -server -Djava.net.preferIPv4Stack=true -Dcom.sun.management.jmxremote=true -
root     21915  0.0  0.0      0     0 ?        S    Feb01   0:00 [pdflush]
root     29996  0.0  0.0  76524  1004 pts/0    S+   14:41   0:00 screen -x

sar -W output:

12:00:01 AM  pswpin/s pswpout/s
12:10:01 AM      0.00      0.00
12:20:01 AM      0.00      0.00
12:30:02 AM      0.00      0.00
12:40:01 AM      0.00      0.00
12:50:01 AM      0.00      0.00
01:00:01 AM      0.00      0.00
01:10:01 AM      0.00      0.00
01:20:01 AM      0.00      0.00
01:30:01 AM      0.00      0.00
01:40:01 AM      0.00      0.00
01:50:01 AM      0.00      0.00
02:00:02 AM      0.00      0.00
02:10:01 AM      0.07      0.00
02:20:01 AM      0.00      0.00
02:30:02 AM      0.00      0.00
02:40:01 AM      0.00      0.00
02:50:01 AM      0.00      0.00
03:00:01 AM      0.00      0.00
03:10:01 AM      0.00      0.00
03:20:01 AM      0.00      0.00
03:30:01 AM      0.00      0.00
03:40:02 AM      0.00      0.00
03:50:01 AM      0.00      0.00
04:00:01 AM      0.00      0.00
04:10:01 AM      0.00      0.00
04:20:02 AM      0.01      0.00
04:30:01 AM      0.00      0.00
04:40:02 AM      0.11      0.00
04:50:01 AM      0.01      0.00
05:00:02 AM      0.03      0.00
05:10:01 AM      0.00      0.00
05:20:02 AM      0.01      0.00
05:30:01 AM      0.04      0.00
05:40:02 AM      0.08      0.00
05:50:01 AM      0.00      0.00
06:00:02 AM      0.11      0.00
06:10:01 AM      0.01      0.00
06:20:01 AM      0.00      0.00
06:30:02 AM      0.00      0.00
06:40:01 AM      0.05      0.00
06:50:02 AM      0.00      0.00
07:00:02 AM      0.01      0.00
07:10:01 AM      0.02      0.00
07:20:02 AM      0.00      0.00
07:30:01 AM      0.00      0.00
07:40:02 AM      0.17      0.00
07:50:01 AM      0.11      0.00
08:00:02 AM      0.00      0.00
08:10:01 AM      0.04      0.00
08:20:02 AM      0.00      0.00
08:30:01 AM      0.00      0.00
08:40:02 AM      0.03      0.00
08:50:01 AM      0.00      0.00
09:00:02 AM      0.08      0.00
09:10:01 AM      0.00      0.00
Average:         0.02      0.00

sar -d 5 0 output

09:18:40 AM       DEV       tps  rd_sec/s  wr_sec/s  avgrq-sz  avgqu-sz     await     svctm     %util
09:18:45 AM    dev8-0     21.96     11.18   2128.54     97.45      0.62     20.33      1.75      3.85
09:18:45 AM    dev8-1      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:18:45 AM    dev8-2     18.16      0.00    507.78     27.96      0.39     21.42      1.09      1.98
09:18:45 AM    dev8-3      3.79     11.18   1620.76    430.32      0.23     15.11      4.95      1.88
09:18:45 AM    dev8-4      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:18:45 AM    dev8-5      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:18:45 AM   dev8-16     19.36      0.00    619.56     32.00      0.35     18.22      0.94      1.82
09:18:45 AM   dev8-17     17.96      0.00    594.01     33.07      0.35     19.61      0.99      1.78
09:18:45 AM   dev8-18      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:18:45 AM   dev8-19      1.40      0.00     25.55     18.29      0.00      0.29      0.29      0.04
09:18:45 AM   dev8-20      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:18:45 AM   dev8-21      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:18:45 AM   dev8-32     25.55    742.51    846.31     62.19      0.69     27.04      1.63      4.17
09:18:45 AM   dev8-33     22.55      0.00    844.71     37.45      0.68     30.18      1.41      3.17
09:18:45 AM   dev8-34      2.99    742.51      1.60    248.53      0.01      3.40      3.40      1.02
09:18:45 AM   dev8-35      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:18:45 AM   dev8-48     18.16      0.00    645.11     35.52      0.41     22.65      1.09      1.98
09:18:45 AM   dev8-49     18.16      0.00    645.11     35.52      0.41     22.65      1.09      1.98
09:18:45 AM   dev8-50      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00
09:18:45 AM   dev8-51      0.00      0.00      0.00      0.00      0.00      0.00      0.00      0.00

output sar -w:

12:00:01 AM   cswch/s
05:10:01 AM  51556.52
05:20:02 AM  54484.96
05:30:01 AM  57533.58
05:40:02 AM  57956.67
05:50:01 AM  57885.02
06:00:02 AM  46017.63
06:10:01 AM  21778.29
06:20:01 AM  43464.68
06:30:02 AM  51766.88
06:40:01 AM  53879.12
06:50:02 AM  56531.08
07:00:02 AM  57732.89
07:10:01 AM  57658.24
07:20:02 AM  58209.16
07:30:01 AM  58446.37
07:40:02 AM  58537.84
07:50:01 AM  58243.39
08:00:02 AM  58362.02
08:10:01 AM  58291.73
08:20:02 AM  58370.15
08:30:01 AM  58545.73
08:40:02 AM  58448.40
08:50:01 AM  58198.55
09:00:02 AM  58313.22
09:10:01 AM  58122.85
09:20:02 AM  58517.96
09:30:01 AM  58338.94
09:40:02 AM  58317.30
09:50:01 AM  58312.40
10:00:01 AM  58337.53
10:10:02 AM  58167.55
10:20:01 AM  58408.61
10:30:01 AM  58133.29
10:40:01 AM  58165.08
10:50:02 AM  58240.77
11:00:01 AM  58236.16

Average:      cswch/s
Average:     55991.47

sar -I SUM output:

12:00:01 AM      INTR    intr/s
05:10:01 AM       sum   3825.03
05:20:02 AM       sum   3999.32
05:30:01 AM       sum   4038.10
05:40:02 AM       sum   4041.99
05:50:01 AM       sum   4015.39
06:00:02 AM       sum   3450.48
06:10:01 AM       sum   2385.73
06:20:01 AM       sum   3355.96
06:30:02 AM       sum   3641.50
06:40:01 AM       sum   3807.91
06:50:02 AM       sum   3853.99
07:00:02 AM       sum   3951.29
07:10:01 AM       sum   3996.06
07:20:02 AM       sum   4005.63
07:30:01 AM       sum   3939.43
07:40:02 AM       sum   3901.39
07:50:01 AM       sum   3920.22
08:00:02 AM       sum   3950.27
08:10:01 AM       sum   3926.09
08:20:02 AM       sum   4072.29
08:30:01 AM       sum   4058.93
08:40:02 AM       sum   3994.94
08:50:01 AM       sum   3969.04
09:00:02 AM       sum   3976.33
09:10:01 AM       sum   3904.43
09:20:02 AM       sum   4054.35
09:30:01 AM       sum   4006.06
09:40:02 AM       sum   3962.65
09:50:01 AM       sum   4016.83
10:00:01 AM       sum   4064.52
10:10:02 AM       sum   3934.29
10:20:01 AM       sum   4029.60
10:30:01 AM       sum   3939.23
10:40:01 AM       sum   3937.37
10:50:02 AM       sum   3961.87
11:00:01 AM       sum   4014.83

Average:         INTR    intr/s
Average:          sum   3794.24

Qualche idea di cosa potrebbe essere o dove dovrei cercare ulteriori informazioni diagnostiche?

Grazie.

linux high-load

— SyRenity
fonte

aggiungi "ps aux" output

— alvosu

L'output è stato aggiunto.

— SyRenity,

7

Intuitivamente, sospetto che un problema del disco sia la causa più diretta, ma ciò non significa che i tuoi dischi siano troppo lenti. Il tuo iowait% di iostat non indica che i processi utente trascorrono molto tempo in attesa di I / O del disco. Tuttavia, il tempo della tua CPU su kswapd mi preoccupa:

root       493  0.1  0.0      0     0 ?        S<    2010  94:48 [kswapd1]

I 242 MB di swap che stai utilizzando potrebbero non sembrare molto, ma per raggiungere quel tipo di tempo della CPU su un sistema che dura da 42 giorni o hai molte attività di swap in corso o ci vuole un'eternità per finire una volta avviato a causa di altri conflitti sul disco. Se questa è la fonte del tuo problema o meno, è qualcosa che vorrei assolutamente esaminare.

Puoi eseguire sar -We pubblicare le statistiche di scambio per il tuo sistema?

— jgoldschrafe
fonte

sar -W risultati aggiunti alla domanda sopra.

— SyRenity,

era perché stavi usando lo swap? Vedi anche serverfault.com/a/524818/27813

— rogerdpack

2

La causa più comune di carico elevato è unità lente. Prova a eseguire quanto segue

sar -d 5 0

e guardando il campo% util. Se quel numero è superiore al 70% per qualsiasi unità, tale unità sarà lenta nella gestione delle richieste di I / O che causano un carico elevato.

Modifica: potrebbe funzionare correttamente al 70%, ma questo è il punto in cui probabilmente inizierai a vedere il degrado delle prestazioni. Più vai in alto, peggio diventerà.

— Patrick
fonte

1

Guarda le dimensioni della coda e i tempi di servizio. Se aumentano, allora hai un problema.

— BillThor,

Sì, questi possono aiutare a indicare i problemi, ma non li ho menzionati perché li ho visti spuntare senza effetti negativi.

— Patrick,

2

Tuttavia, l'utilizzo delle prestazioni del disco come primo arresto in un'analisi della causa principale può essere pericoloso. Uno dei motivi per cui l'I / O su disco è uno degli ultimi posti in cui cercherò un problema, anche se tende sintomaticamente a causare molti problemi, è che ci sono molte cose divertenti che possono andare sbagliato su un sistema che lo farà iniziare a martellare il disco con cose diverse dal carico di lavoro previsto e provare a risolvere il problema aggiungendo mandrini più o più veloci potrebbe non essere sempre l'approccio migliore.

— Jgoldschrafe,

@jgoldschrafe, ovviamente, se il problema è il disco, dovrai capire perché. Non puoi iniziare a capire perché i processi stanno masticando l'IO del disco fino a quando non confermi che l'alto carico è causato dall'IO del disco.

— Patrick,

Ho pubblicato i risultati sar -d, sembra molto al di sotto della soglia del 70%.

— SyRenity,

1

Molti processi di sistema nello stato "S <". Sulla mia macchina sono elencati come solo in "S". Da uomo ps: < high-priority (not nice to other users). Qualcosa sembra davvero incasinato. Prova ad aggiornare il kernel se può essere fatto e riavvia.

— poige
fonte

"dove dovrei cercare ulteriori informazioni diagnostiche?" - prova a cercare nei registri, controlla l'output di dmesg.

— poige,

Questo è normale per questi processi.

— Patrick,

Sto eseguendo la versione 2.6.18-194.el5, che senso ha aggiornare all'ultima versione disponibile?

— SyRenity,

Inoltre, vedo un'omicidio OOM avvenuta la scorsa settimana, non sono sicuro che ciò sia correlato.

— SyRenity,

Se non sbaglio c'è kernel-2.6.18-238 in wild, quindi sicuramente ha senso fino in fondo.

— poige,

1

Che tipo di connessione di rete ha il tuo server? Ho visto carichi vertiginosi in situazioni in cui la connessione a switch doveva essere full duplex a 100 Mbit / s , ma per qualche motivo è stata negoziata come half duplex a 100 Mbit / s . Dopo aver forzato la modalità 100M-FD con ethtool, i carichi sono scesi al di sotto di 1 e le velocità di trasferimento della rete sono tornate alla normalità.

— Janne Pikkarainen
fonte

Ethtool mostra una connessione di Velocità: 1000 Mb / s, Duplex: Completo.

— SyRenity,

OK, probabilmente non è un problema di rete, quindi. La prossima cosa da controllare: numero di interruttori e interruzioni di contesto. Fallo con sar -w (per switch di contesto) e sar -I SUM (per interrupt).

— Janne Pikkarainen,

Ho aggiunto entrambe le uscite nella descrizione sopra.

— SyRenity,

Gli switch di contesto sono piuttosto alti; Di solito vedo qualcosa tra centinaia e un paio di migliaia di switch / s anche su server caricati. Forse quei processi Java stanno facendo qualcosa di strano?

— Janne Pikkarainen,

Le app Java sono fortemente multi-thread, forse questa è la causa del cambio di contesto?

— SyRenity

-1

Hai scritto che vedi un'uccisione OOM nei tuoi registri. Dopo l'attivazione del killer OOM, è necessario riavviare al più presto.

Quindi riavvia e i tuoi problemi spariranno.

Richard

— REMS
fonte

3

L'omicidio OOM ha rimosso il processo offensivo che richiede la maggior parte della memoria, quindi perché è necessario un riavvio?

— SyRenity,