Here's the output from the vmstat command which I ran on a host that was running 4 Gigablast processes which were spidering and answering queries. Combined, the the processes are only using about 1 gigabyte of the 1.5 gigabytes of physical RAM available. From the output you can see that the disk cache is using the other .5 gigabytes.
So why is it continually swapping the Gigablast processes back and forth? This is the one thing about Linux that totally puzzles me. If you have insight into this issue could you please help me out?
$ cat /proc/version
Linux version 2.4.22 (root@lenny) (gcc version 2.96 20000731 (Red Hat Linux 7.1 2.96-81)) #1 Tue Aug 26 22:07:38 MDT 2003
$ vmstat 1
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 120 30180 1736 46696 0 0 424 25 134 77 5 9 86
3 0 0 120 13908 1776 63192 0 0 8220 0 268 275 15 41 44
0 1 1 120 5012 1784 71668 0 0 4344 5196 452 337 9 32 59
0 1 1 120 4756 1784 71796 0 0 128 7824 294 94 0 6 94
2 0 1 120 3200 1788 72436 0 0 1028 3364 266 91 2 6 92
0 2 0 140 3628 1800 70796 0 0 5368 0 216 286 41 30 30
1 0 0 140 23144 1856 51904 0 0 1968 28 339 500 49 16 36
4 1 0 140 20324 1968 53188 0 0 2152 0 210 368 72 7 21
6 0 0 140 15940 2020 54992 0 0 1796 0 413 315 85 10 5
6 3 0 140 12096 2112 56732 0 0 1880 0 468 478 80 19 1
10 0 0 140 8660 2152 58736 0 0 2040 0 232 448 81 19 0
5 3 0 140 4556 2184 61052 0 0 2340 0 482 1198 35 32 34
1 18 0 268 4248 2156 60252 0 216 2632 216 358 1497 63 29 8
15 4 0 496 3920 1804 59096 0 0 2844 8 238 824 50 20 30
2 0 1 496 3652 1828 57412 0 0 2820 0 244 533 84 15 1
8 0 0 496 3888 1844 60892 0 0 912 0 178 314 88 12 0
1 18 1 916 3468 1856 60448 0 388 2260 8488 400 1184 39 31 31
4 0 1 916 3004 1856 60880 0 0 748 10180 453 594 32 22 46
17 1 1 1044 4736 1856 58116 0 292 488 13376 464 379 20 15 65
12 9 0 1044 4216 1860 58112 0 0 2352 32 300 891 66 26 8
1 0 0 1172 3744 1864 57448 0 8 2252 60 225 310 93 7 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
4 0 1 1172 3788 1884 60744 0 0 1936 0 205 477 83 17 0
5 0 0 1236 3900 1884 57156 0 44 3996 152 473 1076 50 39 11
5 1 0 1236 3868 1896 57080 0 0 1012 0 195 261 97 3 0
9 3 1 1236 3316 1896 58052 0 0 4408 0 228 484 72 28 0
6 0 0 1236 4172 1908 60964 0 0 852 0 163 221 81 19 0
10 0 0 1236 3924 1912 59468 0 0 3364 0 279 751 73 27 0
4 0 0 1452 4228 1920 61500 0 228 2868 484 308 540 57 43 0
3 13 0 1580 4392 1912 59852 0 0 2032 4128 313 690 59 37 4
2 0 1 1580 3044 1468 59396 0 220 2768 224 304 231 88 12 0
1 0 0 1580 4492 1200 57944 0 0 584 0 151 276 93 7 0
5 5 0 1580 3920 1196 61264 0 0 1604 0 174 326 76 24 0
17 3 1 1580 3732 1208 59892 0 0 2748 256 253 621 78 22 0
1 0 0 1580 3656 1208 58952 0 0 1748 0 340 345 93 6 1
1 0 0 1580 7156 1208 55484 0 184 528 184 165 266 79 21 0
2 0 0 1580 6136 1224 60136 0 0 536 0 153 161 87 13 0
2 0 1 1580 3016 1228 61516 0 0 2664 0 197 219 96 4 0
3 0 0 1696 3952 1220 58928 0 0 2268 384 260 492 72 28 0
5 0 0 1696 4136 1228 58780 0 0 244 0 132 137 96 4 0
9 3 1 1952 3748 1232 61220 0 28 1612 4156 315 578 66 34 0
5 1 0 2080 4156 1228 60052 0 36 1924 4180 453 433 60 40 0
7 0 1 2080 3752 1232 59024 0 0 2084 4112 239 329 84 16 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
12 2 0 2336 3904 1252 62540 0 0 896 136 233 410 73 20 8
1 9 0 2848 3416 1252 61012 0 692 2824 4760 250 249 82 18 0
8 5 0 3232 3896 1260 58960 0 0 1928 8 388 769 79 21 0
1 0 0 3232 4624 1272 58512 0 0 404 0 284 233 90 10 0
1 1 1 3488 3696 1276 61864 0 0 2104 4132 217 332 72 28 0
3 4 0 3488 3648 1280 59752 0 0 2880 0 201 301 89 11 0
3 0 0 3616 3860 1288 62116 0 0 1812 0 207 601 77 23 0
23 0 0 4128 4452 1288 59788 0 0 2888 4140 352 485 69 31 0
1 0 0 4128 5360 1296 58376 0 608 1272 608 399 447 76 24 0
9 0 0 4128 3812 1300 62368 0 0 1912 0 206 619 79 21 0
2 0 0 4768 3788 1304 61072 0 1176 1772 5312 384 606 67 33 0
2 1 0 5024 5060 1312 62440 0 400 2024 400 188 346 81 19 0
2 0 0 5024 4052 1328 62484 0 176 1168 176 164 338 89 11 0
2 0 0 5152 3832 1328 60952 0 0 2820 0 218 473 78 22 0
1 0 0 5408 4128 1332 59516 0 556 1256 4696 257 375 83 17 0
4 0 0 5408 4856 1332 62376 0 0 1112 0 312 891 68 32 0
4 0 0 5920 3948 1332 60380 0 64 3740 64 239 427 81 19 0
23 0 1 6048 3804 1336 62716 0 0 2136 3972 255 824 58 42 0
2 0 0 6048 3476 1336 63076 0 0 1992 168 423 409 83 17 0
3 3 0 6176 5748 1336 63900 0 0 704 128 250 444 74 26 0
5 0 0 6176 15760 1384 52536 0 0 856 64 221 426 59 26 15
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
6 0 0 6176 11576 1384 53564 0 0 1044 0 231 389 79 21 0
5 0 0 6176 11560 1384 58084 0 0 420 0 164 367 77 23 0
5 0 0 6176 7784 1388 59052 0 0 972 0 171 327 95 5 0
3 3 0 6176 5676 1392 59784 0 0 732 0 173 340 98 2 0
4 3 0 6176 9664 1392 60036 0 0 252 0 195 255 90 10 0
1 0 0 6560 4236 1404 64348 0 0 344 4132 354 460 71 29 0
1 0 0 6560 13640 1404 55920 0 0 260 16 297 428 73 27 0
3 0 0 6560 12892 1404 56208 0 0 288 0 165 262 87 13 0
1 0 0 6560 13128 1408 56624 0 0 420 128 228 400 90 10 0
3 0 0 6560 12332 1408 57064 0 0 368 0 208 380 86 14 0
3 2 0 6560 12232 1420 57196 0 0 212 0 259 218 93 7 0
1 0 0 6560 12432 1420 57340 0 0 144 0 144 483 76 24 0
1 0 0 6560 11560 1420 57636 0 0 296 0 209 500 86 14 0
1 0 0 6560 11744 1420 57740 0 0 104 0 183 426 85 15 0
11 0 0 6560 11308 1420 58092 0 0 352 0 145 259 91 9 0
6 1 0 6560 10612 1420 58432 0 0 288 0 217 433 88 12 0
2 1 0 6560 7680 1428 60100 0 0 1588 8 183 234 87 9 4
2 0 0 6560 3840 1432 61808 0 0 2736 0 177 240 92 8 0
5 1 0 6560 4456 1436 65052 0 0 532 4096 206 196 83 17 0
2 2 0 6560 4316 1436 65228 0 0 180 0 127 71 99 1 0
6 0 1 6944 3740 1440 64516 0 296 2508 296 189 317 87 13 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
6 0 0 7576 4828 1440 61168 0 1576 2136 1576 429 555 70 30 0
4 0 0 7960 3868 1452 64504 0 92 1576 4188 383 245 64 36 0
2 2 0 8088 4088 1448 63124 0 0 2816 132 217 269 83 15 2
4 3 0 8400 3904 1456 66540 0 0 952 4 170 322 68 30 2
2 0 1 8400 2804 1456 64840 0 0 3036 0 301 320 92 8 0
1 0 0 8656 4672 1456 62612 0 0 1080 4032 217 126 97 3 0
1 0 0 8784 5540 1456 61596 0 0 372 0 210 219 92 8 0
6 8 0 8784 4260 1460 66168 0 0 388 7680 252 254 63 37 0
3 2 0 9040 3372 1460 66812 0 680 2596 680 298 339 89 11 0
4 0 0 9424 4076 1460 63328 0 792 2100 796 450 607 64 36 0
3 0 0 9552 3948 1464 67092 0 336 1888 336 269 454 74 26 0
1 0 0 9552 3856 1464 64816 0 0 2708 0 186 295 82 18 0
2 1 0 10064 5488 1440 66676 0 120 352 5188 291 362 78 22 0
2 0 1 10064 3352 1448 67680 0 0 1164 0 172 102 93 7 0
3 3 0 10576 3800 1452 63484 0 988 3000 5476 420 525 62 38 0
5 0 0 10576 4984 1456 63272 0 0 400 0 326 91 89 11 0
3 0 0 10576 4340 1464 67292 0 0 404 0 157 375 77 23 0
3 0 0 10796 4972 1460 64744 0 512 2288 4736 412 518 66 34 0
3 2 0 10924 6548 1460 62516 0 188 1968 4328 516 532 59 41 0
4 0 0 10924 3832 1464 67448 0 0 1588 0 208 230 68 32 0
4 0 0 11216 3836 1452 64720 0 0 2872 344 187 238 86 14 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 11216 4600 1452 64608 0 0 60 0 119 47 96 4 0
2 0 0 11432 3864 1452 69320 0 0 1544 4280 237 346 77 23 0
16 0 0 11432 3880 1424 66400 0 0 2808 0 185 376 92 8 0
3 0 0 11432 4252 1392 65568 0 0 768 0 170 351 91 9 0
3 0 0 11432 4472 1404 69892 0 0 200 0 201 472 79 21 0
8 1 0 11600 4128 1360 69240 0 0 1688 4112 368 382 76 24 0
1 0 0 11628 3124 1344 68120 0 132 3068 136 215 298 84 16 0
6 3 1 11716 3708 1348 70972 0 248 992 3988 160 380 80 20 0
5 0 0 11972 3896 1296 68756 0 456 2208 848 308 422 78 22 0
23 0 0 12908 4096 1296 66788 0 180 1940 4296 481 365 75 25 0
6 0 0 12908 4100 1300 69656 0 0 1352 0 228 423 73 26 1
13 0 0 13292 4132 1292 67128 0 0 3124 4512 292 237 76 24 0
1 0 0 13292 4860 1296 70292 0 0 264 0 176 200 79 21 0
2 0 0 13804 3884 1288 67616 0 0 2940 8 221 485 85 15 0
7 0 0 13932 4264 1288 70340 0 0 2036 0 198 436 65 35 0
4 0 0 13932 3896 1272 69136 0 340 2116 4472 462 361 78 22 0
1 3 0 14040 3916 1204 68368 0 240 2012 368 268 257 93 7 0
1 0 0 14040 4128 1204 68088 0 0 744 0 157 272 89 11 0
2 0 0 14360 4332 1204 70072 0 1380 1808 5472 296 562 61 39 0
5 0 0 14360 4856 1204 67456 0 104 2268 108 223 226 79 21 0
2 5 0 14360 16908 1216 59372 0 0 376 64 253 297 73 22 5
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 14360 16936 1216 59460 0 0 88 0 128 121 98 2 0
4 5 0 14360 15864 1220 60184 0 0 728 0 142 168 93 6 1
9 5 0 14360 8700 1220 63952 0 0 3768 0 223 255 95 5 0
1 0 0 14360 7184 1236 69056 0 0 952 0 325 386 54 46 0
2 1 0 14360 3788 1236 70716 0 0 1788 0 235 159 96 4 0
3 1 0 14532 3884 1236 68536 0 68 2556 8320 267 316 81 19 0
8 0 0 14532 3888 1236 72620 0 348 1116 348 242 316 81 19 0
6 0 0 14576 3716 1236 69252 0 76 3808 76 345 425 76 24 0
6 0 0 14568 4888 1240 72760 0 0 432 0 330 474 71 29 0
1 3 0 14648 3716 1200 73024 0 36 2008 4148 249 221 83 17 0
1 0 0 14744 3884 1196 70032 0 0 2764 128 193 350 79 21 0
4 0 0 14744 3936 1196 74136 0 0 644 0 142 182 85 15 0
7 2 0 14744 3904 1184 71176 0 0 1732 0 212 436 79 21 0
3 3 0 14852 4172 1184 70352 0 128 2984 4168 253 181 86 14 0
15 1 0 14852 3864 1184 73848 0 0 1016 0 209 326 84 16 0
2 2 0 14972 4076 1172 71600 0 176 2524 4292 566 550 62 38 0
1 0 0 14972 4200 1076 70808 0 0 1328 4128 285 82 87 13 0
12 2 0 14972 3824 1040 74840 0 0 720 0 154 206 84 16 0
12 5 0 15100 3872 1040 71360 32 0 3476 3624 266 420 77 23 0
1 0 0 15228 4224 1040 70720 0 0 768 4140 318 225 89 11 0
5 3 0 15228 4500 1040 73580 0 0 1140 0 186 349 77 23 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
4 0 0 15228 3020 1040 72080 0 0 3352 0 209 320 92 8 0
5 1 0 15356 4084 1044 74776 0 408 728 408 165 226 84 16 0
10 2 0 15356 3768 1032 73216 0 132 2192 132 200 223 91 9 0
2 1 0 15612 3864 1020 71064 0 0 2424 4276 557 479 67 33 0
3 0 0 15612 4624 1024 74100 0 0 328 0 276 321 78 22 0
3 0 0 15740 7672 1032 69808 0 0 1572 4140 270 431 66 34 0
2 0 0 15740 4196 1036 70196 0 0 2948 0 215 329 75 25 0
6 3 0 15764 4148 1032 72824 0 0 1640 0 178 369 73 27 0
4 0 0 15764 4432 1032 70740 0 0 2280 4200 467 595 65 33 2
1 0 0 15764 5780 1032 68984 0 0 676 0 132 188 83 17 0
1 0 0 15764 5752 1032 69076 0 0 92 0 124 72 100 0 0
1 2 1 15764 3688 1032 73300 0 48 2496 4148 211 294 77 23 0
1 0 0 15764 4248 1032 71004 0 216 2564 220 230 434 81 19 0
2 1 0 15764 3644 1040 74216 0 0 1512 128 172 198 84 16 0
7 1 1 15764 3220 1040 71672 0 120 2956 128 375 484 84 11 5
3 0 1 15764 2864 1040 74592 0 24 2232 4132 330 292 63 37 0
2 0 0 15764 6332 1040 71552 0 0 2720 0 190 485 92 8 0
10 2 1 15792 3676 1048 74244 0 0 1484 3936 159 281 80 20 0
2 0 1 15792 3468 1048 72784 0 16 2192 4344 543 353 75 25 0
5 0 0 15792 4776 1048 74164 0 0 608 0 270 413 68 32 0
7 0 0 15792 20876 1084 57948 0 0 576 64 163 223 67 16 17
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 15792 21268 1088 58012 0 0 132 0 243 112 95 5 0
4 5 0 15792 20864 1092 58264 0 0 256 0 138 134 98 2 0
4 0 0 15792 20436 1092 58760 0 0 496 128 178 297 90 10 0
1 0 0 15792 20512 1096 58992 0 0 232 0 281 323 87 13 0
3 0 0 15792 17384 1096 60676 0 0 1680 0 311 266 96 4 0
4 0 0 15792 10356 1096 63808 0 0 3124 0 363 266 74 26 0
2 0 0 15792 11172 1096 68080 0 0 164 0 139 135 88 13 0
2 0 0 15792 8556 1096 69688 0 0 1608 0 171 281 92 8 0
6 1 0 15876 4188 1096 71260 0 12 3176 4024 270 325 65 35 0
10 0 0 15876 5520 1112 74192 0 0 48 0 126 181 79 21 0
2 0 1 15988 3608 1112 73096 0 8 3360 4056 429 330 63 37 0
2 0 0 15988 4188 1112 71532 0 0 1252 0 220 408 88 12 0
1 3 0 16004 4112 1120 74908 0 100 640 4176 196 174 75 25 0
5 0 0 16004 3752 1132 71008 0 0 3740 0 211 345 61 39 0
4 0 0 16004 6120 1136 72744 0 0 336 0 152 229 84 16 0
2 4 0 16004 3596 1140 72716 0 0 3176 0 337 528 81 19 0
1 0 0 16080 4336 1140 71036 0 104 1380 4176 270 531 79 21 0
1 0 0 16080 4244 1140 71136 0 0 100 384 183 95 97 3 0
6 0 1 27736 3564 1148 72856 0 24 2628 4120 309 566 62 38 0
3 0 0 27176 3392 1148 72436 0 120 1216 244 251 167 77 23 0
2 2 0 27144 3436 1156 71484 0 0 976 0 148 180 97 3 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
2 0 0 27124 3940 1160 74528 0 1304 956 5360 272 191 51 21 28
3 0 0 27080 3756 1180 72580 0 260 2552 388 218 429 90 10 0
14 2 1 26808 5096 1188 69868 84 1108 2016 9064 490 567 59 22 19
4 0 0 25548 5912 1192 73032 0 0 436 0 478 736 50 50 0
5 0 0 25548 3940 1192 73548 0 0 1560 0 301 385 93 7 0
1 1 1 25548 3336 1192 71800 8 176 2848 432 258 147 91 9 0
1 0 0 25544 4124 1200 70480 116 60 836 60 171 200 97 3 0
4 0 0 25528 3940 1204 74916 0 140 408 140 167 167 81 19 0
2 0 0 25528 3776 1140 74448 32 408 1004 4500 248 208 92 8 0
14 0 1 25624 3664 1032 71516 96 68 3268 68 227 382 80 20 0
2 0 0 25052 4908 1036 74744 0 36 388 36 276 318 74 26 0
2 0 0 25040 3132 1036 75092 4 0 1616 0 211 261 88 8 4
2 0 1 24924 3788 1024 73144 4 20 2372 4156 380 520 54 32 14
3 0 0 24924 4036 1032 72980 24 28 824 36 199 278 92 3 5
3 0 0 24900 4028 1036 75356 28 0 2200 0 172 352 70 23 7
1 0 0 25512 5416 1020 71668 64 0 2472 4132 714 709 47 36 18
3 0 0 24960 6588 1020 70436 4 96 132 96 339 406 78 19 3
4 0 0 24904 6084 1028 74936 24 0 412 0 146 195 81 18 1
1 3 1 24904 3544 1028 75128 36 240 2784 4356 234 378 87 12 1
9 1 0 24828 5392 1036 72052 24 0 1652 0 191 391 80 19 1
4 0 0 24776 4424 1056 75480 0 136 2044 276 233 615 68 32 0
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
7 0 0 24776 3840 1060 74060 4 4 2252 4036 451 354 82 17 1
6 0 0 24776 3848 1060 73552 4 0 652 0 239 247 89 11 0
7 0 0 24736 3864 1068 72728 28 88 4320 88 272 633 51 48 1
1 0 0 23600 9824 1076 67156 4 4 36 4132 271 226 66 34 0
2 0 0 23596 5676 1080 72924 40 0 1696 0 195 304 70 27 3
2 0 0 23596 3904 1060 72480 96 0 3076 0 265 407 80 10 10
2 0 0 23596 5688 1080 75148 24 0 232 0 171 150 86 10 4
1 3 0 23596 3464 1060 75708 40 0 1472 4520 241 172 54 16 30
18 0 0 23552 4180 1000 72592 164 4 2704 1456 253 513 40 23 37
10 0 0 23552 17880 1032 62400 28 0 176 196 272 371 68 19 13
5 0 0 23400 17456 1036 62944 76 0 668 0 386 636 78 20 2
6 3 0 23400 15120 1044 64348 4 0 1416 0 388 461 83 13 4
13 0 0 23400 8936 1044 67536 0 0 3188 0 1300 760 88 12 0
2 0 0 23400 8320 1044 67896 16 0 372 0 415 422 91 9 0
13 4 1 23400 6152 1048 72608 0 0 568 11980 426 386 80 20 0
4 3 0 23384 5800 1056 73308 0 0 752 2356 281 289 87 13 0
6 0 0 23276 4252 1056 71552 44 0 3780 0 298 668 61 31 8
24 0 0 23276 4136 1056 71592 24 0 208 0 176 303 98 2 0
24 0 0 23276 8376 1060 71516 136 0 192 0 192 368 82 7 11
6 2 0 23272 3932 1064 74144 60 0 2044 0 215 668 69 23 9
5 1 0 23232 3880 904 71952 16 40 3000 40 249 758 60 22 18
procs memory swap io system cpu
r b w swpd free buff cache si so bi bo in cs us sy id
6 0 0 23200 7776 916 72736 0 12 364 12 217 516 76 24 0
1 1 0 23180 4044 924 74280 16 0 3776 0 239 492 64 20 16
0 2 1 23180 3888 944 73928 0 68 544 12840 419 291 1 40 59
2 0 0 21424 12764 956 73932 44 0 188 5716 353 61 2 6 92
0 1 0 21324 15400 1004 75112 2204 0 2328 148 414 407 8 13 79
0 0 0 14168 42736 1036 75704 380 0 1004 0 182 112 2 2 96
0 0 0 14168 42736 1036 75704 0 0 0 0 181 85 0 0 100
0 0 0 14168 42736 1036 75704 0 0 0 0 117 7 0 0 100