Hi,
I am in the process of qualifying some hardware to run Observium (planning to deploy it in about 25 locations).
So I was trying to evaluate the hardware we currently have which consist of Dell R610 and IBM X3550. So we can gauge what we need.
The R610 has : - 2x Intel X5660 (6 cores 2.8Ghz + hyperthreading) == total of 12 physical cores and 24 logical - 96Gb of DDR3 1333Mhz - 6x300GB 10K SAS configured as one raid1 of 2 disks for mysql and OS and a raid5 of 4 disks for rrds. - 1Gb network interface
The X3550 has : - 2x Intel L5420 (4 cores 2.5Ghz) == total of 8 physical cores - 32Gb of DDR2 266Mhz - 2x200Gb 7.5K SATA configured as one for mysql and OS and one for rrds - 1 Gb network interface
So reading this, it seems easy to guess which one should be faster.
I am currently testing using : Observium CE 0.15.6.6430 OS Linux 2.6.32-358.14.1.el6.x86_64 [amd64] (CentOS 6.2) Apache 2.2.15 (CentOS) PHP 5.6.11 Python 2.6.6 MySQL 5.6.26-log SNMP NET-SNMP 5.5 RRDtool 1.4.9
From my testing, the poller perform a quite better on the R610 than on the X3550 : Polling the same 310 linux boxes (The slowest devices taking around 35 seconds) this gives an average of : - 8 threads : R610 : 169 seconds X3550 : 213 seconds - 12 threads : R610 : 69 seconds X3550 : 156 seconds - 24 threads R610 : 41 seconds X3550 : 107 seconds - 36 threads R610 : 41 seconds X3550 : 160 seconds - 64 threads R610 : 37 seconds X3550 : 225 seconds - 96 Threads R610 : 40 seconds X3550 : Still running
Ok we clearly have a winner here. Bottom line for the poller, you can come close to the time taken by your slowest devices if you add more threads. But it seems that you need quite a few cores available to be able to increase the number of threads without decreasing the efficiency.
This was kind of expected.
Where the thing becomes weird it's on the web interface and especially the graph rendering : On average starting with a clear cache, the same Linux device page render (graphs included) consistently in about 1 second on the X3550 and in about 4 seconds on the R610 (screenshot attached).
Where it became even worse : if you ask the X3550 to connect to the mysql database of the R610 and export the rrds of the R610 over NFS ... the X3550 still render the page faster (still around 1 second).
Any guess on what is going on here ? It doesn't seem right that a machine that is definitely faster accessing mysql and updating rrds would be 4 times slower when rendering graphs.
Best regards, Louis
Hi Louis,
And if you switched off hyperthreading of the X5660 ? And add on both : elevator=deadline
Xavier
Hi Xavier,
Deadline or CFQ scheduler doesn't change anything (It is currently set to deadline).
cat /sys/block/sd*/queue/scheduler noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq
I was also suspecting hyperthreading ... but disabling it doesn't change much. I just did the test with hyperthreading off on the X5660 :
With 12 threads the poller takes around 60 seconds, 47 seconds with 24 threads and 44 seconds with 36 threads and 44 seconds with 64 threads, 44 seconds with 96 threads.
Meaning that there is not much performance hit in disabling the hyperthreading for the polling but it doesn't solve the issue with the slowness in the web interface.
Best regards, Louis
August 13 2015 12:25 PM, "Xavier Beaudouin" kiwi@oav.net wrote:
Hi Louis,
And if you switched off hyperthreading of the X5660 ? And add on both : elevator=deadline
Xavier _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Luis,
Have you tried to run irqbalance?
Regards,
2015-08-13 10:27 GMT-03:00 Louis Bailleul louis.bailleul@phangos.fr:
Hi Xavier,
Deadline or CFQ scheduler doesn't change anything (It is currently set to deadline).
cat /sys/block/sd*/queue/scheduler noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq
I was also suspecting hyperthreading ... but disabling it doesn't change much. I just did the test with hyperthreading off on the X5660 :
With 12 threads the poller takes around 60 seconds, 47 seconds with 24 threads and 44 seconds with 36 threads and 44 seconds with 64 threads, 44 seconds with 96 threads.
Meaning that there is not much performance hit in disabling the hyperthreading for the polling but it doesn't solve the issue with the slowness in the web interface.
Best regards, Louis
August 13 2015 12:25 PM, "Xavier Beaudouin" kiwi@oav.net wrote:
Hi Louis,
And if you switched off hyperthreading of the X5660 ? And add on both : elevator=deadline
Xavier _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Eduardo,
Irqbalance is already running:
/etc/init.d/irqbalance status irqbalance (pid 2133) is running...
Below is the output of collectl that shows that interrupt are somewhat balanced across cores
# collectl -scj waiting for 1 second sample... #<----CPU[HYPER]-----><---------------------------------------------------------Int----------------------------------------------------------> #cpu sys inter ctxsw Cpu0 Cpu1 Cpu2 Cpu3 Cpu4 Cpu5 Cpu6 Cpu7 Cpu8 Cpu9 Cp10 Cp11 Cp12 Cp13 Cp14 Cp15 Cp16 Cp17 Cp18 Cp19 Cp20 Cp21 Cp22 Cp23 0 0 111 162 20 8 13 2 4 5 1 1 3 1 2 0 31 3 1 1 0 1 1 2 2 1 3 0 40 10 13555 8477 683 722 549 520 546 564 469 777 432 453 465 464 783 651 648 614 624 693 528 595 526 524 574 478 99 24 30988 22481 1516 1448 1077 1103 1212 1215 1186 1077 1117 1097 1107 1139 2418 1117 1360 1123 1332 1163 1395 1217 1516 1246 1455 1250 100 10 27754 13589 1239 1199 1054 1042 1086 1084 1116 1037 1068 1063 1034 1045 1757 1106 1196 1065 1196 1100 1245 1149 1340 1146 1243 1129 100 10 27409 11518 1225 1193 1023 1022 1104 1085 1095 1035 1027 1024 1046 1014 1722 1075 1220 1110 1197 1062 1237 1102 1278 1142 1220 1144 100 10 28549 12565 1502 1336 1057 1029 1103 1154 1201 1051 1022 1031 1046 1035 1874 1072 1196 1093 1147 1055 1343 1155 1389 1160 1339 1191 91 16 38001 29505 2955 1844 1133 1081 1296 1465 1838 1113 1131 1087 1131 1104 2979 1101 1580 1086 1489 1072 1970 1653 2314 1854 2127 1641 97 25 40323 33199 4896 1325 1235 1117 1341 1509 3184 1239 1171 1209 1179 1234 1548 1179 1447 1189 1380 1241 1458 2411 1650 2552 1488 2187 99 21 37512 28788 4341 1201 1141 1158 1237 1602 2764 1280 1130 1208 1121 1122 1451 1253 1289 1243 1314 1258 1353 2155 1257 2284 1288 2028 100 18 31300 19933 2452 1223 1214 1107 1214 1258 2021 1072 1085 1128 1070 1098 1170 1113 1173 1084 1243 1158 1199 1651 1183 1632 1162 1531 100 13 30579 15636 2225 1212 1110 1027 1120 1256 1982 1038 1095 1071 1043 1054 1114 1112 1188 1091 1200 1135 1198 1725 1194 1510 1214 1650 99 14 33408 18170 3201 1283 1151 1104 1147 1410 2128 1103 1106 1098 1050 1075 1194 1167 1220 1132 1199 1079 1238 1891 1214 1854 1213 2208 99 23 41141 31271 4692 1543 2216 1118 1282 1722 2717 1105 1233 1182 1190 1126 1255 1197 1545 1157 1877 1164 1766 2221 1773 2340 1494 2361 99 25 38586 30454 3007 1634 2441 1081 1367 1545 1744 1184 1128 1155 1175 1093 1223 1205 1984 1189 2259 1209 2197 1793 1903 1802 1570 1633 100 19 35075 25055 3438 1330 1227 1143 1162 1431 2406 1090 1167 1133 1092 1078 1136 1171 1248 1145 1253 1300 1396 1598 1220 2259 1325 2218 100 16 31435 19261 1697 1341 2135 1082 1243 1389 1542 1050 1050 1078 1079 1096 1086 1183 1586 1104 1215 1088 1378 1404 1397 1311 1449 1449 100 14 33085 19090 2114 1450 2323 1038 1180 1433 1403 1057 1072 1121 1078 1028 1114 1096 2045 1106 1386 1126 1555 1353 1466 1459 1690 1454 99 17 35296 22261 2393 1432 2256 1075 1257 1552 1857 1068 1089 1076 1160 1085 1133 1263 2074 1148 1510 1098 1575 1475 1457 1677 1954 1619 99 23 40571 30239 5102 1357 1321 1235 1306 1708 3211 1146 1206 1134 1166 1154 1306 1270 1365 1211 1337 1208 1381 2560 1348 2299 1480 2838 100 23 37307 28846 3768 1262 1287 1128 1257 1566 2600 1227 1194 1228 1149 1216 1245 1160 1504 1187 1235 1236 1301 2106 1290 2239 1251 2576 100 17 33775 23029 2666 1393 1582 1142 1243 1500 2188 1159 1137 1064 1099 1148 1174 1094 1361 1200 1244 1213 1383 1647 1350 1721 1425 1740 100 17 32365 19986 1861 1588 2257 1077 1132 1240 1455 1048 1056 1090 1110 1063 1139 1114 1503 1083 1376 1081 1638 1296 1625 1381 1571 1435 #<----CPU[HYPER]-----><---------------------------------------------------------Int----------------------------------------------------------> #cpu sys inter ctxsw Cpu0 Cpu1 Cpu2 Cpu3 Cpu4 Cpu5 Cpu6 Cpu7 Cpu8 Cpu9 Cp10 Cp11 Cp12 Cp13 Cp14 Cp15 Cp16 Cp17 Cp18 Cp19 Cp20 Cp21 Cp22 Cp23 100 15 33864 20571 3562 1238 1251 1097 1191 1430 2530 1127 1090 1074 1098 1128 1213 1105 1198 1092 1236 1162 1267 1672 1294 1921 1234 1694 97 20 39682 31105 5326 1303 1309 1143 1322 1897 3575 1144 1187 1101 1150 1118 1263 1194 1295 1139 1414 1141 1437 2137 1339 2362 1311 2157 59 17 36713 40328 5434 1900 1272 1148 1240 1542 3310 815 785 732 725 690 1244 1189 1348 1213 1270 1141 1279 2264 1160 2009 1061 1720 10 3 11369 10951 1285 715 618 418 364 374 1129 136 75 135 39 66 524 511 624 459 483 451 371 716 314 569 346 571 1 0 3541 1767 804 574 257 75 20 39 329 1 3 1 20 0 122 247 209 256 56 109 6 183 3 20 12 179 1 0 3829 2328 956 692 200 31 41 37 393 2 2 0 20 37 269 190 249 126 80 86 22 129 6 46 3 195 0 0 2017 1010 260 260 85 61 47 65 204 2 3 1 20 30 123 156 209 115 39 22 6 127 5 22 4 176 0 0 3668 2323 1063 1074 158 2 45 47 168 0 1 0 29 26 63 229 162 176 22 18 15 269 0 10 6 71 0 0 1980 964 218 309 90 48 30 56 151 10 4 7 17 29 10 249 172 75 80 28 23 208 9 91 6 55 1 0 2581 1086 389 261 244 49 30 28 224 0 1 1 20 57 83 184 267 120 111 89 29 77 2 164 6 137
Best regards, Louis
August 13 2015 2:31 PM, "Eduardo Schoedler" listas@esds.com.br wrote:
Luis,
Have you tried to run irqbalance?
Regards,
2015-08-13 10:27 GMT-03:00 Louis Bailleul louis.bailleul@phangos.fr:
Hi Xavier,
Deadline or CFQ scheduler doesn't change anything (It is currently set to deadline).
cat /sys/block/sd*/queue/scheduler noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq noop anticipatory [deadline] cfq
I was also suspecting hyperthreading ... but disabling it doesn't change much. I just did the test with hyperthreading off on the X5660 :
With 12 threads the poller takes around 60 seconds, 47 seconds with 24 threads and 44 seconds
with
36 threads and 44 seconds with 64 threads, 44 seconds with 96 threads.
Meaning that there is not much performance hit in disabling the hyperthreading for the polling
but
it doesn't solve the issue with the slowness in the web interface.
Best regards, Louis
August 13 2015 12:25 PM, "Xavier Beaudouin" kiwi@oav.net wrote:
Hi Louis,
And if you switched off hyperthreading of the X5660 ? And add on both : elevator=deadline
Xavier _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Eduardo Schoedler _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Louis,
Can you give me http/ssh access to the x3550 to see where the performance drops are?
Thanks, adam. On 13/08/2015 11:58:42, Louis Bailleul louis.bailleul@phangos.fr wrote: Hi,
I am in the process of qualifying some hardware to run Observium (planning to deploy it in about 25 locations).
So I was trying to evaluate the hardware we currently have which consist of Dell R610 and IBM X3550. So we can gauge what we need.
The R610 has : - 2x Intel X5660 (6 cores 2.8Ghz + hyperthreading) == total of 12 physical cores and 24 logical - 96Gb of DDR3 1333Mhz - 6x300GB 10K SAS configured as one raid1 of 2 disks for mysql and OS and a raid5 of 4 disks for rrds. - 1Gb network interface
The X3550 has : - 2x Intel L5420 (4 cores 2.5Ghz) == total of 8 physical cores - 32Gb of DDR2 266Mhz - 2x200Gb 7.5K SATA configured as one for mysql and OS and one for rrds - 1 Gb network interface
So reading this, it seems easy to guess which one should be faster.
I am currently testing using : Observium CE 0.15.6.6430 OS Linux 2.6.32-358.14.1.el6.x86_64 [amd64] (CentOS 6.2) Apache 2.2.15 (CentOS) PHP 5.6.11 Python 2.6.6 MySQL 5.6.26-log SNMP NET-SNMP 5.5 RRDtool 1.4.9
From my testing, the poller perform a quite better on the R610 than on the X3550 :
Polling the same 310 linux boxes (The slowest devices taking around 35 seconds) this gives an average of : - 8 threads : R610 : 169 seconds X3550 : 213 seconds - 12 threads : R610 : 69 seconds X3550 : 156 seconds - 24 threads R610 : 41 seconds X3550 : 107 seconds - 36 threads R610 : 41 seconds X3550 : 160 seconds - 64 threads R610 : 37 seconds X3550 : 225 seconds - 96 Threads R610 : 40 seconds X3550 : Still running
Ok we clearly have a winner here. Bottom line for the poller, you can come close to the time taken by your slowest devices if you add more threads. But it seems that you need quite a few cores available to be able to increase the number of threads without decreasing the efficiency.
This was kind of expected.
Where the thing becomes weird it's on the web interface and especially the graph rendering : On average starting with a clear cache, the same Linux device page render (graphs included) consistently in about 1 second on the X3550 and in about 4 seconds on the R610 (screenshot attached).
Where it became even worse : if you ask the X3550 to connect to the mysql database of the R610 and export the rrds of the R610 over NFS ... the X3550 still render the page faster (still around 1 second).
Any guess on what is going on here ? It doesn't seem right that a machine that is definitely faster accessing mysql and updating rrds would be 4 times slower when rendering graphs.
Best regards, Louis _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Adam,
Sorry this is a fairly isolated corporate network. I won't be able to give access from the external world to this boxes.
But I am glad to help investigate in anyway I can.
Best regards, Louis August 13 2015 12:58 PM, "Adam Armstrong" wrote:
Hi Louis,
Can you give me http/ssh access to the x3550 to see where the performance drops are?
Thanks, adam.
On 13/08/2015 11:58:42, Louis Bailleul wrote:Hi,
I am in the process of qualifying some hardware to run Observium (planning to deploy it in about 25 locations).
So I was trying to evaluate the hardware we currently have which consist of Dell R610 and IBM X3550. So we can gauge what we need.
The R610 has : - 2x Intel X5660 (6 cores 2.8Ghz + hyperthreading) == total of 12 physical cores and 24 logical - 96Gb of DDR3 1333Mhz - 6x300GB 10K SAS configured as one raid1 of 2 disks for mysql and OS and a raid5 of 4 disks for rrds. - 1Gb network interface
The X3550 has : - 2x Intel L5420 (4 cores 2.5Ghz) == total of 8 physical cores - 32Gb of DDR2 266Mhz - 2x200Gb 7.5K SATA configured as one for mysql and OS and one for rrds - 1 Gb network interface
So reading this, it seems easy to guess which one should be faster.
I am currently testing using : Observium CE 0.15.6.6430 OS Linux 2.6.32-358.14.1.el6.x86_64 [amd64] (CentOS 6.2) Apache 2.2.15 (CentOS) PHP 5.6.11 Python 2.6.6 MySQL 5.6.26-log SNMP NET-SNMP 5.5 RRDtool 1.4.9
From my testing, the poller perform a quite better on the R610 than on the X3550 : Polling the same 310 linux boxes (The slowest devices taking around 35 seconds) this gives an average of : - 8 threads : R610 : 169 seconds X3550 : 213 seconds - 12 threads : R610 : 69 seconds X3550 : 156 seconds - 24 threads R610 : 41 seconds X3550 : 107 seconds - 36 threads R610 : 41 seconds X3550 : 160 seconds - 64 threads R610 : 37 seconds X3550 : 225 seconds - 96 Threads R610 : 40 seconds X3550 : Still running
Ok we clearly have a winner here. Bottom line for the poller, you can come close to the time taken by your slowest devices if you add more threads. But it seems that you need quite a few cores available to be able to increase the number of threads without decreasing the efficiency.
This was kind of expected.
Where the thing becomes weird it's on the web interface and especially the graph rendering : On average starting with a clear cache, the same Linux device page render (graphs included) consistently in about 1 second on the X3550 and in about 4 seconds on the R610 (screenshot attached).
Where it became even worse : if you ask the X3550 to connect to the mysql database of the R610 and export the rrds of the R610 over NFS ... the X3550 still render the page faster (still around 1 second).
Any guess on what is going on here ? It doesn't seem right that a machine that is definitely faster accessing mysql and updating rrds would be 4 times slower when rendering graphs.
Best regards, Louis _______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
For we interface pages, what do the bottom left performance info boxes show?
Screenshots of those might be useful!
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 13 August 2015 14:22:48 "Louis Bailleul" louis.bailleul@phangos.fr wrote:
Hi Adam,
Sorry this is a fairly isolated corporate network. I won't be able to give access from the external world to this boxes.
But I am glad to help investigate in anyway I can.
Best regards, Louis August 13 2015 12:58 PM, "Adam Armstrong" wrote:
Hi Louis,
Can you give me http/ssh access to the x3550 to see where the performance drops are?
Thanks, adam.
On 13/08/2015 11:58:42, Louis Bailleul wrote:Hi,
I am in the process of qualifying some hardware to run Observium (planning to deploy it in about 25 locations).
So I was trying to evaluate the hardware we currently have which consist of Dell R610 and IBM X3550. So we can gauge what we need.
The R610 has :
- 2x Intel X5660 (6 cores 2.8Ghz + hyperthreading) == total of 12 physical
cores and 24 logical
- 96Gb of DDR3 1333Mhz
- 6x300GB 10K SAS configured as one raid1 of 2 disks for mysql and OS and a
raid5 of 4 disks for rrds.
- 1Gb network interface
The X3550 has :
- 2x Intel L5420 (4 cores 2.5Ghz) == total of 8 physical cores
- 32Gb of DDR2 266Mhz
- 2x200Gb 7.5K SATA configured as one for mysql and OS and one for rrds
- 1 Gb network interface
So reading this, it seems easy to guess which one should be faster.
I am currently testing using : Observium CE 0.15.6.6430 OS Linux 2.6.32-358.14.1.el6.x86_64 [amd64] (CentOS 6.2) Apache 2.2.15 (CentOS) PHP 5.6.11 Python 2.6.6 MySQL 5.6.26-log SNMP NET-SNMP 5.5 RRDtool 1.4.9
From my testing, the poller perform a quite better on the R610 than on the X3550 : Polling the same 310 linux boxes (The slowest devices taking around 35 seconds) this gives an average of :
- 8 threads :
R610 : 169 seconds X3550 : 213 seconds
- 12 threads :
R610 : 69 seconds X3550 : 156 seconds
- 24 threads
R610 : 41 seconds X3550 : 107 seconds
- 36 threads
R610 : 41 seconds X3550 : 160 seconds
- 64 threads
R610 : 37 seconds X3550 : 225 seconds
- 96 Threads
R610 : 40 seconds X3550 : Still running
Ok we clearly have a winner here. Bottom line for the poller, you can come close to the time taken by your slowest devices if you add more threads. But it seems that you need quite a few cores available to be able to increase the number of threads without decreasing the efficiency.
This was kind of expected.
Where the thing becomes weird it's on the web interface and especially the graph rendering : On average starting with a clear cache, the same Linux device page render (graphs included) consistently in about 1 second on the X3550 and in about 4 seconds on the R610 (screenshot attached).
Where it became even worse : if you ask the X3550 to connect to the mysql database of the R610 and export the rrds of the R610 over NFS ... the X3550 still render the page faster (still around 1 second).
Any guess on what is going on here ? It doesn't seem right that a machine that is definitely faster accessing mysql and updating rrds would be 4 times slower when rendering graphs.
Best regards, Louis _______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Adam,
I have attached screenshot of both web interfaces with the database metrics popup.
As expected the R610 seems to be a little faster in querying the database.
Best regards, Louis
August 13 2015 2:36 PM, "Adam Armstrong" adama@memetic.org wrote:
For we interface pages, what do the bottom left performance info boxes show?
Screenshots of those might be useful!
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 13 August 2015 14:22:48 "Louis Bailleul" louis.bailleul@phangos.fr wrote:
Hi Adam,
Sorry this is a fairly isolated corporate network. I won't be able to give access from the external world to this boxes.
But I am glad to help investigate in anyway I can.
Best regards, Louis
August 13 2015 12:58 PM, "Adam Armstrong" adama@memetic.org wrote:
Hi Louis,
Can you give me http/ssh access to the x3550 to see where the performance drops are? Thanks, adam.
On 13/08/2015 11:58:42, Louis Bailleul louis.bailleul@phangos.fr wrote:
Hi,
I am in the process of qualifying some hardware to run Observium (planning to deploy it in
about
25
locations).
So I was trying to evaluate the hardware we currently have which consist of Dell R610 and IBM X3550. So we can gauge what we need.
The R610 has :
- 2x Intel X5660 (6 cores 2.8Ghz + hyperthreading) == total of 12 physical cores and 24 logical
- 96Gb of DDR3 1333Mhz
- 6x300GB 10K SAS configured as one raid1 of 2 disks for mysql and OS and a raid5 of 4 disks
for
rrds.
- 1Gb network interface
The X3550 has :
- 2x Intel L5420 (4 cores 2.5Ghz) == total of 8 physical cores
- 32Gb of DDR2 266Mhz
- 2x200Gb 7.5K SATA configured as one for mysql and OS and one for rrds
- 1 Gb network interface
So reading this, it seems easy to guess which one should be faster.
I am currently testing using : Observium CE 0.15.6.6430 OS Linux 2.6.32-358.14.1.el6.x86_64 [amd64] (CentOS 6.2) Apache 2.2.15 (CentOS) PHP 5.6.11 Python 2.6.6 MySQL 5.6.26-log SNMP NET-SNMP 5.5 RRDtool 1.4.9
From my testing, the poller perform a quite better on the R610 than on the X3550 : Polling the same 310 linux boxes (The slowest devices taking around 35 seconds) this gives an average of :
- 8 threads :
R610 : 169 seconds X3550 : 213 seconds
- 12 threads :
R610 : 69 seconds X3550 : 156 seconds
- 24 threads
R610 : 41 seconds X3550 : 107 seconds
- 36 threads
R610 : 41 seconds X3550 : 160 seconds
- 64 threads
R610 : 37 seconds X3550 : 225 seconds
- 96 Threads
R610 : 40 seconds X3550 : Still running
Ok we clearly have a winner here. Bottom line for the poller, you can come close to the time taken by your slowest devices if you
add
more threads. But it seems that you need quite a few cores available to be able to increase the number of
threads
without decreasing the efficiency.
This was kind of expected.
Where the thing becomes weird it's on the web interface and especially the graph rendering : On average starting with a clear cache, the same Linux device page render (graphs included) consistently in about 1 second on the X3550 and in about 4 seconds on the R610 (screenshot attached).
Where it became even worse : if you ask the X3550 to connect to the mysql database of the R610
and
export the rrds of the R610 over NFS ... the X3550 still render the page faster (still around 1 second).
Any guess on what is going on here ? It doesn't seem right that a machine that is definitely faster accessing mysql and updating
rrds
would be 4 times slower when rendering graphs.
Best regards, Louis _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Hi Louis,
These images show the r610 as faster, which I'd expect.
I guess that because the R610 is processing things faster, it's putting higher load on the MySQL server, causing the MySQL performance for the UI to tank when the poller is running.
I'd be inclined to run only 8 poller threads on the R610, since it completes in half of the poller period, and it'll keep MySQL performance up for the web UI.
adam. On 13/08/2015 14:40:48, Louis Bailleul louis.bailleul@phangos.fr wrote: Hi Adam,
I have attached screenshot of both web interfaces with the database metrics popup.
As expected the R610 seems to be a little faster in querying the database.
Best regards, Louis
August 13 2015 2:36 PM, "Adam Armstrong" wrote:
For we interface pages, what do the bottom left performance info boxes show?
Screenshots of those might be useful!
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com
On 13 August 2015 14:22:48 "Louis Bailleul" wrote:
Hi Adam,
Sorry this is a fairly isolated corporate network. I won't be able to give access from the external world to this boxes.
But I am glad to help investigate in anyway I can.
Best regards, Louis
August 13 2015 12:58 PM, "Adam Armstrong" wrote:
Hi Louis,
Can you give me http/ssh access to the x3550 to see where the performance drops are? Thanks, adam.
On 13/08/2015 11:58:42, Louis Bailleul wrote:
Hi,
I am in the process of qualifying some hardware to run Observium (planning to deploy it in
about
25
locations).
So I was trying to evaluate the hardware we currently have which consist of Dell R610 and IBM X3550. So we can gauge what we need.
The R610 has :
- 2x Intel X5660 (6 cores 2.8Ghz + hyperthreading) == total of 12 physical cores and 24 logical
- 96Gb of DDR3 1333Mhz
- 6x300GB 10K SAS configured as one raid1 of 2 disks for mysql and OS and a raid5 of 4 disks
for
rrds.
- 1Gb network interface
The X3550 has :
- 2x Intel L5420 (4 cores 2.5Ghz) == total of 8 physical cores
- 32Gb of DDR2 266Mhz
- 2x200Gb 7.5K SATA configured as one for mysql and OS and one for rrds
- 1 Gb network interface
So reading this, it seems easy to guess which one should be faster.
I am currently testing using : Observium CE 0.15.6.6430 OS Linux 2.6.32-358.14.1.el6.x86_64 [amd64] (CentOS 6.2) Apache 2.2.15 (CentOS) PHP 5.6.11 Python 2.6.6 MySQL 5.6.26-log SNMP NET-SNMP 5.5 RRDtool 1.4.9
From my testing, the poller perform a quite better on the R610 than on the X3550 : Polling the same 310 linux boxes (The slowest devices taking around 35 seconds) this gives an average of :
- 8 threads :
R610 : 169 seconds X3550 : 213 seconds
- 12 threads :
R610 : 69 seconds X3550 : 156 seconds
- 24 threads
R610 : 41 seconds X3550 : 107 seconds
- 36 threads
R610 : 41 seconds X3550 : 160 seconds
- 64 threads
R610 : 37 seconds X3550 : 225 seconds
- 96 Threads
R610 : 40 seconds X3550 : Still running
Ok we clearly have a winner here. Bottom line for the poller, you can come close to the time taken by your slowest devices if you
add
more threads. But it seems that you need quite a few cores available to be able to increase the number of
threads
without decreasing the efficiency.
This was kind of expected.
Where the thing becomes weird it's on the web interface and especially the graph rendering : On average starting with a clear cache, the same Linux device page render (graphs included) consistently in about 1 second on the X3550 and in about 4 seconds on the R610 (screenshot attached).
Where it became even worse : if you ask the X3550 to connect to the mysql database of the R610
and
export the rrds of the R610 over NFS ... the X3550 still render the page faster (still around 1 second).
Any guess on what is going on here ? It doesn't seem right that a machine that is definitely faster accessing mysql and updating
rrds
would be 4 times slower when rendering graphs.
Best regards, Louis _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
Adam,
The screenshots were taken with the poller stopped.
But I'll also think that it is not worth having that many threads running as as you said it impact mysql performance and the poller still complete under 5min. Best regards, Louis August 13 2015 4:33 PM, "Adam Armstrong" wrote:
Hi Louis,
These images show the r610 as faster, which I'd expect.
I guess that because the R610 is processing things faster, it's putting higher load on the MySQL server, causing the MySQL performance for the UI to tank when the poller is running.
I'd be inclined to run only 8 poller threads on the R610, since it completes in half of the poller period, and it'll keep MySQL performance up for the web UI.
adam.
On 13/08/2015 14:40:48, Louis Bailleul wrote:Hi Adam,
I have attached screenshot of both web interfaces with the database metrics popup.
As expected the R610 seems to be a little faster in querying the database. Best regards, Louis August 13 2015 2:36 PM, "Adam Armstrong" wrote:
For we interface pages, what do the bottom left performance info boxes show?
Screenshots of those might be useful!
Adam.
Sent with AquaMail for Android http://www.aqua-mail.com (http://www.aqua-mail.com)
On 13 August 2015 14:22:48 "Louis Bailleul" wrote:
Hi Adam,
Sorry this is a fairly isolated corporate network. I won't be able to give access from the external world to this boxes.
But I am glad to help investigate in anyway I can.
Best regards, Louis
August 13 2015 12:58 PM, "Adam Armstrong" wrote:
Hi Louis,
Can you give me http/ssh access to the x3550 to see where the performance drops are? Thanks, adam.
On 13/08/2015 11:58:42, Louis Bailleul wrote:
Hi,
I am in the process of qualifying some hardware to run Observium (planning to deploy it in
about
25
locations).
So I was trying to evaluate the hardware we currently have which consist of Dell R610 and IBM X3550. So we can gauge what we need.
The R610 has :
- 2x Intel X5660 (6 cores 2.8Ghz + hyperthreading) == total of 12 physical cores and 24 logical
- 96Gb of DDR3 1333Mhz
- 6x300GB 10K SAS configured as one raid1 of 2 disks for mysql and OS and a raid5 of 4 disks
for
rrds.
- 1Gb network interface
The X3550 has :
- 2x Intel L5420 (4 cores 2.5Ghz) == total of 8 physical cores
- 32Gb of DDR2 266Mhz
- 2x200Gb 7.5K SATA configured as one for mysql and OS and one for rrds
- 1 Gb network interface
So reading this, it seems easy to guess which one should be faster.
I am currently testing using : Observium CE 0.15.6.6430 OS Linux 2.6.32-358.14.1.el6.x86_64 [amd64] (CentOS 6.2) Apache 2.2.15 (CentOS) PHP 5.6.11 Python 2.6.6 MySQL 5.6.26-log SNMP NET-SNMP 5.5 RRDtool 1.4.9
From my testing, the poller perform a quite better on the R610 than on the X3550 : Polling the same 310 linux boxes (The slowest devices taking around 35 seconds) this gives an average of :
- 8 threads :
R610 : 169 seconds X3550 : 213 seconds
- 12 threads :
R610 : 69 seconds X3550 : 156 seconds
- 24 threads
R610 : 41 seconds X3550 : 107 seconds
- 36 threads
R610 : 41 seconds X3550 : 160 seconds
- 64 threads
R610 : 37 seconds X3550 : 225 seconds
- 96 Threads
R610 : 40 seconds X3550 : Still running
Ok we clearly have a winner here. Bottom line for the poller, you can come close to the time taken by your slowest devices if you
add
more threads. But it seems that you need quite a few cores available to be able to increase the number of
threads
without decreasing the efficiency.
This was kind of expected.
Where the thing becomes weird it's on the web interface and especially the graph rendering : On average starting with a clear cache, the same Linux device page render (graphs included) consistently in about 1 second on the X3550 and in about 4 seconds on the R610 (screenshot attached).
Where it became even worse : if you ask the X3550 to connect to the mysql database of the R610
and
export the rrds of the R610 over NFS ... the X3550 still render the page faster (still around 1 second).
Any guess on what is going on here ? It doesn't seem right that a machine that is definitely faster accessing mysql and updating
rrds
would be 4 times slower when rendering graphs.
Best regards, Louis _______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium)
_______________________________________________ observium mailing list observium@observium.org (mailto:observium@observium.org) http://postman.memetic.org/cgi-bin/mailman/listinfo/observium (http://postman.memetic.org/cgi-bin/mailman/listinfo/observium) @phangos.fr>@memetic.org>@phangos.fr>@memetic.org>
IBM X3550 is the worst server I ever used. I do not recommend for anybody.
-- Eduardo Schoedler
2015-08-13 8:04 GMT-03:00 Louis Bailleul louis.bailleul@phangos.fr:
Hi,
I am in the process of qualifying some hardware to run Observium (planning to deploy it in about 25 locations).
So I was trying to evaluate the hardware we currently have which consist of Dell R610 and IBM X3550. So we can gauge what we need.
The R610 has :
- 2x Intel X5660 (6 cores 2.8Ghz + hyperthreading) == total of 12 physical cores and 24 logical
- 96Gb of DDR3 1333Mhz
- 6x300GB 10K SAS configured as one raid1 of 2 disks for mysql and OS and a raid5 of 4 disks for rrds.
- 1Gb network interface
The X3550 has :
- 2x Intel L5420 (4 cores 2.5Ghz) == total of 8 physical cores
- 32Gb of DDR2 266Mhz
- 2x200Gb 7.5K SATA configured as one for mysql and OS and one for rrds
- 1 Gb network interface
So reading this, it seems easy to guess which one should be faster.
I am currently testing using : Observium CE 0.15.6.6430 OS Linux 2.6.32-358.14.1.el6.x86_64 [amd64] (CentOS 6.2) Apache 2.2.15 (CentOS) PHP 5.6.11 Python 2.6.6 MySQL 5.6.26-log SNMP NET-SNMP 5.5 RRDtool 1.4.9
From my testing, the poller perform a quite better on the R610 than on the X3550 : Polling the same 310 linux boxes (The slowest devices taking around 35 seconds) this gives an average of :
- 8 threads :
R610 : 169 seconds X3550 : 213 seconds
- 12 threads :
R610 : 69 seconds X3550 : 156 seconds
- 24 threads
R610 : 41 seconds X3550 : 107 seconds
- 36 threads
R610 : 41 seconds X3550 : 160 seconds
- 64 threads
R610 : 37 seconds X3550 : 225 seconds
- 96 Threads
R610 : 40 seconds X3550 : Still running
Ok we clearly have a winner here. Bottom line for the poller, you can come close to the time taken by your slowest devices if you add more threads. But it seems that you need quite a few cores available to be able to increase the number of threads without decreasing the efficiency.
This was kind of expected.
Where the thing becomes weird it's on the web interface and especially the graph rendering : On average starting with a clear cache, the same Linux device page render (graphs included) consistently in about 1 second on the X3550 and in about 4 seconds on the R610 (screenshot attached).
Where it became even worse : if you ask the X3550 to connect to the mysql database of the R610 and export the rrds of the R610 over NFS ... the X3550 still render the page faster (still around 1 second).
Any guess on what is going on here ? It doesn't seem right that a machine that is definitely faster accessing mysql and updating rrds would be 4 times slower when rendering graphs.
Best regards, Louis
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
it would be nice for you to explain WHY its the worse sever you have used?
IO?, Memory?, crashes?, Other?
Simon
On 13 Aug 2015, at 1:56 pm, Eduardo Schoedler listas@esds.com.br wrote:
IBM X3550 is the worst server I ever used. I do not recommend for anybody.
-- Eduardo Schoedler
2015-08-13 8:04 GMT-03:00 Louis Bailleul louis.bailleul@phangos.fr:
Hi,
I am in the process of qualifying some hardware to run Observium (planning to deploy it in about 25 locations).
So I was trying to evaluate the hardware we currently have which consist of Dell R610 and IBM X3550. So we can gauge what we need.
The R610 has :
- 2x Intel X5660 (6 cores 2.8Ghz + hyperthreading) == total of 12 physical cores and 24 logical
- 96Gb of DDR3 1333Mhz
- 6x300GB 10K SAS configured as one raid1 of 2 disks for mysql and OS and a raid5 of 4 disks for rrds.
- 1Gb network interface
The X3550 has :
- 2x Intel L5420 (4 cores 2.5Ghz) == total of 8 physical cores
- 32Gb of DDR2 266Mhz
- 2x200Gb 7.5K SATA configured as one for mysql and OS and one for rrds
- 1 Gb network interface
So reading this, it seems easy to guess which one should be faster.
I am currently testing using : Observium CE 0.15.6.6430 OS Linux 2.6.32-358.14.1.el6.x86_64 [amd64] (CentOS 6.2) Apache 2.2.15 (CentOS) PHP 5.6.11 Python 2.6.6 MySQL 5.6.26-log SNMP NET-SNMP 5.5 RRDtool 1.4.9
From my testing, the poller perform a quite better on the R610 than on the X3550 : Polling the same 310 linux boxes (The slowest devices taking around 35 seconds) this gives an average of :
- 8 threads :
R610 : 169 seconds X3550 : 213 seconds
- 12 threads :
R610 : 69 seconds X3550 : 156 seconds
- 24 threads
R610 : 41 seconds X3550 : 107 seconds
- 36 threads
R610 : 41 seconds X3550 : 160 seconds
- 64 threads
R610 : 37 seconds X3550 : 225 seconds
- 96 Threads
R610 : 40 seconds X3550 : Still running
Ok we clearly have a winner here. Bottom line for the poller, you can come close to the time taken by your slowest devices if you add more threads. But it seems that you need quite a few cores available to be able to increase the number of threads without decreasing the efficiency.
This was kind of expected.
Where the thing becomes weird it's on the web interface and especially the graph rendering : On average starting with a clear cache, the same Linux device page render (graphs included) consistently in about 1 second on the X3550 and in about 4 seconds on the R610 (screenshot attached).
Where it became even worse : if you ask the X3550 to connect to the mysql database of the R610 and export the rrds of the R610 over NFS ... the X3550 still render the page faster (still around 1 second).
Any guess on what is going on here ? It doesn't seem right that a machine that is definitely faster accessing mysql and updating rrds would be 4 times slower when rendering graphs.
Best regards, Louis
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Eduardo Schoedler _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
All the options above. The server simply hung up when I compile the kernel. A LOT OF stress with IBM to exchange it... and the issue persisted. They sent me A LOT of firmwares do test... basically I was a guinea pig.
2015-08-13 10:10 GMT-03:00 Simon Smith simonsmith5521@gmail.com:
it would be nice for you to explain WHY its the worse sever you have used?
IO?, Memory?, crashes?, Other?
Simon
On 13 Aug 2015, at 1:56 pm, Eduardo Schoedler listas@esds.com.br wrote:
IBM X3550 is the worst server I ever used. I do not recommend for anybody.
-- Eduardo Schoedler
2015-08-13 8:04 GMT-03:00 Louis Bailleul louis.bailleul@phangos.fr:
Hi,
I am in the process of qualifying some hardware to run Observium (planning to deploy it in about 25 locations).
So I was trying to evaluate the hardware we currently have which consist of Dell R610 and IBM X3550. So we can gauge what we need.
The R610 has :
- 2x Intel X5660 (6 cores 2.8Ghz + hyperthreading) == total of 12 physical cores and 24 logical
- 96Gb of DDR3 1333Mhz
- 6x300GB 10K SAS configured as one raid1 of 2 disks for mysql and OS and a raid5 of 4 disks for rrds.
- 1Gb network interface
The X3550 has :
- 2x Intel L5420 (4 cores 2.5Ghz) == total of 8 physical cores
- 32Gb of DDR2 266Mhz
- 2x200Gb 7.5K SATA configured as one for mysql and OS and one for rrds
- 1 Gb network interface
So reading this, it seems easy to guess which one should be faster.
I am currently testing using : Observium CE 0.15.6.6430 OS Linux 2.6.32-358.14.1.el6.x86_64 [amd64] (CentOS 6.2) Apache 2.2.15 (CentOS) PHP 5.6.11 Python 2.6.6 MySQL 5.6.26-log SNMP NET-SNMP 5.5 RRDtool 1.4.9
From my testing, the poller perform a quite better on the R610 than on the X3550 : Polling the same 310 linux boxes (The slowest devices taking around 35 seconds) this gives an average of :
- 8 threads :
R610 : 169 seconds X3550 : 213 seconds
- 12 threads :
R610 : 69 seconds X3550 : 156 seconds
- 24 threads
R610 : 41 seconds X3550 : 107 seconds
- 36 threads
R610 : 41 seconds X3550 : 160 seconds
- 64 threads
R610 : 37 seconds X3550 : 225 seconds
- 96 Threads
R610 : 40 seconds X3550 : Still running
Ok we clearly have a winner here. Bottom line for the poller, you can come close to the time taken by your slowest devices if you add more threads. But it seems that you need quite a few cores available to be able to increase the number of threads without decreasing the efficiency.
This was kind of expected.
Where the thing becomes weird it's on the web interface and especially the graph rendering : On average starting with a clear cache, the same Linux device page render (graphs included) consistently in about 1 second on the X3550 and in about 4 seconds on the R610 (screenshot attached).
Where it became even worse : if you ask the X3550 to connect to the mysql database of the R610 and export the rrds of the R610 over NFS ... the X3550 still render the page faster (still around 1 second).
Any guess on what is going on here ? It doesn't seem right that a machine that is definitely faster accessing mysql and updating rrds would be 4 times slower when rendering graphs.
Best regards, Louis
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
-- Eduardo Schoedler _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
participants (5)
-
Adam Armstrong -
Eduardo Schoedler -
Louis Bailleul -
Simon Smith -
Xavier Beaudouin