Not sure when this started, but I've got a couple of Linux servers where suddenly the Graphs tab on them blank. The page layout is all there, but the part under the heading Graphs where the graphs and netstats links and graphs should be is empty. Checked the apache logs and see nothing there indicating any problem and poller.php runs fine - one of them:
Observium Poller v0.12.4.3008
Starting polling run:
pegasus.domain.com 65 linux (unix) Using hrSystemUptime (384:13:39:50.94) Uptime: 1 years, 19 days, 13h 39m 50s Hardware: Generic x86 64-bit Version: 2.6.18-164.6.1.el5 Features: CentOS 5.7 Processor Intel Xeon 2.80GHz... 10% Processor Intel Xeon 2.80GHz... 9% Processor Intel Xeon 2.80GHz... 11% Processor Intel Xeon 2.80GHz... 11% Mempool Physical memory: 88.23% Mempool Virtual memory: 43.62% Mempool Swap space: 0.52% Storage /: 19% Storage /boot: 35% Polling Netstats: IP TCP UDP ICMP SNMP HR Stats: Processes Users Polling IP-MIB ipSystemStats ipv4 ipv6 Caching Oids: ifEntry ifXEntry dot3StatsDuplexStatus Port lo(1) HC HC VLAN == bps(37.4kbps/37.4kbps)bytes(434kB/434kB)pkts(38.9pps/38.9pps) Port eth0(2) HC HC dot3Duplex VLAN == bps(340kbps/336kbps)bytes(3.86MB/3.81MB)pkts(153pps/149pps) Port eth1(3) HC dot3Duplex VLAN == bps(0bps/0bps)bytes(0B/0B)pkts(0pps/0pps) Port sit0(4) HC VLAN == bps(0bps/0bps)bytes(0B/0B)pkts(0pps/0pps) Module [ bgp-peers ] disabled on host. Module [ junose-atm-vp ] disabled on host. Module [ toner ] disabled on host. Checking UCD DiskIO MIB: sda sda1 sda2 sda3 Applications: apache mailscanner Module [ wifi ] disabled on host. Module [ ospf ] disabled on host. Module [ cisco-ipsec-flow-monitor ] disabled on host. Module [ cisco-remote-access-monitor ] disabled on host. Module [ cisco-cef ] disabled on host. Module [ cisco-mac-accounting ] disabled on host. Module [ cipsec-tunnels ] disabled on host. Module [ cisco-ace-loadbalancer ] disabled on host. Module [ cisco-ace-serverfarms ] disabled on host. Module [ netscaler-vsvr ] disabled on host. Entity Physical: Module [ unix-agent ] disabled globally. Polled in 1.292 seconds UPDATED! MySQL: Cell[0/0s] Row[3/0s] Rows[11/0s] Column[0/0s] Update[14/0.01s] Insert[1/0s] Delete[0/0s]
Ran poller also in debug mode and saw no errors - everything appeared normal. Just see this on a couple of CentOS 5.x and CentOS 6.x servers - have many others (same OSs) that are fine....
Ideas?
Chris
On Mon, Apr 9, 2012 at 11:47 AM, Chris Stone axisml@gmail.com wrote:
Not sure when this started, but I've got a couple of Linux servers where suddenly the Graphs tab on them blank. The page layout is all there, but the part under the heading Graphs where the graphs and netstats links and graphs should be is empty. Checked the apache logs and see nothing there indicating any problem and poller.php runs fine
Source for the page where I get nothing where the graphs should be:
<div class="tabcontent"><div style="clear: both;"> <div class='rounded-5px' style='border: 1px solid #ccc; display: block; background: #eee; text-align: left; margin-top: 0px; margin-bottom: 5px; padding: 7px 14px 8px 14px'><span style='font-weight: bold;'>Graphs</span> » </div></div> </div> <div class="clearer"></div>
Comparing to a page that does have the graphs, after the » should be a bunch of span and div tags with the graphs. So, something must be aborting during the generation of the graphs - right off since none at all are there - all or nothing it would seem.
New to this code, so I've not done a log of digging around looking for reasons and why some similar systems and not others....
Chris
On 10/04/2012 1:23, Chris Stone wrote:
On Mon, Apr 9, 2012 at 11:47 AM, Chris Stoneaxisml@gmail.com wrote:
Not sure when this started, but I've got a couple of Linux servers where suddenly the Graphs tab on them blank. The page layout is all there, but the part under the heading Graphs where the graphs and netstats links and graphs should be is empty. Checked the apache logs and see nothing there indicating any problem and poller.php runs fine
Source for the page where I get nothing where the graphs should be:
<div class="tabcontent"><div style="clear: both;"> <div class='rounded-5px' style='border: 1px solid #ccc; display: block; background: #eee; text-align: left; margin-top: 0px; margin-bottom: 5px; padding: 7px 14px 8px 14px'><span style='font-weight: bold;'>Graphs</span> »</div></div> </div> <div class="clearer"></div>
Comparing to a page that does have the graphs, after the» should be a bunch of span and div tags with the graphs. So, something must be aborting during the generation of the graphs - right off since none at all are there - all or nothing it would seem.
New to this code, so I've not done a log of digging around looking for reasons and why some similar systems and not others....
Chris,
Try adding /debug=yes to the url and see what it says.
Tom
Tom,
On Mon, Apr 9, 2012 at 5:30 PM, Tom Laermans tom.laermans@powersource.cx wrote:
Try adding /debug=yes to the url and see what it says.
Thanks - that does yield more info - was not aware of that option. Anyway, it shows:
Graphs » SQL[SELECT * FROM device_graphs WHERE device_id = '65'] Notice: Undefined index: group in /opt/observium/html/pages/device/graphs.inc.php on line 51 Notice: Undefined variable: graph_enable in /opt/observium/html/pages/device/graphs.inc.php on line 51
where the graphs would be. Above that are a ton of undefined index and variable notes, but that's true even on pages where the graphs show. Just looked (with /debug=yes) at one that does show the graphs on that tab and it has similar notes, but many more along with the graphs:
Graphs » SQL[SELECT * FROM device_graphs WHERE device_id = '82'] Notice: Undefined index: group in /opt/observium/html/pages/device/graphs.inc.php on line 31 System Notice: Undefined index: firewall in /opt/observium/html/pages/device/graphs.inc.php on line 28 | Netstats Notice: Undefined index: wireless in /opt/observium/html/pages/device/graphs.inc.php on line 28 Notice: Undefined index: storage in /opt/observium/html/pages/device/graphs.inc.php on line 28 Notice: Undefined index: vpdn in /opt/observium/html/pages/device/graphs.inc.php on line 28 Notice: Undefined index: load balancer in /opt/observium/html/pages/device/graphs.inc.php on line 28 Notice: Undefined index: wifi_clients in /opt/observium/html/pages/device/graphs.inc.php on line 55 [...snip..snip...]
Maybe differences in enabled/disabled modules? Can't think of anything else really that might be different.....
Thanks
Chris
On Mon, Apr 9, 2012 at 5:42 PM, Chris Stone axisml@gmail.com wrote:
Maybe differences in enabled/disabled modules? Can't think of anything else really that might be different.....
Apparently not the problem - matched with another system - no change. Ran discovery/poller, same.
Chris
Had some time to look at this more indepth and with /debug=yes, took the query just after the Graphs label and ran it manually against the database and found:
mysql> SELECT * FROM device_graphs WHERE device_id = '65' -> ; Empty set (0.00 sec)
OOPS - there were some there at one point. Checked a similar server and found:
mysql> SELECT * FROM device_graphs WHERE device_id = '55'; +-----------+-------------------+ | device_id | graph | +-----------+-------------------+ | 55 | uptime | | 55 | netstat_icmp | | 55 | netstat_icmp_info | | 55 | netstat_snmp | | 55 | netstat_snmp_pkt | | 55 | hr_processes | | 55 | hr_users | | 55 | ucd_cpu | | 55 | ucd_swap_io | | 55 | ucd_io | | 55 | ucd_contexts | | 55 | ucd_interrupts | | 55 | ucd_memory | | 55 | ucd_load | | 55 | netstat_ip | | 55 | netstat_ip_frag | | 55 | netstat_tcp | | 55 | netstat_udp | +-----------+-------------------+ 18 rows in set (0.00 sec)
So, inserted the uptime one back in for the server where the graphs are missing (id 65) and refreshed the page - uptime graph is back with no gaps in the graphs. Inserted the records for the others and they are now all back. Had another server with the same problem - it's fixed now too.
So, not sure what did it, but seems that at some point, apparently with an update since no other access of the database directly has been done, these records were deleted from the device_graphs table.
Chris
These are synced from an array at the end if the poller function.
The array is built from each module as it updates the rrd. So this is a particularly weird one!
Adam.
participants (3)
-
Adam Armstrong -
Chris Stone -
Tom Laermans