Hi Adam,
Essentially as per the first post, prior to 4737, everything (that I use and play with) was aok, all switches were seen;
Was working in realtime, rddtool showed output (in gui) and showing graphs "4670" : Fabric A (M6220 Stacked pair) Fabric B (M6348 stacked pair) Fabric C (M6220 stacked pair) Nexetra (storage) Firewall *customer site ASA5510* Vyatta A and B - aok everytihg works Cisco switch - aok everything works Xen Servers So all above showed correctly.
After upgrade : Fabric A works gui graphs, rdd info
Fabric B no graph updated, click "RRD command" on right hand side, shows all the RRD stuff, click Graph Previous and hey presto yesterdays graph appears.
Fabric C (same as B) namely : no graph updated, click "RRD command" on right hand side, shows all the RRD stuff, click Graph Previous and hey presto yesterdays graph appears.
Nexetra no graph updated Firewall no graph updated Vyatta - works as before Cisco switch works as before
Now something interesting just found in the observium.log file :
Before and after update.
The line at 11:38 is from manually running "./discovery.php -h none"
/opt/observium/discovery.php new November 4, 2013, 11:35 - 0 devices discovered in 0.022 secs /opt/observium/poller.php 6/8 November 4, 2013, 11:35 - 1 devices polled in 10.65 secs /opt/observium/poller.php 7/8 November 4, 2013, 11:35 - 1 devices polled in 10.66 secs /opt/observium/poller.php 0/8 November 4, 2013, 11:35 - 1 devices polled in 13.32 secs /opt/observium/poller.php 1/8 November 4, 2013, 11:35 - 2 devices polled in 15.21 secs /opt/observium/poller.php 5/8 November 4, 2013, 11:35 - 2 devices polled in 15.87 secs /opt/observium/poller.php 2/8 November 4, 2013, 11:35 - 2 devices polled in 18.84 secs /opt/observium/poller.php 4/8 November 4, 2013, 11:35 - 2 devices polled in 19.79 secs /opt/observium/poller.php 3/8 November 4, 2013, 11:35 - 2 devices polled in 28.62 secs
Upgraded to latest and ran discovery :
./discovery.php none November 4, 2013, 11:38 - 0 devices discovered in 0.504 secs
/opt/observium/discovery.php new November 4, 2013, 11:40 - 0 devices discovered in 0.019 secs /opt/observium/poller.php 6/8 November 4, 2013, 11:40 - 1 devices polled in 7.070 secs /opt/observium/poller.php 7/8 November 4, 2013, 11:40 - 1 devices polled in 7.489 secs /opt/observium/poller.php 0/8 November 4, 2013, 11:40 - 1 devices polled in 10.92 secs /opt/observium/discovery.php new November 4, 2013, 11:45 - 0 devices discovered in 0.007 secs /opt/observium/poller.php 6/8 November 4, 2013, 11:45 - 1 devices polled in 7.188 secs /opt/observium/poller.php 7/8 November 4, 2013, 11:45 - 1 devices polled in 7.456 secs /opt/observium/poller.php 0/8 November 4, 2013, 11:45 - 1 devices polled in 11.01 secs
and for good measure, just in case this is the crontab -l
33 */6 * * * /opt/observium/discovery.php -h all >> /dev/null 2>&1 */5 * * * * /opt/observium/discovery.php -h new >> /dev/null 2>&1 */5 * * * * /opt/observium/poller.php -i 8 -n 0 >> /dev/null 2>&1 */5 * * * * /opt/observium/poller.php -i 8 -n 1 >> /dev/null 2>&1 */5 * * * * /opt/observium/poller.php -i 8 -n 2 >> /dev/null 2>&1 */5 * * * * /opt/observium/poller.php -i 8 -n 3 >> /dev/null 2>&1 */5 * * * * /opt/observium/poller.php -i 8 -n 4 >> /dev/null 2>&1 */5 * * * * /opt/observium/poller.php -i 8 -n 5 >> /dev/null 2>&1 */5 * * * * /opt/observium/poller.php -i 8 -n 6 >> /dev/null 2>&1 */5 * * * * /opt/observium/poller.php -i 8 -n 7 >> /dev/null 2>&1
*/5 * * * * /opt/observium/poll-billing.php >> /dev/null 2>&1 01 * * * * /opt/observium/billing-calculate.php >> /dev/null 2>&1
Adam - more than willing to create you an account if you want to take a peak.
Kind regards,
Chris
----- Original Message ----- From: "Adam Armstrong" adama@memetic.org To: "Observium Network Observation System" observium@observium.org Sent: Tuesday, 5 November, 2013 1:44:44 PM Subject: Re: [Observium] Upgrade to 4737 broke some graphs
Hi Chris,
What *exactly* is and isn't working?
adam.
On 2013-11-05 13:24, Chris Pelling wrote:
Hi Adam,
Cheers for the confirmation of the rdd stuff, what puzzles me (and this could be something) is nothing else changed, all the crons are as they were before, so no users have changed or permissions (unless doing the svn upgrade changed ownership of the files which I will check shortly).
The most curious thing is from the pic sent, some work, some dont. I assume (rightly or wrongly) that as the realtime graphs worked ok - it was not connection based.
Would a downgrade back to 4670 be advisable, what implication DB wise would this have??
Kind regards,
Chris
----- Original Message ----- From: "Adam Armstrong" adama@memetic.org To: "Observium Network Observation System" observium@observium.org Sent: Tuesday, 5 November, 2013 1:04:43 PM Subject: Re: [Observium] Upgrade to 4737 broke some graphs
Hi,
The poller output is normal, it shows rrdtool updating the rrd file fine without errors.
You probably want to check the cron entries for syntax errors and make sure the user the cron job is run as has the correct permissions.
I doubt it's related to the svn up.
adam.
On 2013-11-05 12:43, Chris Pelling wrote: Also, apart from the shared debug output, if this helps this was what the svn upgrade did :
A attic/adva.inc.php A attic/observernms-custom.inc.php A html/images/os/audiocodes.png A html/images/os/barracuda.png U html/includes/functions.inc.php U html/includes/cache-data.inc.php U html/includes/authenticate.inc.php U html/includes/graphs/graph.inc.php U html/includes/graphs/cefswitching/graph.inc.php D html/includes/graphs/device/fanspeed.inc.php D html/includes/graphs/device/voltage.inc.php D html/includes/graphs/device/dbm.inc.php D html/includes/graphs/device/humidity.inc.php D html/includes/graphs/device/current.inc.php D html/includes/graphs/device/power.inc.php D html/includes/graphs/device/frequency.inc.php D html/includes/graphs/device/temperature.inc.php A html/includes/graphs/device/graph.inc.php D html/includes/graphs/sensor/humidity.inc.php A html/includes/graphs/sensor/percent.inc.php U html/includes/graphs/sensor/graph.inc.php A html/includes/graphs/sensor/runtime.inc.php U html/includes/collectd/functions.php U html/includes/print/device.inc.php U html/includes/print/events.inc.php U html/includes/api/functions.inc.php U html/includes/authentication/ldap.inc.php A html/js/ref.js U html/img/sprite.png U html/css/sprite.css A html/css/ref.css U html/pages/device/port.inc.php A html/pages/device/port/data.inc.php U html/pages/device/apps/mssql.inc.php U html/pages/device/apps/mssql/memory.inc.php U html/pages/device/apps/mssql/processor.inc.php U html/pages/health/toner.inc.php U html/pages/health/mempool.inc.php U html/pages/health/sensors.inc.php U html/pages/health/processor.inc.php U html/pages/health/storage.inc.php U html/pages/ports/list.inc.php U html/pages/eventlog.inc.php U html/pages/bill/ports.inc.php U html/pages/graphs.inc.php U html/pages/routing/vrf.inc.php U html/pages/front/default.php U html/pages/device.inc.php U html/pages/pollerlog.inc.php U includes/definitions/os.inc.php A includes/definitions/sensors.inc.php U includes/snmp.inc.php U includes/common.php U includes/rrdtool.inc.php A includes/debug/ref.inc.php A includes/wmi.inc.php U includes/rewrites.php U includes/functions.php U includes/geolocation.inc.php U includes/polling/fdb-table.inc.php U includes/polling/os/ciena.inc.php A includes/polling/os/barracudangfw.inc.php U includes/polling/os/apc.inc.php U includes/polling/bgp-peers.inc.php U includes/polling/applications/exchange.inc.php U includes/polling/applications/mssql.inc.php U includes/polling/applications/powerdns-recursor.inc.php U includes/polling/wmi.inc.php U includes/polling/mempools/ciena.inc.php U includes/polling/functions.inc.php U includes/polling/unix-agent.inc.php U includes/polling/system.inc.php U includes/polling/sensors.inc.php U includes/polling/processors/ciena.inc.php U includes/polling/storage.inc.php U includes/polling/ospf.inc.php U includes/polling/ports.inc.php U includes/defaults.inc.php D includes/discovery/current D includes/discovery/voltages D includes/discovery/power D includes/discovery/dbm D includes/discovery/temperatures D includes/discovery/humidity D includes/discovery/frequencies U includes/discovery/sensors/ipomanii-mib.inc.php U includes/discovery/sensors/ups-mib.inc.php U includes/discovery/sensors/sentry3-mib.inc.php A includes/discovery/sensors/oadwdm-mib.inc.php U includes/discovery/sensors/dell-vendor-mib.inc.php U includes/discovery/sensors/zyxel-as-mib.inc.php U includes/discovery/sensors/juniper-mib.inc.php U includes/discovery/sensors/p8510-mib.inc.php U includes/discovery/sensors/wwp-leos-port-xcvr-mib.inc.php U includes/discovery/sensors/papouch-smi.inc.php A includes/discovery/sensors/f10-chassis-mib.inc.php U includes/discovery/sensors/gamatronic-mib.inc.php U includes/discovery/sensors/mikrotik-mib.inc.php A includes/discovery/sensors/f10-c-series-chassis-mib.inc.php U includes/discovery/sensors/nscrtv-root.inc.php U includes/discovery/sensors/supermicro-health-mib.inc.php U includes/discovery/sensors/juniper-system-mib.inc.php A includes/discovery/sensors/foundry-sn-agent-mib.inc.php U includes/discovery/sensors/cisco-envmon-mib.inc.php A includes/discovery/sensors/alcatel-ind1-health-mib.inc.php U includes/discovery/sensors/lm-sensors-mib.inc.php U includes/discovery/sensors/siconups-mib.inc.php A includes/discovery/sensors/powernet-mib.inc.php A includes/discovery/sensors/carel-ug40cdz-mib.inc.php U includes/discovery/sensors/wwp-leos-chassis-mib.inc.php A includes/discovery/sensors/f10-s-series-chassis-mib.inc.php U includes/discovery/sensors/eqlmember-mib.inc.php U includes/discovery/sensors/fastpath-boxservices-private-mib.inc.php U includes/discovery/sensors/radlan-hwenviroment.inc.php U includes/discovery/sensors/roomalert24e-mib.inc.php U includes/discovery/sensors/deltaups-mib.inc.php U includes/discovery/sensors/rbn-envmon-mib.inc.php A includes/discovery/sensors/s5-chassis-mib.inc.php U includes/discovery/sensors/mg-snmp-ups-mib.inc.php U includes/discovery/sensors/dell-rac-mib.inc.php U includes/discovery/sensors/areca-snmp-mib.inc.php U includes/discovery/sensors/airespace-wireless-mib.inc.php U includes/discovery/sensors/pdu-mib.inc.php U includes/discovery/sensors/juniper-dom-mib.inc.php U includes/discovery/sensors/sw-mib.inc.php U includes/discovery/sensors/webgraph-8xthermometer-us-mib.inc.php A includes/discovery/sensors/foundry-sn-switch-group-mib.inc.php A includes/discovery/sensors/netbotzv2-mib.inc.php U includes/discovery/sensors/asyncos-mail-mib.inc.php U includes/discovery/sensors/xups-mib.inc.php U includes/discovery/sensors/spagent-mib.inc.php U includes/discovery/sensors/ns-root-mib.inc.php U includes/discovery/functions.inc.php A includes/discovery/os/barracuda.inc.php U includes/discovery/os/ciena.inc.php U includes/discovery/sensors.inc.php U includes/discovery/mempools/ciena.inc.php U includes/definitions.inc.php U includes/port-descr-parser.inc.php U poll-billing.php A update/105.sql D mibs/apc U mibs/PowerNet-MIB Updated to revision 4737. root@observium:/opt/observium# ./discovery.php -h none Observium v0.13.11.4737 Discovery
-- Updating database/file schema 104 -> 105 ... (db) done. -- Done Current Revision : 4737
MySQL: Cell[2/0s] Row[16/0s] Rows[3/0s] Column[0/0s] Update[1/0.06s] Insert[0/0s] Delete[0/0s]
Kind regards,
Chris
----- Original Message ----- From: "Chris Pelling" chris@netearth.net To: "Observium Network Observation System" observium@observium.org Sent: Tuesday, 5 November, 2013 9:28:50 AM Subject: Re: [Observium] Upgrade to 4737 broke some graphs
Morning Adam :)
Ok, I have removed as much as I can.
Here is a link to the file : https://drive.google.com/file/d/0BwFhv5Lh2EoiRWNBUzU3VXAxUjQ/edit?usp=sharin...
Kind regards,
Chris
On 05/11/2013 9:00 AM, Adam Armstrong wrote: The commandlines will include SNMP communities (but they *should* be worthless without source-ip, and are clear text on the wire anyways).
The debug output will also contain interface descriptions and bgp session information.
Nothing really worth protecting, but also not something I'd put in public :)
adam.
On 2013-11-05 08:52, Chris Pelling wrote: Hi Mike,
Here is the screen shot, before I paste up the debug output (and me being a "tool") does the debug output carry anything that needs removing, I see the hostname in there, but, although I removed the hostnames from the screenshot, pretty pointless now when looking at this file, I just want to make sure no passwords or anything are in it. :)
Here is the pic of the networks page. Sorry about the inverted screen, but I am blind as a bat :-)
Kind regards,
Chris
On 05/11/2013 4:00 AM, Mike Stupalov wrote: that we could help, show:
- Screenshots
- debug poller out:
./poller.php -d -h [your_f_dell] > /tmp/poller.debug (file /tmp/poller.debug send by pastebin or in mail attachment)
On Mon, Nov 4, 2013 at 11:12 PM, Chris Pelling chris@netearth.net wrote: Hi Guys,
Upgraded today using "svn update" to 4737 from 4760 and 2 switch graphs (dell) stopped showing data, cisco, vyatta and one dell unaffected.
For example, we have 2 x Dell M6220 stacked blades *a and c* and 1 x Dell M6348 stacked blade switch all in M1000e chassis.
a = worked before and after, graph looks aok b = worked before, now stopped c = worked before, now stopped
An interesting observation, if I drill down to a port on B, and click realtime, its showing realtime information - so we know that works. Billing also seems unaffected, just the graphs at : /devices/type=network/format=graphs/graph=bits/ stopped for these 2 devices (albeit one is identical to a working device) exactly at the time of upgrade.
I confirmed we did do the upgrade as per the wiki :
svn update ./discovery.php -h none
So now stumped, what can I do to get them back showing the current data (they have the historical few hours before update) and/or what can I run to aid in finding a poss issue ? :-)
-- Kind regards, Chris
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium _______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium
_______________________________________________ observium mailing list observium@observium.org http://postman.memetic.org/cgi-bin/mailman/listinfo/observium