Fwiw, we have a similar issue in that existing RRD files can be updated from both our pollers but only one poller can create NEW RRD files.

 

For now we workaround this by polling every device from the ‘working’ poller periodically to create any new RRDs.

 

cid:image001.jpg@01CE9215.D39244B0

Richard Wigzell
Technical Lead, Storage Platforms
IT Services | University of Oxford

13 Banbury Road, Oxford OX2 6NN


T: 01865 616282
www.it.ox.ac.uk

From: observium [mailto:observium-bounces@observium.org] On Behalf Of Christian Eriksson
Sent: 02 August 2018 14:24
To: Mike Stupalov <mike@observium.org>; Observium <observium@observium.org>
Subject: Re: [Observium] rrdcached

 

The file does not exist on the rrdcached server

 

Vänliga hälsningar / Best regards,

 

Advania_Trappa_CMYK

 

Christian Eriksson

System Consultant

 

Advania-Bikupa FacebookAdvania-Bikupa LinkedInAdvania-Bikupa Twitter

 

+4641090488 | +46761400400

www.advania.se | cid:image013.png@01D3683B.F3231C40 Stockholm

 

Från: Mike Stupalov <mike@stupalov.ru> För Mike Stupalov
Skickat: den 2 augusti 2018 15:21
Till: Observium <observium@observium.org>; Christian Eriksson <christian.eriksson@advania.se>
Ämne: Re: [Observium] rrdcached

 

Show file permissions:


ls -l /opt/observium/rrd/device.net.ad.customer.se/ping.rrd

2 August 2018 at 14:48

Attached poller.php -d for one of the devices

RRD /opt/observium/rrd/device.net.ad.customer.se/ping.rrd already exists - no need to create.

ERROR: rrdcached: RRD Error: opening '/opt/observium/rrd/device.net.ad.customer.se/ping.rrd': No such file or directory

 

”rrdtool info” gives me:

root@server:/opt/observium# rrdtool info --daemon device/ping.rrd

ERROR: rrdcached: No such file: /opt/observium/rrd/device/ping.rrd.

 

The same command for uptime.rrd works

 

Vänliga hälsningar / Best regards,

 

Advania_Trappa_CMYK

 

Christian Eriksson

System Consultant

 

Advania-Bikupa FacebookAdvania-Bikupa LinkedInAdvania-Bikupa Twitter

 

+4641090488 | +46761400400

www.advania.se | cid:image013.png@01D3683B.F3231C40 Stockholm

 

Från: Christian Eriksson
Skickat: den 2 augusti 2018 13:11
Till: Observium <observium@observium.org>
Ämne: rrdcached

 

Hi guys,

 

Hope someone here can shed some light on my issue with rrdcached and observium. I am using a distributed setup with web/sql/poller on separate machines. Followed the guide here http://blog.best-practice.se/2017/10/scaling-observium-horizontally.html and currently using two pollers with great success overall

 

What I can’t seem to figure out is why some rrd files won’t get created. It doesn’t matter what poller I am using or what devices, it is always the following rrd-files

ping.rrd

ping_snmp.rrd

status.rrd

 

 

           

 

Some information regarding my system

 

Observium        18.7.9354 (30th July 2018)

OS                      Linux 4.4.0-130-generic [amd64] (Ubuntu 16.04)

Apache              2.4.18 (Ubuntu)

PHP                    7.0.30-0ubuntu0.16.04.1 (OPcache: ENABLED)

Python               2.7.12

MySQL               5.7.23-0ubuntu0.16.04.1 (extension: mysqli 5.0.12-dev)

SNMP                NET-SNMP 5.7.3

RRDtool            1.5.5 (rrdcached 1.5.5: unix:/var/run/rrdcached.sock)

Fping                  3.13 (IPv4 and IPv6)

 

rrdcached settings

$config['rrdcached'] = 'ip address';

$config['rrd']['no_local'] = TRUE;

 

root@server:/home/user# ps aux | grep rrdcached

observi+ 13596  1.6  0.5 1727784 45980 ?       Ssl  Aug01  15:07 /usr/bin/rrdcached -B, -F, -O, -R -L -w 1800 -z 1800 -t 8 -b /opt/observium/rrd/ -j /var/lib/rrdcached/journal/ -G www-data -U observium -p /var/run/rrdcached.pid -s www-data -m 0660 -l unix:/var/run/rrdcached.sock

 

rrdcached journal file show no trace of the files

 

Syslog errors on rrdcached server

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device1.net.ad.handels.se/status.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device1.net.ad.handels.se/ping.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device2/status.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device1.net.ad.handels.se/ping_snmp.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device2/ping.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device2/ping_snmp.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device3/status.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device3/ping.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device3/ping_snmp.rrd) failed.

 

I can manually create a rrd file from the poller with success

rrdtool create device/status.rrd --daemon ip-address RRA:AVERAGE:0.5:1:2016 DS:status:GAUGE:600:-20000:U

 

Best regards

Christian Eriksson

 

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

2 August 2018 at 14:10

Hi guys,

 

Hope someone here can shed some light on my issue with rrdcached and observium. I am using a distributed setup with web/sql/poller on separate machines. Followed the guide here http://blog.best-practice.se/2017/10/scaling-observium-horizontally.html and currently using two pollers with great success overall

 

What I can’t seem to figure out is why some rrd files won’t get created. It doesn’t matter what poller I am using or what devices, it is always the following rrd-files

ping.rrd

ping_snmp.rrd

status.rrd

 

 

           

 

Some information regarding my system

 

Observium        18.7.9354 (30th July 2018)

OS                      Linux 4.4.0-130-generic [amd64] (Ubuntu 16.04)

Apache              2.4.18 (Ubuntu)

PHP                    7.0.30-0ubuntu0.16.04.1 (OPcache: ENABLED)

Python               2.7.12

MySQL               5.7.23-0ubuntu0.16.04.1 (extension: mysqli 5.0.12-dev)

SNMP                NET-SNMP 5.7.3

RRDtool            1.5.5 (rrdcached 1.5.5: unix:/var/run/rrdcached.sock)

Fping                  3.13 (IPv4 and IPv6)

 

rrdcached settings

$config['rrdcached'] = 'ip address';

$config['rrd']['no_local'] = TRUE;

 

root@server:/home/user# ps aux | grep rrdcached

observi+ 13596  1.6  0.5 1727784 45980 ?       Ssl  Aug01  15:07 /usr/bin/rrdcached -B, -F, -O, -R -L -w 1800 -z 1800 -t 8 -b /opt/observium/rrd/ -j /var/lib/rrdcached/journal/ -G www-data -U observium -p /var/run/rrdcached.pid -s www-data -m 0660 -l unix:/var/run/rrdcached.sock

 

rrdcached journal file show no trace of the files

 

Syslog errors on rrdcached server

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device1.net.ad.handels.se/status.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device1.net.ad.handels.se/ping.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device2/status.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device1.net.ad.handels.se/ping_snmp.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device2/ping.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device2/ping_snmp.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device3/status.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device3/ping.rrd) failed.

Aug  2 13:01:01 server rrdcached[13596]: handle_request_update: stat (/opt/observium/rrd/device3/ping_snmp.rrd) failed.

 

I can manually create a rrd file from the poller with success

rrdtool create device/status.rrd --daemon ip-address RRA:AVERAGE:0.5:1:2016 DS:status:GAUGE:600:-20000:U

 

Best regards

Christian Eriksson

 

_______________________________________________
observium mailing list
observium@observium.org
http://postman.memetic.org/cgi-bin/mailman/listinfo/observium

 

--
Mike Stupalov
Observium Limited, http://observium.org