Graph creation fails: internal error while talking to rrdcached

[Juha Mustonen]

Juha Mustonen [http://community.zenoss.org/people/jmu] created the discussion

"Graph creation fails: internal error while talking to rrdcached"

To view the discussion, visit: http://community.zenoss.org/message/69585#69585

--------------------------------------------------------------
Most likely due the error shown below, all the graphs are broken:

2012-10-29 14:26:37,487 ERROR zen.zenperfsnmp: Failed to write to RRD file: Devices/localhost/os/interfaces/lo/ifHCOutOctets_ifHCOutOctets error request: internal error while talking to rrdcached
2012-10-29 14:26:37,488 ERROR zen.zenperfsnmp: Failed to write to RRD file: Devices/localhost/os/interfaces/eth0/ifHCOutOctets_ifHCOutOctets error request: internal error while talking to rrdcached
2012-10-29 14:26:37,489 ERROR zen.ThresholdInstance: Unable to read RRD file for usedBlocks_usedBlocks
Traceback (most recent call last):
  File "/opt/zenoss/Products/ZenModel/ThresholdInstance.py", line 150, in checkRaw
    cycleTime, rrdType = self._getRRDType(dataPoint)
  File "/opt/zenoss/Products/ZenModel/ThresholdInstance.py", line 169, in _getRRDType
    data = rrdtool_fn()
  File "/opt/zenoss/Products/ZenUtils/Utils.py", line 1739, in _inner
    return fn(*args,**kwargs)
  File "/opt/zenoss/Products/ZenModel/ThresholdInstance.py", line 168, in rrdtool_fn
    return rrdtool.info(self.context().path(dp), *rrd_daemon_args())
error: request: internal error while talking to rrdcached
2012-10-29 14:26:37,490 ERROR zen.zenperfsnmp: Failed to write to RRD file: Devices/localhost/os/interfaces/eth0/ifOutError

The installed versions:

- rrdtool: 1.4.7
- zenoss: 4.2.0

Any suggestions how to fix / where to look for the root cause?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69585#69585]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Juha Mustonen]

Juha Mustonen [http://community.zenoss.org/people/jmu] created the discussion

"Re: Graph creation fails: internal error while talking to rrdcached"

To view the discussion, visit: http://community.zenoss.org/message/69660#69660

--------------------------------------------------------------
Anyone? An error is clear but the root cause is not. All the RRD files seems empty but I suppose it is because the writing fails. Filesystem permissions seems ok (also checked the selinux). Any hints how to debug the issue?

$ rrdtool dump sysUpTime_sysUpTime.rrd |tail -20

                        <!-- 2012-10-13 03:00:00 EEST / 1350086400 --> <row><v>NaN</v></row>
                        <!-- 2012-10-14 03:00:00 EEST / 1350172800 --> <row><v>NaN</v></row>
                        <!-- 2012-10-15 03:00:00 EEST / 1350259200 --> <row><v>NaN</v></row>
                        <!-- 2012-10-16 03:00:00 EEST / 1350345600 --> <row><v>NaN</v></row>
                        <!-- 2012-10-17 03:00:00 EEST / 1350432000 --> <row><v>NaN</v></row>
                        <!-- 2012-10-18 03:00:00 EEST / 1350518400 --> <row><v>NaN</v></row>
                        <!-- 2012-10-19 03:00:00 EEST / 1350604800 --> <row><v>NaN</v></row>
                        <!-- 2012-10-20 03:00:00 EEST / 1350691200 --> <row><v>NaN</v></row>
                        <!-- 2012-10-21 03:00:00 EEST / 1350777600 --> <row><v>NaN</v></row>
                        <!-- 2012-10-22 03:00:00 EEST / 1350864000 --> <row><v>NaN</v></row>
                        <!-- 2012-10-23 03:00:00 EEST / 1350950400 --> <row><v>NaN</v></row>
                        <!-- 2012-10-24 03:00:00 EEST / 1351036800 --> <row><v>NaN</v></row>
                        <!-- 2012-10-25 03:00:00 EEST / 1351123200 --> <row><v>NaN</v></row>
                        <!-- 2012-10-26 03:00:00 EEST / 1351209600 --> <row><v>NaN</v></row>
                        <!-- 2012-10-27 03:00:00 EEST / 1351296000 --> <row><v>NaN</v></row>
                        <!-- 2012-10-28 03:00:00 EEST / 1351382400 --> <row><v>NaN</v></row>
                        <!-- 2012-10-29 02:00:00 EET / 1351468800 --> <row><v>NaN</v></row>
                  </database>
           </rra>
    </rrd>
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69660#69660]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Matthew Yeazel]

Matthew Yeazel [http://community.zenoss.org/people/myeazel] created the discussion

"Re: Graph creation fails: internal error while talking to rrdcached"

To view the discussion, visit: http://community.zenoss.org/message/69652#69652

--------------------------------------------------------------
I had this same issue this week. I got lucky and it may not be the same problem but the Traceback looked the same for me. What I had was a rogue rrdcached process running. service rrdcached start|stop|status looked fine but when I did a ps aux | grep rrd there were two processes running. Once I was able to stop rrdcached and kill the other process, rrdcached seemed to work properly from there. So check that a bad process isn't running. Also a system reboot would fix it but as my server was already in production for something else, I couldn't and therefore had the same problem. Hope that helps.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69652#69652]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Juha Mustonen]

Juha Mustonen [http://community.zenoss.org/people/jmu] created the discussion

"Re: Graph creation fails: internal error while talking to rrdcached"

To view the discussion, visit: http://community.zenoss.org/message/69668#69668

--------------------------------------------------------------
Thanks for the tip, but unfortunatelly the duplicate process does not seem to be the issue here. Just tried restarting the server as well (just to ensure the updated system packages are loaded etc) but outcome still remains the same.

Possibly related info (as I have one working CentOS 6.3 setup already):

* OS: CentOS 6.1
* RRDTool installed from rpmforge
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69668#69668]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[ju55i]

ju55i [http://community.zenoss.org/people/ju55i] created the discussion

"Re: Graph creation fails: internal error while talking to rrdcached"

To view the discussion, visit: http://community.zenoss.org/message/69684#69684

--------------------------------------------------------------
I have the same problem with RHEL 6.3. It seems the rrdcached is refusing the connections. At least that is the message which I got when running the zenrrdcached in foreground (with the run option).
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69684#69684]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jussi Talaskivi]

Jussi Talaskivi [http://community.zenoss.org/people/ju55i] created the discussion

"Re: Graph creation fails: internal error while talking to rrdcached"

To view the discussion, visit: http://community.zenoss.org/message/69685#69685

--------------------------------------------------------------
I got this working. /etc/hosts.allow needs a line:
rrdcached: localhost
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69685#69685]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Juha Mustonen]

Juha Mustonen [http://community.zenoss.org/people/jmu] created the discussion

"Re: Graph creation fails: internal error while talking to rrdcached"

To view the discussion, visit: http://community.zenoss.org/message/69670#69670

--------------------------------------------------------------
Excellent finding, Jussi! This fixed the issue in mine setup as well.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69670#69670]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]