Problem with zenperfsnmp

[German Zuccoli]

German Zuccoli [http://community.zenoss.org/people/germanzuccoli] created the discussion

"Problem with zenperfsnmp"

To view the discussion, visit: http://community.zenoss.org/message/71184#71184

--------------------------------------------------------------
Hi everyone,

I manage a network of 2000 devices using Zenoss 3.2.1. 2000 routers are monitored only by ICMP. The server localhost is the only one that is monitored with SNMP.

The process zenperfsnmp falls periodically. We verified that the SNMP service its ok and that the problem is in the application.

The graphs below show how process fails and how it affects the localhost.


***@zenoss:~> zenperfsnmp status
program running; pid=3214

zenperfsnmp run -v10
2013-01-21 21:31:04,104 INFO zen.zenperfsnmp: Initializing daemon...
2013-01-21 21:31:04,108 DEBUG zen.zenperfsnmp: Starting PBDaemon initialization
2013-01-21 21:31:04,108 INFO zen.zenperfsnmp: Connecting to localhost:8789
2013-01-21 21:31:04,109 DEBUG zen.zenperfsnmp: Logging in as admin
2013-01-21 21:31:04,116 INFO zen.zenperfsnmp: Connected to ZenHub
2013-01-21 21:31:04,116 DEBUG zen.zenperfsnmp: Setting up initial services: EventService, SnmpPerfConfig
2013-01-21 21:31:04,117 DEBUG zen.zenperfsnmp: Chaining getInitialServices with d2
2013-01-21 21:31:04,117 DEBUG zen.zenperfsnmp: Loaded service EventService from zenhub
2013-01-21 21:31:04,118 DEBUG zen.zenperfsnmp: Loaded service SnmpPerfConfig from zenhub
2013-01-21 21:31:04,118 DEBUG zen.zenperfsnmp: Queueing event {'severity': 0, 'component': 'zenperfsnmp', 'agent': 'zenperfsnmp', 'summary': 'started', 'manager': 'zenoss.XXX.com.uy', 'device': 'localhost', 'eventClass': '/App/Start', 'monitor': 'localhost'}
2013-01-21 21:31:04,118 DEBUG zen.zenperfsnmp: Total of 1 queued events
2013-01-21 21:31:04,118 DEBUG zen.zenperfsnmp: Calling connected.
2013-01-21 21:31:04,118 DEBUG zen.zenperfsnmp: Connected to zenhub
2013-01-21 21:31:04,118 INFO zen.zenperfsnmp: Fetching property items...
2013-01-21 21:31:04,131 DEBUG zen.zenperfsnmp: Updated configCycleInterval config to 360
2013-01-21 21:31:04,131 INFO zen.zenperfsnmp: Getting threshold classes...
2013-01-21 21:31:04,132 DEBUG zen.zenperfsnmp: Loading classes ['Products.ZenModel.MinMaxThreshold']
2013-01-21 21:31:04,132 INFO zen.zenperfsnmp: Checking for outdated configs...
2013-01-21 21:31:07,594 INFO zen.zenperfsnmp: Fetching configs for ['localhost']...
2013-01-21 21:31:08,838 INFO zen.zenperfsnmp: Fetching default RRDCreateCommand...
2013-01-21 21:31:09,111 INFO zen.zenperfsnmp: Getting collector thresholds...
2013-01-21 21:31:09,301 DEBUG zen.thresholds: Updating threshold ('high event queue', ('localhost', ''))
2013-01-21 21:31:09,301 DEBUG zen.thresholds: Updating threshold ('zenmodeler cycle time', ('localhost', ''))
2013-01-21 21:31:09,302 DEBUG zen.thresholds: Updating threshold ('zenperfsnmp cycle time', ('localhost', ''))
2013-01-21 21:31:09,302 DEBUG zen.thresholds: Updating threshold ('zenping cycle time', ('localhost', ''))
2013-01-21 21:31:09,302 DEBUG zen.thresholds: Updating threshold ('zenprocess cycle time', ('localhost', ''))
2013-01-21 21:31:09,302 INFO zen.zenperfsnmp: Fetching SNMP status...
2013-01-21 21:31:10,060 INFO zen.zenperfsnmp: Initiating incremental device load
2013-01-21 21:31:10,060 DEBUG zen.zenperfsnmp: Fetching configs for 1 devices
2013-01-21 21:31:10,060 DEBUG zen.zenperfsnmp: Fetching config for ['localhost']
2013-01-21 21:31:10,152 DEBUG zen.zenperfsnmp: Received config for localhost
2013-01-21 21:31:10,152 DEBUG zen.thresholds: Updating threshold ('high load', ('localhost', ''))
2013-01-21 21:31:10,152 DEBUG zen.thresholds: Updating threshold ('low CPU idle', ('localhost', ''))
2013-01-21 21:31:10,152 DEBUG zen.thresholds: Updating threshold ('high utilization', ('localhost', 'eth0'))
2013-01-21 21:31:10,152 DEBUG zen.thresholds: Updating threshold ('high utilization', ('localhost', 'lo'))
2013-01-21 21:31:10,153 DEBUG zen.thresholds: Updating threshold ('high disk usage', ('localhost', '/boot'))
2013-01-21 21:31:10,153 DEBUG zen.thresholds: Updating threshold ('high disk usage', ('localhost', '/dev'))
2013-01-21 21:31:10,153 DEBUG zen.thresholds: Updating threshold ('high disk usage', ('localhost', '/'))
2013-01-21 21:31:10,153 DEBUG zen.zenperfsnmp: Finished fetching configs for 1 devices
2013-01-21 21:31:10,153 INFO zen.zenperfsnmp: Configured 1 of 1 devices
2013-01-21 21:31:10,153 DEBUG zen.zenperfsnmp: Getting device ping issues
2013-01-21 21:31:10,239 DEBUG zen.zenperfsnmp: unresponsive devices: [('SUB_65_14', 5, '121380'), ... DEVICES ... ('SUB_37_10', 5, '1')]
2013-01-21 21:31:10,248 DEBUG zen.zenperfsnmp: Sent a 'stop' event
2013-01-21 21:31:10,248 INFO zen.zenperfsnmp: success:0 fail:0 pending:0 todo:0
2013-01-21 21:31:10,248 INFO zen.zenperfsnmp: ******** Cycle completed ********
2013-01-21 21:31:10,248 INFO zen.zenperfsnmp: Sent 0 OID requests
2013-01-21 21:31:10,248 INFO zen.zenperfsnmp: Queried 0 devices
2013-01-21 21:31:10,248 INFO zen.zenperfsnmp:   0 in queue still unqueried
2013-01-21 21:31:10,248 INFO zen.zenperfsnmp:   Successes: 0  Failures: 0  Not reporting: 0
2013-01-21 21:31:10,248 INFO zen.zenperfsnmp: Waited on 0 queries from previous cycles.
2013-01-21 21:31:10,248 INFO zen.zenperfsnmp:   Successes: 0  Failures: 0  Not reporting: 0
2013-01-21 21:31:10,248 INFO zen.zenperfsnmp: Cycle lasted 0.00 seconds
2013-01-21 21:31:10,248 INFO zen.zenperfsnmp: *********************************
2013-01-21 21:31:10,249 DEBUG zen.thresholds: Checking value 2.90870666504e-05 on Daemons/localhost/zenperfsnmp_cycleTime
2013-01-21 21:31:10,249 DEBUG zen.MinMaxCheck: Checking zenperfsnmp_cycleTime 2.90870666504e-05 against min None and max 240.0
2013-01-21 21:31:10,249 DEBUG zen.zenperfsnmp: Queueing event {'manager': 'zenoss.labanca.com.uy', 'eventKey': 'zenperfsnmp cycle time', 'device': 'localhost', 'eventClass': '/Perf/Snmp', 'summary': 'threshold of zenperfsnmp cycle time restored: current value 0.00', 'component': '', 'monitor': 'localhost', 'agent': 'zenperfsnmp', 'severity': 0}
2013-01-21 21:31:10,249 DEBUG zen.zenperfsnmp: Total of 1 queued events
2013-01-21 21:31:10,250 DEBUG zen.zenperfsnmp: stop() called when not running
2013-01-21 21:31:10,250 INFO zen.zenperfsnmp: Daemon zenperfsnmp shutting down
2013-01-21 21:31:10,250 DEBUG zen.zenperfsnmp: Removing service EventService
2013-01-21 21:31:10,250 DEBUG zen.zenperfsnmp: Removing service SnmpPerfConfig

Looking for help on this one.

Thanks
-German
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71184#71184]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Problem with zenperfsnmp"

To view the discussion, visit: http://community.zenoss.org/message/71203#71203

--------------------------------------------------------------
I don't think we can fix the crash of zenperfsnmp really unless you upgrade to the later versions - they may fix it. However, have you enabled the watchdog for that process? That should bring it back up quickly.

--
James Pulver
ZCA Member
LEPP Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71203#71203]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[German Zuccoli]

German Zuccoli [http://community.zenoss.org/people/germanzuccoli] created the discussion

"Re: Problem with zenperfsnmp"

To view the discussion, visit: http://community.zenoss.org/message/71215#71215

--------------------------------------------------------------
thanks for the reply.

to configure watchdog All I have to do is this?

[***@comtelecom ~]# cat /usr/local/zenoss/zenoss/etc/zenperfsnmp.conf
#PARAMETER      VALUE
monitor         localhost
watchdog True

In that case the problem persists.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71215#71215]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dhopp]

dhopp [http://community.zenoss.org/people/dhopp] created the discussion

"Re: Problem with zenperfsnmp"

To view the discussion, visit: http://community.zenoss.org/message/71223#71223

--------------------------------------------------------------
German -

When data stops being collected (the gaps in your graphs) are there any events active for the localhost device? 

Also, your zenperfsnmp output shows that it didn't try to monitor anything.  Can you rerun it like this:

zenperfsnmp run now -v10 -d localhost

--Dennis
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71223#71223]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[German Zuccoli]

German Zuccoli [http://community.zenoss.org/people/germanzuccoli] created the discussion

"Re: Problem with zenperfsnmp"

To view the discussion, visit: http://community.zenoss.org/message/71240#71240

--------------------------------------------------------------
Hi Dennis

The are no events for the localhost.

Comand output is

2013-01-25 14:35:21,599 DEBUG zen.zenperfsnmp: Watchdog cycleTime=1200 startTimeout=1320 maxTime=600
2013-01-25 14:35:21,603 DEBUG watchdog: waiting 1319.999994 seconds
2013-01-25 14:35:21,603 DEBUG watchdog: Running ['/usr/local/zenoss/python/bin/.python.bin', '/usr/local/zenoss/zenoss/Products/ZenRRD/zenperfsnmp.py', '--configfile', '/usr/local/zenoss/zenoss/etc/zenperfsnmp.conf', 'now', '-v10', '-d', 'localhost', '--watchdogPath', '/usr/local/zenoss/zenoss/var/.zenperfsnmp-watchdog-24527']
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71240#71240]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dhopp]

dhopp [http://community.zenoss.org/people/dhopp] created the discussion

"Re: Problem with zenperfsnmp"

To view the discussion, visit: http://community.zenoss.org/message/71254#71254

--------------------------------------------------------------
That's not the full output.  You should have gotten something similar to what you originally posted but what I had you run would have ran against only the localhost device.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71254#71254]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[German Zuccoli]

German Zuccoli [http://community.zenoss.org/people/germanzuccoli] created the discussion

"Re: Problem with zenperfsnmp"

To view the discussion, visit: http://community.zenoss.org/message/71310#71310

--------------------------------------------------------------
Dennis,

After adding watchdog I have the following error

# zenperfsnmp restart
stopping...
starting...

#zenperfsnmp run -v10
2013-01-30 13:53:43,579 DEBUG zen.zenperfsnmp: Watchdog cycleTime=1200 startTime                                                                          out=1320 maxTime=600
2013-01-30 13:53:43,586 DEBUG watchdog: waiting 1319.999994 seconds
2013-01-30 13:53:43,587 DEBUG watchdog: Running ['/usr/local/zenoss/python/bin/.                                                                          python.bin', '/usr/local/zenoss/zenoss/Products/ZenRRD/zenperfsnmp.py', '--confi                                                                          gfile', '/usr/local/zenoss/zenoss/etc/zenperfsnmp.conf', '-v10', '--watchdogPath                                                                          ', '/usr/local/zenoss/zenoss/var/.zenperfsnmp-watchdog-2924']
^C2013-01-30 13:55:48,141 DEBUG watchdog: Waiting for command to connect ['/usr/local/zenoss/python/bin/.python.bin', '/usr/local/zenoss/zenoss/Products/ZenRRD/zenperfsnmp.py', '--configfile', '/usr/local/zenoss/zenoss/etc/zenperfsnmp.conf', '-v10', '--watchdogPath', '/usr/local/zenoss/zenoss/var/.zenperfsnmp-watchdog-2924']
2013-01-30 13:55:48,141 DEBUG watchdog: Killing 2967 with 2
2013-01-30 13:55:48,141 ERROR watchdog: [Errno 11] Resource temporarily unavailable
Traceback (most recent call last):
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Watchdog.py", line 217, in run
    self._runOnce()
  File "/usr/local/zenoss/zenoss/Products/ZenUtils/Watchdog.py", line 161, in _runOnce
    conn, addr = sock.accept()
File "/usr/local/zenoss/python/lib/python2.6/socket.py", line 195, in accept
    sock, addr = self._sock.accept()
error: [Errno 11] Resource temporarily unavailable
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71310#71310]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]