Discussion:
Zenperfsnmp performance graphs not correlating with logs
yong ly
2012-03-15 03:33:26 UTC
Permalink
yong ly [http://community.zenoss.org/people/yongly] created the discussion

"Zenperfsnmp performance graphs not correlating with logs"

To view the discussion, visit: http://community.zenoss.org/message/65205#65205

--------------------------------------------------------------
Hi all,


So I was troubleshooting heartbeat failure alarms for zenperfsnmp and so I went to the 'Advanced' > 'Collectors' > 'Performance' graphs and there it tells me that the cycle times for zenperfsnmp don't really go above 120.. but looking in the zenperfsnmp.log files it clearly shows that it's taking over 300s due to waiting for devices to respond.


So a couple of questions:


1) Is this a bug or am I reading the performance graphs wrong?
2) Could load on the zenoss server or memory leaks be contributing to the errors about devices:


2012-03-14 04:34:04,969 WARNING zen.zenperfsnmp: Continuing to wait for response from DEVICE after 346.249866962 seconds


Before anyone asks, there's about 400 devices being polled.





Thanks,
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65205#65205]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
wangleileo
2012-03-15 06:34:11 UTC
Permalink
wangleileo [http://community.zenoss.org/people/wangleileo] created the discussion

"Re: Zenperfsnmp performance graphs not correlating with logs"

To view the discussion, visit: http://community.zenoss.org/message/65207#65207

--------------------------------------------------------------
I have the same question,the zenperfsnmp log is:
2012-02-28 06:19:42,438 INFO zen.zenperfsnmp: *********************************
2012-02-28 06:19:42,439 INFO zen.zenperfsnmp: success:147 fail:0 pending:0 todo:0
2012-02-28 06:23:52,960 INFO zen.zenperfsnmp: success:0 fail:0 pending:0 todo:147
2012-02-28 06:23:52,976 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.6.202 after 0.015692949295 seconds
2012-02-28 06:23:52,976 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FX10-NB_WinAION_DMZ-C4948-8 after 0.0158669948578 seconds
2012-02-28 06:23:52,976 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.7.6.103 after 0.0159659385681 seconds
2012-02-28 06:23:52,976 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FD1-NB_WinAION_DMZ-CBS3020-S8 after 0.0160579681396 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.7.6.102 after 0.0161499977112 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.24 after 0.0162410736084 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.37 after 0.0163309574127 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FC6-NB_ISG2K-3 after 0.0164411067963 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.32 after 0.0165309906006 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.31 after 0.0166211128235 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.30 after 0.0167109966278 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.39 after 0.0167999267578 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.133.16 after 0.0168890953064 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.133.17 after 0.0169789791107 seconds
2012-02-28 06:23:52,977 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FD11-NB_RS6K_DMZ-switch-S7 after 0.0170669555664 seconds
2012-02-28 06:23:52,978 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FD2-NB_WinTrans_DMZ-CBS3020-S7 after 0.0171589851379 seconds
2012-02-28 06:23:52,978 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.1.92.151 after 0.0172500610352 seconds
2012-02-28 06:23:52,978 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.133.18 after 0.0173399448395 seconds
2012-02-28 06:23:52,978 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.133.19 after 0.0174520015717 seconds
2012-02-28 06:23:52,978 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FX11-NB_WinTran_DMZ-C4948-8 after 0.0175421237946 seconds
2012-02-28 06:23:52,978 ERROR zen.zenperfsnmp: 127 devices still queued at end of cycle and did not get queried.
2012-02-28 06:23:55,459 INFO zen.zenperfsnmp: ******** Cycle completed ********
2012-02-28 06:23:55,459 INFO zen.zenperfsnmp: Sent 1106 OID requests
2012-02-28 06:23:55,460 INFO zen.zenperfsnmp: Queried 20 devices
2012-02-28 06:23:55,460 INFO zen.zenperfsnmp:   127 in queue still unqueried
2012-02-28 06:23:55,460 INFO zen.zenperfsnmp:   Successes: 0  Failures: 0  Not reporting: 20
2012-02-28 06:23:55,460 INFO zen.zenperfsnmp: Waited on 0 queries from previous cycles.
2012-02-28 06:23:55,460 INFO zen.zenperfsnmp:   Successes: 0  Failures: 0  Not reporting: 0
2012-02-28 06:23:55,460 INFO zen.zenperfsnmp: Cycle lasted 2.50 seconds
2012-02-28 06:23:55,461 INFO zen.zenperfsnmp: *********************************
2012-02-28 06:23:55,463 INFO zen.zenperfsnmp: success:0 fail:0 pending:0 todo:127
2012-02-28 06:23:55,489 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.6.203 after 0.0266668796539 seconds
2012-02-28 06:23:55,490 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.6.202 after 2.52917408943 seconds
2012-02-28 06:23:55,490 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.6.201 after 0.0270719528198 seconds
2012-02-28 06:23:55,490 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FX10-NB_WinAION_DMZ-C4948-8 after 2.52951407433 seconds
2012-02-28 06:23:55,490 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.2.16.198 after 0.0273988246918 seconds
2012-02-28 06:23:55,490 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.1.222.222 after 0.0275828838348 seconds
2012-02-28 06:23:55,490 WARNING zen.zenperfsnmp: Continuing to wait for response from 4FX11-NS5400-38 after 0.0277488231659 seconds
2012-02-28 06:23:55,491 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.1.222.221 after 0.0279748439789 seconds
2012-02-28 06:23:55,491 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.7.6.103 after 2.53041601181 seconds
2012-02-28 06:23:55,491 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.7.6.102 after 2.53057694435 seconds
2012-02-28 06:23:55,491 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.2.16.197 after 0.0284600257874 seconds
2012-02-28 06:23:55,491 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.37 after 2.53089809418 seconds
2012-02-28 06:23:55,491 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.8.129 after 0.0287799835205 seconds
2012-02-28 06:23:55,492 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.32 after 2.53127193451 seconds
2012-02-28 06:23:55,492 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.31 after 2.53143405914 seconds
2012-02-28 06:23:55,492 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.30 after 2.53159308434 seconds
2012-02-28 06:23:55,492 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.55 after 0.0294740200043 seconds
2012-02-28 06:23:55,492 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.0.226 after 0.0296349525452 seconds
2012-02-28 06:23:55,492 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.0.224 after 0.0297949314117 seconds
2012-02-28 06:23:55,493 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.39 after 2.53227996826 seconds
2012-02-28 06:23:55,493 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.133.16 after 2.5324420929 seconds
2012-02-28 06:23:55,493 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.133.17 after 2.53260302544 seconds
2012-02-28 06:23:55,493 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FD11-NB_RS6K_DMZ-switch-S7 after 2.5327630043 seconds
2012-02-28 06:23:55,493 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FD2-NB_WinTrans_DMZ-CBS3020-S7 after 2.53292512894 seconds
2012-02-28 06:23:55,493 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.8.105 after 0.0308089256287 seconds
2012-02-28 06:23:55,494 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.133.36 after 0.0309858322144 seconds
2012-02-28 06:23:55,494 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.6.204 after 0.0311479568481 seconds
2012-02-28 06:23:55,494 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.1.92.151 after 2.53358602524 seconds
2012-02-28 06:23:55,494 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.133.18 after 2.53374505043 seconds
2012-02-28 06:23:55,494 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.133.19 after 2.5339050293 seconds
2012-02-28 06:23:55,494 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FD1-NB_WinTran_DMZ-CBS3020-S7 after 0.0317859649658 seconds
2012-02-28 06:23:55,495 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.2.16.205 after 0.0319669246674 seconds
2012-02-28 06:23:55,495 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FX11-NB_WinTran_DMZ-C4948-8 after 2.53440713882 seconds
2012-02-28 06:23:55,495 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FD1-NB_WinAION_DMZ-CBS3020-S8 after 2.53456902504 seconds
2012-02-28 06:23:55,495 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.24 after 2.53472995758 seconds
2012-02-28 06:23:55,495 WARNING zen.zenperfsnmp: Continuing to wait for response from 5FC6-NB_ISG2K-3 after 2.53488898277 seconds
2012-02-28 06:23:55,495 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.44 after 0.0327708721161 seconds
2012-02-28 06:23:55,496 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.1.18.197 after 0.032968044281 seconds
2012-02-28 06:23:55,496 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.48 after 0.0331330299377 seconds
2012-02-28 06:23:55,496 WARNING zen.zenperfsnmp: Continuing to wait for response from 10.0.161.49 after 0.0332930088043 seconds
2012-02-28 06:23:55,496 ERROR zen.zenperfsnmp: 107 devices still queued at end of cycle and did not get queried.
2012-02-28 06:23:55,498 INFO zen.zenperfsnmp: ******** Cycle completed ********
the perfsnmp cycle is 60s
Need to wait a long time into the next cycle




--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65207#65207]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
James Stewart
2013-02-27 04:40:46 UTC
Permalink
James Stewart [http://community.zenoss.org/people/amorphic] created the discussion

"Re: Zenperfsnmp performance graphs not correlating with logs"

To view the discussion, visit: http://community.zenoss.org/message/72178#72178

--------------------------------------------------------------
In case somebody else comes across this...

If the 'Continuing to wait...' message occurs in consecutive zernperfsnmp runs for a given device, an event is raised:

*Summary/Message*: 'Slow snmp response from <device> after 2 cycles.'
*Event Class*: '/Perf/Snmp'
*Severity*: 'Debug'

After some investigation of a similar issue, I've come to the conclusion that the ‘SNMP Slow Response’ events and ‘Continuing to wait’ messages are misleading. They actually indicate +a delay in writing metrics out to rrd files.+

In our case this was due to I/O write spikes on our Zenoss server. According to the standard Device graphs, I/O writes and subsequently CPU Waits were spiking just before the events started to roll in. The writes were unrelated to zenperfsnmp, but zenperfsnmp was negatively impact by them because it is so write-heavy, (outputting data to lots of rrd files).
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/72178#72178]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Loading...