zenoss.snmp.APCDeviceMap crashing zenmodeler?

[moss2k10]

moss2k10 [http://community.zenoss.org/people/moss2k10] created the discussion

"zenoss.snmp.APCDeviceMap crashing zenmodeler?"

To view the discussion, visit: http://community.zenoss.org/message/63369#63369

--------------------------------------------------------------
Hello,
Lately we are experiencing strange errors on only one APC UPS device occuring once or twice a day.

Platform details:

Zenoss  Zenoss 2.5.2
OS  Linux (x86_64) 2.6.18 (Linux 2.6.18-238.12.1.el5 #1 SMP Tue May 31 13:22:04 EDT 2011 x86_64)
Zope  Zope 2.11.2
Python  Python 2.4.4
Database  MySQL 5.0.77 (Ver 5.0.77)
RRD  RRDtool 1.3.8
Twisted  Twisted 8.1.0
NetSnmp  NetSnmp 5.3.2
PyNetSnmp  PyNetSnmp 0.28.13
WMI  Wmi 1.3.6

I always get these three errors:

1. Problem while executing plugin zenoss.snmp.APCDeviceMap
2. Problem while executing plugin zenoss.snmp.DeviceMap
3. Problem while executing plugin zenoss.snmp.NewDeviceMap

Error details:


| | agent | zenmodeler |
| dedupid | TEPLWROUPS04||/Status/Update||4|Problem while executing plugin zenoss.snmp.APCDeviceMap |
| manager | us000xxx |
| Location | |
| ownerid | |
| firstTime | 2011/12/20 15:33:15.000 |
| eventClass | /Status/Update |
| message | Traceback (most recent call last): File "/opt/zenoss/Products/DataCollector/zenmodeler.py", line 610, in processClient results = plugin.preprocess(results, self.log) File "/opt/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 269, in preprocess getdata = self.snmpGetMap.mapdata(getdata) File "/opt/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 318, in mapdata data[self.oidmap[oid]] = value KeyError: '.1.3.6.1.6.3.15.1.1.2.0' |
|



| | agent | zenmodeler |
| dedupid | TEPLWROUPS04||/Status/Update||4|Problem while executing plugin zenoss.snmp.DeviceMap |
| manager | us000xxx |
| Location | |
| ownerid | |
| firstTime | 2011/12/20 15:33:15.000 |
| eventClass | /Status/Update |
| message | Traceback (most recent call last): File "/opt/zenoss/Products/DataCollector/zenmodeler.py", line 610, in processClient results = plugin.preprocess(results, self.log) File "/opt/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 269, in preprocess getdata = self.snmpGetMap.mapdata(getdata) File "/opt/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 318, in mapdata data[self.oidmap[oid]] = value KeyError: '.1.3.6.1.6.3.15.1.1.2.0' |
|



| | agent | zenmodeler |
| dedupid | TEPLWROUPS04||/Status/Update||4|Problem while executing plugin zenoss.snmp.NewDeviceMap |
| manager | us000xxx |
| Location | |
| ownerid | |
| firstTime | 2011/12/20 15:33:15.000 |
| eventClass | /Status/Update |
| message | Traceback (most recent call last): File "/opt/zenoss/Products/DataCollector/zenmodeler.py", line 610, in processClient results = plugin.preprocess(results, self.log) File "/opt/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 269, in preprocess getdata = self.snmpGetMap.mapdata(getdata) File "/opt/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 318, in mapdata data[self.oidmap[oid]] = value KeyError: '.1.3.6.1.6.3.15.1.1.2.0' |
|


What we already checked:

1. restarting zenmodeler doesn't help
2. template for this particular device used for other APC UPSes works correctly without any errors
3. remodelling the device shows no errors but doesn't solve the problem:

2011-12-21 09:31:36 INFO zen.Utils Executing command: /opt/zenoss/bin/zenmodeler run --now --monitor localhost -F -d TEPLWROUPS04 --weblog
2011-12-21 09:31:42 INFO zen.ZenModeler Connecting to localhost:8789
2011-12-21 09:31:42 INFO zen.ZenModeler Connected to ZenHub
2011-12-21 09:31:42 INFO zen.ZenModeler Collecting for device TEPLWROUPS04
2011-12-21 09:31:42 INFO zen.ZenModeler No WMI plugins found for TEPLWROUPS04
2011-12-21 09:31:42 INFO zen.ZenModeler No Python plugins found for TEPLWROUPS04
2011-12-21 09:31:42 INFO zen.ZenModeler No command plugins found for TEPLWROUPS04
2011-12-21 09:31:42 INFO zen.ZenModeler SNMP collection device TEPLWROUPS04
2011-12-21 09:31:42 INFO zen.ZenModeler plugins: zenoss.snmp.NewDeviceMap, zenoss.snmp.DeviceMap, zenoss.snmp.APCDeviceMap
2011-12-21 09:31:43 INFO zen.ZenModeler No portscan plugins found for TEPLWROUPS04
2011-12-21 09:31:45 INFO zen.SnmpClient snmp client finished collection for TEPLWROUPS04
2011-12-21 09:31:45 INFO zen.ZenModeler Processing zenoss.snmp.NewDeviceMap for device TEPLWROUPS04
2011-12-21 09:31:45 INFO zen.ZenModeler processing zenoss.snmp.DeviceMap for device TEPLWROUPS04
2011-12-21 09:31:45 INFO zen.ZenModeler processing zenoss.snmp.APCDeviceMap for device TEPLWROUPS04
2011-12-21 09:31:46 INFO zen.ZenModeler Changes in configuration applied
2011-12-21 09:31:46 INFO zen.ZenModeler Scan time: 4.19 seconds
2011-12-21 09:31:46 INFO zen.ZenModeler Daemon ZenModeler shutting down
2011-12-21 09:31:47 INFO zen.PerformanceConf configuration collected

If You had similar problem please share Your knowledge about it - thanks in advance

BR,
Peter
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63369#63369]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jcurry]

jcurry [http://community.zenoss.org/people/jcurry] created the discussion

"Re: zenoss.snmp.APCDeviceMap crashing zenmodeler?"

To view the discussion, visit: http://community.zenoss.org/message/63421#63421

--------------------------------------------------------------
It looks like you are using the ApcUps ZenPack that I wrote.  Which version of the ZenPack do you have??  I did update it to have more debug checks to catch this sort of error.  If you have the 1.1 version, then it has extra checks.  If you have 1.0 then I would suggest you get the latest version, 1.2, which I put up on github yesterday.

There are no real differences but 1.1 and 1.2, simply packaging.

You can get the source code from https://github.com/jcurry/ZenPacks.ZenSystems.ApcUps https://github.com/jcurry/ZenPacks.ZenSystems.ApcUps (use the zip button if you don't use git) or the egg file from https://github.com/jcurry/ZenPacks.ZenSystems.ApcUps/blob/master/dist/ZenPacks.ZenSystems.ApcUps-1.2-py2.6.egg?raw=true https://github.com/jcurry/ZenPacks.ZenSystems.ApcUps/blob/master/dist/ZenPacks.ZenSystems.ApcUps-1.2-py2.6.egg?raw=true .

Cheers,
Jane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63421#63421]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[moss2k10]

moss2k10 [http://community.zenoss.org/people/moss2k10] created the discussion

"Re: zenoss.snmp.APCDeviceMap crashing zenmodeler?"

To view the discussion, visit: http://community.zenoss.org/message/63482#63482

--------------------------------------------------------------
Hello,
Actually I don't use any APC zenpack - I simply added some OIDs in standard APC template
Loading Image... Loading Image...
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63482#63482]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jcurry]

jcurry [http://community.zenoss.org/people/jcurry] created the discussion

"Re: zenoss.snmp.APCDeviceMap crashing zenmodeler?"

To view the discussion, visit: http://community.zenoss.org/message/63489#63489

--------------------------------------------------------------
Adding stuff to a template should not affect zenmodeler.  It is the zenoss.snmp.APCDeviceMap  modeler plugin that is failing and I'm guessing that you have your modeler at the default, 12-hour, cycle - hence you seeing the problem twice a day.  Are you saying that the zenmodeler daemon actually stops and you have to manually restart it??

The performance template that you screenshotted above is run by the zenperfsnmp daemon and would normally run every 5 minutes.  Do you get performance data for your problem devices (check under the Graphs item on the left-hand menu)???

Chances are that it is an SNMP communications issue on these devices.  Does an snmpwalk command run OK - navigate to the detail page for a device and use the Command menu at the bottom of the left-hand menu to run snmpwalk - it will show you the community name that Zenoss uses and will get system information for that device.  If this fails, chances are that the community name is wrong or possibly you have a firewall blocking SNMP access??

If this works OK, I would use snmpwalk from the command line and try to get the OID that it is wingeing about (.1.3.6.1.6.3.15.1.1.2.0).

snmpwalk -v <snmp version> -c <community name> <device name> 1.3.6.1.6.3.15.1.1.2

See if you get any answers.  Try both problem devices and successful devices.  It may be that you have slightly different firmware in your APC devices, some of which support this SNMP query and some of which don't.

Cheers,
Jane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63489#63489]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[moss2k10]

moss2k10 [http://community.zenoss.org/people/moss2k10] created the discussion

"Re: zenoss.snmp.APCDeviceMap crashing zenmodeler?"

To view the discussion, visit: http://community.zenoss.org/message/63524#63524

--------------------------------------------------------------
Hello Jane,

Post by jcurry
Are you saying that the zenmodeler daemon actually stops and you have to manually restart it??

No, zenmodeler works normally, but for this particular device shows three errors mentioned in my first post

Post by jcurry
Do you get performance data for your problem devices ?

Yes, graphs looks normally with few "snmp agent down" breaks

Post by jcurry
Does an snmpwalk command run OK

Yes, without any warnings/errors

Post by jcurry
snmpwalk -v 3 -a MD5 -A xxxxxxxxxxxxxxx -x DES -X xxxxxxxxxxxxxxx -l authpriv -u zenoss xxxxxxxxxxx .1.3.6.1.6.3.15.1.1.2.0

SNMP-USER-BASED-SM-MIB::usmStatsNotInTimeWindows.0 = Counter32: 3486
Answer looks correct to me and it's the same for both (working and problematic) devices

I think that the problem is caused by this particular device but within several zenoss core installations that I manage this is the first time it occured..
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63524#63524]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[moss2k10]

moss2k10 [http://community.zenoss.org/people/moss2k10] created the discussion

"Re: zenoss.snmp.APCDeviceMap crashing zenmodeler?"

To view the discussion, visit: http://community.zenoss.org/message/63559#63559

--------------------------------------------------------------
As You pointed out before all errors have 12h cycle of recurrence so exactly default cycletime of zenmodeler..I'm having problems with running zenmodeler in debug mode from cli - Could You help me up in this ?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63559#63559]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[luizzmizz]

luizzmizz [http://community.zenoss.org/people/luizzmizz] created the discussion

"Re: zenoss.snmp.APCDeviceMap crashing zenmodeler?"

To view the discussion, visit: http://community.zenoss.org/message/63528#63528

--------------------------------------------------------------
Hi,

If you're referring to not knowing how to run it: on cli, log as zenoss and run *zenmodeler run -v10 -d <devicename>*

And post the output!*
*
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63528#63528]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[moss2k10]

moss2k10 [http://community.zenoss.org/people/moss2k10] created the discussion

"Re: zenoss.snmp.APCDeviceMap crashing zenmodeler?"

To view the discussion, visit: http://community.zenoss.org/message/63560#63560

--------------------------------------------------------------
For this moment debug zenmodeler shows no warnings/errors - I'll try it again around 15:30 when the errors are usually showing up and share the results..
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63560#63560]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[moss2k10]

moss2k10 [http://community.zenoss.org/people/moss2k10] created the discussion

"Re: zenoss.snmp.APCDeviceMap crashing zenmodeler?"

To view the discussion, visit: http://community.zenoss.org/message/65141#65141

--------------------------------------------------------------
Unfortunately I don't have any updates in this topic..Still we have 2 UPS's which constantly show these errors..We will try to upgrade software on affected devices and announce the results.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65141#65141]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]