error with zenoss.snmp.NewDeviceMap on 3

[Jeff Creek]

Jeff Creek [http://community.zenoss.org/people/jcreek] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/62864#62864

--------------------------------------------------------------
Bump... me too...

Receiving the same error with several Enterasys switches.

"Traceback (most recent call last): File "/usr/local/zenoss/zenoss/Products/DataCollector/zenmodeler.py", line 621, in processClient results = plugin.preprocess(results, self.log) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 271, in preprocess getdata = self.snmpGetMap.mapdata(getdata) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 320, in mapdata data[self.oidmap[oid]] = value KeyError: (1, 3, 6, 1, 2, 1, 1, 2, 0)"

When I perform an snmpget for this MIB, I receive a correct response.
"SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.52.3.9.20.1.4"

OR

"Traceback (most recent call last): File "/usr/local/zenoss/zenoss/Products/DataCollector/zenmodeler.py", line 621, in processClient results = plugin.preprocess(results, self.log) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 271, in preprocess getdata = self.snmpGetMap.mapdata(getdata) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 320, in mapdata data[self.oidmap[oid]] = value KeyError: (1, 3, 6, 1, 2, 1, 1, 1, 0)"

snmpget:
"SNMPv2-MIB::sysDescr.0 = STRING: Enterasys Networks, Inc. G3G124-24 Rev 06.03.05.0004"

on a SuSE Linux server:
"Traceback (most recent call last): File "/usr/local/zenoss/zenoss/Products/DataCollector/zenmodeler.py", line 621, in processClient results = plugin.preprocess(results, self.log) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 271, in preprocess getdata = self.snmpGetMap.mapdata(getdata) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 320, in mapdata data[self.oidmap[oid]] = value KeyError: (1, 3, 6, 1, 2, 1, 1, 2, 0)"

snmpget:
"SNMPv2-MIB::sysObjectID.0 = OID: NET-SNMP-MIB::netSnmpAgentOIDs.10"


I receive no errors when modeling these devices.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/62864#62864]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[kotique]

kotique [http://community.zenoss.org/people/kotique] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/62946#62946

--------------------------------------------------------------
Highly possible that's happening because of duplicate SNMP Engine ID -- in my case that was because of VM cloning. Though I don't remember precisely if that triggered the above behaviour.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/62946#62946]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jeff Creek]

Jeff Creek [http://community.zenoss.org/people/jcreek] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/63060#63060

--------------------------------------------------------------
So, I need to configure a unique SNMP engine ID on each switch?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63060#63060]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Gert Burger]

Gert Burger [http://community.zenoss.org/people/gertburger] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/63036#63036

--------------------------------------------------------------
Hi

We are experiencing the same problem across all our linux machines(VM or real). This seems to happen most often directly after a reboot but does NOT happen on a forced modelling of a device.

Has anyone figured out why this happens?

The stack trace we receive is:
Traceback (most recent call last):
File "/usr/local/zenoss/zenoss/Products/DataCollector/zenmodeler.py", line 621, in processClient
results = plugin.preprocess(results, self.log)
File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 271, in preprocess
getdata = self.snmpGetMap.mapdata(getdata)
File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 320, in mapdata
data[self.oidmap[oid]] = value
KeyError: (1, 3, 6, 1, 2, 1, 1, 2, 0)
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63036#63036]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[zenphil]

zenphil [http://community.zenoss.org/people/zenphil] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/63155#63155

--------------------------------------------------------------
I've created an internal ticket #29598 to look into this issue. I'm unaware of any plans to create a new 3.x patch so the fix may only be in the  4.x community release.

Regards,

Phil Bowman
Sr. Software Developer, Zenoss
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63155#63155]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jeff Creek]

Jeff Creek [http://community.zenoss.org/people/jcreek] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/63141#63141

--------------------------------------------------------------
Thank you, Phil!

Jeff
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/63141#63141]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[chas williams]

chas williams [http://community.zenoss.org/people/chasmosaurus] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/65137#65137

--------------------------------------------------------------
i was getting the same thing.  i tracked this down to a problem with twistedsnmp.py.  i am not sure exactly what is happening but it appears to be a race with return_dct and the callback's from netsnmp.  the following patch fixes the problem for me:


--- ../python/lib/python2.6/site-packages/pynetsnmp/twistedsnmp.py.orig 2012-03-13 11:11:50.752632482 -0400
+++ ../python/lib/python2.6/site-packages/pynetsnmp/twistedsnmp.py      2012-03-12 17:52:00.384273688 -0400
@@ -167,8 +167,9 @@

             return
         for oid, value in response:
-            if self.return_dct:
-                oid = asOidStr(oid)
+            oid = asOidStr(oid)
+            #if self.return_dct:
+            #    oid = asOidStr(oid)
             if isinstance(value, tuple):
                 value = asOidStr(value)
             result.append((oid, value))



it seems safe to just always stuff the non-tuple version of the oid into the result's array.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65137#65137]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[chas williams]

chas williams [http://community.zenoss.org/people/chasmosaurus] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/65158#65158

--------------------------------------------------------------
well that fix broke table browsing but now i think i see the "right" fix.
--- ./python/lib/python2.6/site-packages/pynetsnmp/twistedsnmp.py.orig  2012-03-13 11:11:50.752632482 -0400
+++ ./python/lib/python2.6/site-packages/pynetsnmp/twistedsnmp.py       2012-03-13 14:54:25.387611655 -0400
@@ -179,7 +179,6 @@
             result = []
         if self.return_dct:
             result = dict(result)
-            self.return_dct = False
         reactor.callLater(0, d.callback, result)
            
     def timeout_(self, reqid):
@@ -262,6 +261,7 @@

     def getTable(self, oids, **kw):
         from tableretriever import TableRetriever
+        self.return_dct = False
         try:
             t = TableRetriever(self, oids, **kw)
         except Exception, ex:
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65158#65158]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jeff Creek]

Jeff Creek [http://community.zenoss.org/people/jcreek] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/65216#65216

--------------------------------------------------------------
Tried this fix.  Now I receive errors on (I believe) every device.

Problem while executing plugin zenoss.snmp.DeviceMap
Traceback (most recent call last): File "/usr/local/zenoss/zenoss/Products/DataCollector/zenmodeler.py", line 621, in processClient results = plugin.preprocess(results, self.log) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 271, in preprocess getdata = self.snmpGetMap.mapdata(getdata) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 320, in mapdata data[self.oidmap[oid]] = value KeyError: '...1...3...6...1...2...1...1...5...0'

and

Problem while executing plugin zenoss.snmp.NewDeviceMap

Traceback (most recent call last): File "/usr/local/zenoss/zenoss/Products/DataCollector/zenmodeler.py", line 621, in processClient results = plugin.preprocess(results, self.log) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 271, in preprocess getdata = self.snmpGetMap.mapdata(getdata) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/CollectorPlugin.py", line 320, in mapdata data[self.oidmap[oid]] = value KeyError: '...1...3...6...1...2...1...1...1...0'

--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65216#65216]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[chas williams]

chas williams [http://community.zenoss.org/people/chasmosaurus] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/65202#65202

--------------------------------------------------------------
that certainly does look odd.  i am using zenos 3.2.1 but not sure it matters.  make sure to use the second version i posted though.  the first version does break table queries/walks.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65202#65202]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jeff Creek]

Jeff Creek [http://community.zenoss.org/people/jcreek] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/65304#65304

--------------------------------------------------------------
Went back to .orig file and tried it again.  Must have fat fingered something.

However, now receiving the following on a few devices

Problem while executing plugin zenoss.snmp.NewDeviceMap

Traceback (most recent call last): File "/usr/local/zenoss/zenoss/Products/DataCollector/zenmodeler.py", line 623, in processClient datamaps = plugin.process(device, results, self.log) File "/usr/local/zenoss/zenoss/Products/DataCollector/plugins/zenoss/snmp/NewDeviceMap.py", line 81, in process if om.snmpOid: AttributeError: ObjectMap instance has no attribute 'snmpOid'

--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65304#65304]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[chas williams]

chas williams [http://community.zenoss.org/people/chasmosaurus] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/65743#65743

--------------------------------------------------------------
i believe this is a slightly different bug in twistedsnmp.py but possibly related (or perhaps concealed by the other bug).  are you doing snmp v3?  if so, there is an initial exchange between the snmp client and server to get the snmp engineid.  the twistedsnmp module is handling this first exchange as if it is valid response to the initial walk used by the snmpclient module to see if there is an active snmp server on the other side of the connection.  when the real snmp resonse is finally handled (the next get from newdevicemap typically the snmpOid or otherwise) this response is popped off the socket and is handled as a KeyError which deletes all the pending callbacks in the current get request, i.e. snmpOid)

here is what an snmp v3 exchange looks like:
19:52:03.896225 IP 192.68.3.12.62633 > 192.68.3.94.snmp:  F=r U= E=  C= GetRequest(14)
19:52:03.896591 IP 192.68.3.94.snmp > 192.68.3.12.62633:  F= U= E= 0x800x000x270xEE0x030x000x060x6A0xFF0xA40xB2 C= Report(32)  .1.3.6.1.6.3.15.1.1.4.0=383
19:52:03.896779 IP 192.68.3.12.62633 > 192.68.3.94.snmp:  F=ar U=nagios E= 0x800x000x270xEE0x030x000x060x6A0xFF0xA40xB2 C= GetNextRequest(21)  .1.3
19:52:03.897097 IP 192.68.3.94.snmp > 192.68.3.12.62633:  F=a U=nagios E= 0x800x000x270xEE0x030x000x060x6A0xFF0xA40xB2 C= Report(31)  .1.3.6.1.6.3.15.1.1.2.0=30


you can see the extra initial exchange in this tcpdump.  i solved this by ignoring the first exchange for snmp v3 sessions.  this isnt the right way to do it i suspect, but i dont see to be able to find a way to get access to the pdu/response information i need (the flag field i gather).
--- /opt/zenoss/python/lib/python2.6/site-packages/pynetsnmp/twistedsnmp.py.001 2012-04-04 17:10:48.174927355 -0400
+++ /opt/zenoss/python/lib/python2.6/site-packages/pynetsnmp/twistedsnmp.py     2012-04-05 15:11:41.147844755 -0400
@@ -151,11 +151,16 @@
         self.defers = {}
         self.session = None
         self.return_dct = False
+       self.engineid_seen = False

     def callback(self, pdu):
         """netsnmp session callback"""
         result = []
         response = netsnmp.getResult(pdu)
+       if self.snmpVersion == "v3":
+           if not self.engineid_seen:
+               self.engineid_seen = True
+               return
         try:
             d = self.defers.pop(pdu.reqid)
         except KeyError:
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65743#65743]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[chas williams]

chas williams [http://community.zenoss.org/people/chasmosaurus] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/65747#65747

--------------------------------------------------------------
grrrrr....  i  keep forgetting that the daemon doesnt recreate the proxyagent object for each query, so the engineid_seen flag needs to be reset when a session is (re)opened.
--- ./python/lib/python2.6/site-packages/pynetsnmp/twistedsnmp.py.001   2012-04-04 17:10:48.174927355 -0400
+++ ./python/lib/python2.6/site-packages/pynetsnmp/twistedsnmp.py       2012-04-05 17:46:14.479189004 -0400
@@ -156,6 +156,10 @@
         """netsnmp session callback"""
         result = []
         response = netsnmp.getResult(pdu)
+       if self.snmpVersion == "v3":
+           if not self.engineid_seen:
+               self.engineid_seen = True
+               return
         try:
             d = self.defers.pop(pdu.reqid)
         except KeyError:
@@ -212,6 +216,7 @@
         return cmdLineArgs

     def open(self):
+       self.engineid_seen = False
         self.session = netsnmp.Session(cmdLineArgs=self._getCmdLineArgs())
         self.session.callback = self.callback
         self.session.timeout = self.timeout_
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65747#65747]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[chas williams]

chas williams [http://community.zenoss.org/people/chasmosaurus] created the discussion

"Re: error with zenoss.snmp.NewDeviceMap on 3"

To view the discussion, visit: http://community.zenoss.org/message/65724#65724

--------------------------------------------------------------
even stranger.  on some of my devices, this initial exchange for snmp v3 isnt exposed to twistedsnmp.  ergo, the following patch instead:

--- /opt/zenoss/python/lib/python2.6/site-packages/pynetsnmp/twistedsnmp.py.001 2012-04-04 17:10:48.174927355 -0400
+++ /opt/zenoss/python/lib/python2.6/site-packages/pynetsnmp/twistedsnmp.py     2012-04-06 10:09:12.625186484 -0400
@@ -156,6 +156,19 @@
         """netsnmp session callback"""
         result = []
         response = netsnmp.getResult(pdu)
+        if self.snmpVersion == "v3":
+            if not self.engineid_exchanged:
+                for oid, value in response:
+                    # for some reason, sometimes the engineid exchange is seen
+                    # at this layer so for the first exchange of this oid
+                    # we will ignore it since it is likely part of the snmp v3
+                    # protocol and not something we requested.  luckily, we
+                    # always do a walk(1,3) first
+                    if asOidStr(oid) == ".1.3.6.1.6.3.15.1.1.2.0":
+                        print "discarding this pdu as engineid exchange", response
+                        self.engineid_exchanged = True
+                        return
+            self.engineid_exchanged = True
         try:
             d = self.defers.pop(pdu.reqid)
         except KeyError:
@@ -212,6 +225,7 @@
         return cmdLineArgs

     def open(self):
+        self.engineid_exchanged = False
         self.session = netsnmp.Session(cmdLineArgs=self._getCmdLineArgs())
         self.session.callback = self.callback
         self.session.timeout = self.timeout_
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65724#65724]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]