4.2.3 No Graph Data and Reindex() fails

[Johnathan Bravo]

Johnathan Bravo [http://community.zenoss.org/people/MRJB] created the discussion

"4.2.3 No Graph Data and Reindex() fails"

To view the discussion, visit: http://community.zenoss.org/message/71076#71076

--------------------------------------------------------------
as the subject says. I recent ran through the 4.2.3 upgrade(?) from 4.2. sp1 on cent 6.3 . Now I'm unable to see any graph data on my servers. All return with NAN. Being a long itme Zenoss user I figured maybe a re-index would help but when I try to run it I get the following error.

reindex()

  File "<console>", line 1, in <module>
  File "/opt/zenoss/Products/ZenModel/zendmd.py", line 146, in reindex
    commit()
  File "/opt/zenoss/lib/python/transaction/_manager.py", line 89, in commit
    return self.get().commit()
  File "/opt/zenoss/lib/python/transaction/_transaction.py", line 342, in commit
    reraise(t, v, tb)
  File "/opt/zenoss/lib/python/transaction/_transaction.py", line 333, in commit
    self._commitResources()
  File "/opt/zenoss/lib/python/transaction/_transaction.py", line 473, in _commitResources
    reraise(t, v, tb)
  File "/opt/zenoss/lib/python/transaction/_transaction.py", line 450, in _commitResources
    rm.tpc_vote(self)
  File "/opt/zenoss/lib/python/ZODB/Connection.py", line 781, in tpc_vote
    s = vote(transaction)
  File "/opt/zenoss/lib/python/relstorage/storage.py", line 794, in tpc_vote
    return self._vote()
  File "/opt/zenoss/lib/python/relstorage/storage.py", line 839, in _vote
    oid=oid, serials=(actual, expect))
ReadConflictError: database read conflict error (oid 0x045a3e, serial this txn started with 0x039c077ea2729ecc 2013-01-18 14:22:38.073688, serial currently committed 0x039c07fed814e5aa 2013-01-18 16:30:50.644132)

This install is fairly small. Only about 20 servers and 60 or so ping monitored clients so if I'm borked I can rebuild in a day or so but would rather not.

Thanks for any assistance.

Mr Johnathan Bravo
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71076#71076]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Philip Warren]

Philip Warren [http://community.zenoss.org/people/pwarren] created the discussion

"Re: 4.2.3 No Graph Data and Reindex() fails"

To view the discussion, visit: http://community.zenoss.org/message/71077#71077

--------------------------------------------------------------
In some cases I have seen these ReadConflictErrors crop up with stale memcached entries. Can you try restarting memcached (if it is running) and see if this helps?

-- Philip
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71077#71077]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Johnathan Bravo]

Johnathan Bravo [http://community.zenoss.org/people/MRJB] created the discussion

"Re: 4.2.3 No Graph Data and Reindex() fails"

To view the discussion, visit: http://community.zenoss.org/message/71097#71097

--------------------------------------------------------------
zencatalog run --createcatalog --forceindexPhilip
   Thanks for your response. I have restarted memcached. However before I did that it did the following.

zencatalog run --createcatalog --forceindex

That took a little bit but it came back and then I went ahead and did the reindex from zendmd, probably a moot point after the zencatalog but I did it anyway and it went through without error. After I did the zencatalog I saw your post and restarted memcached just for good measure.


This restored the wmi graphs, however the snmp graphs are still showing NAN. Or maybe that's vice versa , I'm having a hard time distinguishing between WMI graphs and SNMP graphs since they both say things like CPU percent utilization. One of them works and the other doesn't

MrJohnBravo
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71097#71097]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Johnathan Bravo]

Johnathan Bravo [http://community.zenoss.org/people/MRJB] created the discussion

"Re: 4.2.3 No Graph Data and Reindex() fails"

To view the discussion, visit: http://community.zenoss.org/message/71105#71105

--------------------------------------------------------------
I spoke too soon. I am still seeing very spotty graphing . The above fix lasted about 8hours and then Zenoss stopped graphing again. I am still not seeing any graph data from any server via snmp even though an snmpwalk on the specific oid's listed in the monitoring template are returning values. If anyone has a pointer to where I can look to find the error/ problem I'd appreciate it.

Thanks

MrJohnBravo
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71105#71105]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[joanypony]

joanypony [http://community.zenoss.org/people/joanypony] created the discussion

"Re: 4.2.3 No Graph Data and Reindex() fails"

To view the discussion, visit: http://community.zenoss.org/message/71128#71128

--------------------------------------------------------------
A long shot, but are your SNMP collection settings the same on your upgrade as they were on the old install? eg, maxoidperrequest, snmptries etc..

Joan
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71128#71128]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Johnathan Bravo]

Johnathan Bravo [http://community.zenoss.org/people/MRJB] created the discussion

"Re: 4.2.3 No Graph Data and Reindex() fails"

To view the discussion, visit: http://community.zenoss.org/message/71141#71141

--------------------------------------------------------------
I just double checked. Only thing different was ZsnmpTimeout which was 1 on the new install and 2.5 on the old. Thanks for the long shot though.

WMI graphs seem to be working ok but I'd prefer not to rely on them since WMI is so flaky and drops out . Does anyone know of a log I can look at that might give some insight on this or where the RRD graph data is stored? Is it in the database or file ?

Mr John Bravo
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71141#71141]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jcurry]

jcurry [http://community.zenoss.org/people/jcurry] created the discussion

"Re: 4.2.3 No Graph Data and Reindex() fails"

To view the discussion, visit: http://community.zenoss.org/message/71135#71135

--------------------------------------------------------------
Is your system very busy?  Try using top to see if memory / cpu are maxed out.  This is a known cause of "spotty" graphs.
Cheers,
Jane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71135#71135]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Johnathan Bravo]

Johnathan Bravo [http://community.zenoss.org/people/MRJB] created the discussion

"Re: 4.2.3 No Graph Data and Reindex() fails"

To view the discussion, visit: http://community.zenoss.org/message/71144#71144

--------------------------------------------------------------
I'm going to give that VM a few more g ram to see if that clears up the spotty WMI graphing. Thanks for the tip.

Post by jcurry
Is your system very busy?  Try using top to see if memory / cpu are maxed out.  This is a known cause of "spotty" graphs.
Cheers,
Jane

--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71144#71144]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: 4.2.3 No Graph Data and Reindex() fails"

To view the discussion, visit: http://community.zenoss.org/message/71147#71147

--------------------------------------------------------------
Also, in 4.2.x, some things that might have been WMI in 3.x are now perfmon based - i.e. if you see zenwinperf - that's not WMI, but perfmon.

--
James Pulver
ZCA Member
LEPP Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71147#71147]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Johnathan Bravo]

Johnathan Bravo [http://community.zenoss.org/people/MRJB] created the discussion

"Re: 4.2.3 No Graph Data and Reindex() fails"

To view the discussion, visit: http://community.zenoss.org/message/71143#71143

--------------------------------------------------------------
Ok. I think I've got this sorted out. Sometimes it's the simple stuff that you overlook. Thanks to joan for pointing out that I should look at the old install and check differences.

In my old install I had been using the Nova.Windows.SNMPPerfMonitor. I've never used snmpinformant. Never had the need with that Zenpack installed . After looking at the OID's currently being used on the new install in Monitoring Templates -> Device /Server/Windows to get cpu info ( i had been trying my snmpwalk with 1.3.6.1.2.1.25.3.3.1.2 which are the ones exposed by windows OS) and finding that Zenoss Core uses 1.3.6.1.4.1.9600.1.1.5.1.5.6.95.84.111.116.97.108 ( for those of you keeping track anything past .9600 is what's exposed by SNMPInformant) I realized my mistake.  So without SNMPInformant installed on the clients I'm not going to get any RRD files because the OID being used isn't returning data.

Hopefully that made some sense and anyone looking into rrd graph issues will find some help in this thread. In addition I also learned that zenperfsnmp.log is your friend and is located in /opt/zenoss/log . Had I tailed that first I might have saved myself from myself because it list's the OID return errors well and concisely.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71143#71143]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]