Zenoss 4.2.4 not graphing new data

[Jared Ready]

Jared Ready [http://community.zenoss.org/people/residentbiscuit] created the discussion

"Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74097#74097

--------------------------------------------------------------
As the title states, I cannot get Zenoss to graph any new performance data. It is graphing historical data up until last Thursday, and about 10 minutes of Monday for some reason. The data was all copied over from a 3.1 install.

In zenactiond.log, I have one error that seems relevant that is constantly getting generated.
http://fpaste.org/27197/45863961/ http://fpaste.org/27197/45863961/

2013-07-23 08:30:19,747 INFO zen.maintenance: Performing periodic maintenance
2013-07-23 08:31:19,729 ERROR zen.Schedule: Error processing maintenance windows - will try again in 60econds
Traceback (most recent call last):
  File "/opt/zenoss/Products/ZenEvents/Schedule.py", line 51, in configCycle
    self.run()
  File "/opt/zenoss/Products/ZenEvents/Schedule.py", line 95, in run
    self.runEvents()
  File "/opt/zenoss/Products/ZenEvents/Schedule.py", line 148, in runEvents
    devices = mw.target().getId()
  File "/opt/zenoss/Products/ZenModel/MaintenanceWindow.py", line 369, in target
    return self.productionState().primaryAq()
AttributeError: productionState

In the zenmodeler log, I have some errors as well. I'll just provide the fpaste as it's a pretty rough copy: http://fpaste.org/27198/45864761/ http://fpaste.org/27198/45864761/

Any assistance would be greatly appreciated. This is the last thing holding me back from making this instance our production server.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74097#74097]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jared Ready]

Jared Ready [http://community.zenoss.org/people/residentbiscuit] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74101#74101

--------------------------------------------------------------
Doing some more research, the only data that is currently graphed is data that copied over from my other install. So, I don't think it's an issue with the graphing, but an issue with the polling of data. I have tested several data sources on devices and I am still getting values, so I'm not sure what the issue is or how to proceed in diagnosing.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74101#74101]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74115#74115

--------------------------------------------------------------
Well, the obvious place to start is - do you get any events on the devices that aren't graphing new data such as SNMP Agent Down?

   * as you copied over RRD files, make sure you didn't break filesystem permissions to  $ZENHOME/perf/Devices for your non-graphers.

   * Check log files. If they don't show anything, change the daemons to use debug logging for a little while and restart zenoss - see if anything is useful there...

--
James Pulver
ZCA Member
CLASSE Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74115#74115]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jared Ready]

Jared Ready [http://community.zenoss.org/people/residentbiscuit] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74107#74107

--------------------------------------------------------------
SNMP is up and running on the devices. I can snmpwalk with no issues.
Permissions are good. I double checked.
Logs don't show anything useful. Been running zenperfsnmp with -v10 and there are no errors. There is also no indication of data being gathered or RRD being written to. Like it's not even trying.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74107#74107]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jared Ready]

Jared Ready [http://community.zenoss.org/people/residentbiscuit] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74159#74159

--------------------------------------------------------------
After furthing troubleshooting, I am pretty certain it is zenoss not gathering remote data. RRD files and graphs are working for $ZENHOME/perf/Daemons, but nothing for Devices. I don't believe it is an issue with rrdcached, as I have spent awhile looking at that. I'm kind of stuck now though,
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74159#74159]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74150#74150

--------------------------------------------------------------
Well, I'm stumped also. I've seen graphs stop when it can't talk to zenhub, but that shows up in the logs. So what *does* zenperfsnmp debug show? pastebin? Does it show it talking to any of the problem devices?

What production state are your devices in?

Heh, I missed the pastebined log. No event service - is rabbitmq running?

--
James Pulver
ZCA Member
CLASSE Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74150#74150]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jared Ready]

Jared Ready [http://community.zenoss.org/people/residentbiscuit] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74160#74160

--------------------------------------------------------------
Devices are mostly in Production state. After digging around more yesterday I noticed that the OSProcesses monitoring template is graphing new data. On servers that have the the OS Processes component I can click on that and for each process (most just seem to be running httpd there) and see new data being graphed.

Running ps aux | grep rabbitmq does show several results. /usr/sbin/rabbitmq-server is running, I don't see anything from zenoss for rabbitmq, though. Not sure if that's normal. I do see that No event service error pop up in several logs.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74160#74160]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74151#74151

--------------------------------------------------------------
Are you getting any new events? Maybe try re-creating rabbit MQ queues.
http://community.zenoss.org/thread/19057 http://community.zenoss.org/thread/19057

--
James Pulver
ZCA Member
CLASSE Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74151#74151]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jared Ready]

Jared Ready [http://community.zenoss.org/people/residentbiscuit] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74161#74161

--------------------------------------------------------------
I am getting events still, I'll go ahead and rebuild the queues though. Who knows, might do something. I'm getting desperate at this point :(
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74161#74161]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jared Ready]

Jared Ready [http://community.zenoss.org/people/residentbiscuit] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74166#74166

--------------------------------------------------------------
I seem to have the issue is with the RRD files that were copied over. After deleting some rrd's I noticed zenperfsnmp was now gathering data for those devices. I have no wiped that entire directory and now am remodeling all devices to rebuild it and recreate all the rrd's. I will look into why the copy of old data did not work and update.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74166#74166]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jmp242]

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74171#74171

--------------------------------------------------------------
Could it be that you went from 32 bit OS to 64bit OS? That doesn't work without converting the RRDs. Also you may have an incompatible graph definition, which would require Zenoss creating new RRDs.

--
James Pulver
ZCA Member
CLASSE Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74171#74171]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jared Ready]

Jared Ready [http://community.zenoss.org/people/residentbiscuit] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74167#74167

--------------------------------------------------------------
It went from 64 bit OS to a 64 bit OS, so that shouldn't be an issue. Even made sure RRD tools and the processor architecture were both 64 bit, and they were. I'm not sure what you mean by incompatible graph definitions. This 4.2 server got all of it's data and templates directly from the 3.1 server during the upgrade.

What I did to upgrade was install zenoss 3.2 on a fresh RHEL6 server (Using hackman's RPM).
I then copied over Data.fs, the $ZENHOME/etc directory, and all plugins we use (ldap mainly), and made sure zenpacks were all over.
Then I upgraded up to 4.2.3, then to 4.2.4 once it came out

So graph definitions shouldn't be any different.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74167#74167]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[rgartley]

rgartley [http://community.zenoss.org/people/rgartley] created the discussion

"Re: Zenoss 4.2.4 not graphing new data"

To view the discussion, visit: http://community.zenoss.org/message/74282#74282

--------------------------------------------------------------
I had this same issue, but only after I noticed that zenhub wasn't working after the upgrade from 4.2.3 to 4.2.4. To fix my zenhub issue, I followed the information in this post....

http://community.zenoss.org/message/73697#73697 http://community.zenoss.org/message/73697#73697

... and that got me thinking that's probably what's happening with my graphing data as well. I followed the same procedure with the $ZENHOME/etc/zenrender.conf file. After modifying and restarting Zensoss, all graphs are working great now.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74282#74282]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]