Zenoss 4.2 -> zenwinperf crashing

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70020#70020

--------------------------------------------------------------
Hey
from some reason the zenwinperf process is crashing 1-2 a day.
I know I can configure a watchdog for the process but I still would like to know how to troubleshoot it.
All other processes are running as expected and the problem is only with zenwinperf, any ideas?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70020#70020]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Ryan Matte]

Ryan Matte [http://community.zenoss.org/people/rmatte] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70113#70113

--------------------------------------------------------------
Have you installed SP1?  http://bit.ly/PnH8T4 http://bit.ly/PnH8T4

There is a memory leak with PySamba that could account for that.  SP1 patches it.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70113#70113]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70135#70135

--------------------------------------------------------------
I didn't install SP1.
I'll install it and will update if it solved the problem or not.

Thanks!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70135#70135]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70194#70194

--------------------------------------------------------------
Hey

I have installed SP1

It's important to mention that the main reason I installed SP1 is the fix for removing interfaces / components from a device / the problem to unmonitor wmi device, see http://jira.zenoss.com/jira/browse/ZEN-2966 http://jira.zenoss.com/jira/browse/ZEN-2966 which was fixed in SP1 (verified).
And also the memory leak, I installed Zenoss using the auto install script and the memory consumption is very high, also should be fixed in SP1.
The memory is still very high.

*The new situation is even worse:* Zenoss processes are crashing very often (2-3 times in an hour), and especially zeneventd & zenactiond which is *crashing ALL the time*, I set those processes to run under watchdog but it keeps crashing, making it almost impossible to work with.

In the heartbeat event view it looks like zeneventd & zenactiond are always down..
Loading Image... Loading Image...

*Log files attached:* zeneventd, zenactiond, aenjobs, zenwinperf

In addition to the log files, The next screenshots are the zenoss server performance, from the device monitoring (localhost) & the collector performace.

*Zenoss server:*

CentOS 6.3
4GB RAM
2 CPU
Zenoss 4.2 SP1

*Zenoss server performance:*
Loading Image... Loading Image...
Loading Image... Loading Image...
Loading Image... Loading Image...
Loading Image... Loading Image...
Loading Image... Loading Image...
Loading Image... Loading Image...



--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70194#70194]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70238#70238

--------------------------------------------------------------
OK
There is defiantly memory leak. changed the server to 8GB RAM and it gone 90+ %used in no time....
(And I'm guessing that's what causing the services to crash)

It looks like it's much worse after SP1.

Can anyone direct me to a manual fix to this problem?
It looks like the bug is still open ( http://jira.zenoss.com/jira/browse/ZEN-3193 http://jira.zenoss.com/jira/browse/ZEN-3193)

but I need to fix that as soon as possible because this way Zenoss isn't stable at all...

Any suggestions?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70238#70238]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Ryan Matte]

Ryan Matte [http://community.zenoss.org/people/rmatte] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70241#70241

--------------------------------------------------------------
The memory leak is in PySamba, and Egor Puzanov developed a patch for it which is included in SP1.  If you installed SP1 and had the PySamba ZenPack installed at the time, it doesn't make any sense that you're still seeing the issue.  If you installed SP1 before installing the PySamba pack it would not have applied the patch for it.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70241#70241]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70239#70239

--------------------------------------------------------------
Hey Ryan.

I installed zenoss using the auto install script.
And I did install SP1.

ZenPack ZenPacks.zenoss.PySamba is installed (was installed by the auto install script)

Honestly.. I'm not sure how to troubleshoot it further, as I said, I increased the RAM to 8GB and it didn't help..

Is there any log I can upload? can you direct me what to check?

Thanks..
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70239#70239]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70366#70366

--------------------------------------------------------------
Any suggestions??
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70366#70366]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[C James]

C James [http://community.zenoss.org/people/Smiler] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70370#70370

--------------------------------------------------------------
Hi James M,

I was having this same issue on my install even following my install of SP1 for Zenoss,  I enabled watchdog yesterday and this has resolved the issue for me.  Not the solution your looking for I know but the only other thing I could suggest is lowering the amount of collections it can do at any one time - in the config I think the default is 500 for zenwinperf ..  I didn't try this myself as I am only testing and monitoring about 5 Windows devices as I try to sell my team on the benefits of Zenoss but I agree it was annoying.

My install - Centos 6.3 x64 base install
Zenoss installed by autoinstall.sh
Zenoss SP1 patch applied..

Biggest remaining issue for me - Network map only shows a grey bar and nothing else regardless of browser or object start device :(

I hope this helps
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70370#70370]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70423#70423

--------------------------------------------------------------
Hey C James.
The problem is that after installing SP1 all the services crashs a lot..
I decided to install Zenoss on another server using the VMware apliance.
I had some problems with the zenhub service, solved it by following:
http://community.zenoss.org/message/68815#68815#68815 http://community.zenoss.org/message/68815#68815

And again with Zenwinperf, this time after SP1 and without no memory leak.
So I enabled watchdog and set the amount of collections from 500 to 250.

I haven't add all my devices yet, only something like 30 devices (not all win) And I'm waiting a few days to see if there are no stability issues again, if there will be I'll probablly fallback to CORE 3.

Will update you guys on the progress..

Thanks for your help!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70423#70423]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Doug C]

Doug C [http://community.zenoss.org/people/thedukeness] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70435#70435

--------------------------------------------------------------
I was able to fix mine by adding
watchdog True

to /opt/zenoss/etc/zenwinperf.conf
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70435#70435]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[reighnman]

reighnman [http://community.zenoss.org/people/reighnman] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70564#70564

--------------------------------------------------------------
What does adding watchdog to the config change?

winperf still crashing for me on SP1, usually every day or two.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70564#70564]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70609#70609

--------------------------------------------------------------
After
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70609#70609]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Doug C]

Doug C [http://community.zenoss.org/people/thedukeness] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70637#70637

--------------------------------------------------------------
If you enable watchdog, it will restart zenwinperf every time it dies.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70637#70637]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/70623#70623

--------------------------------------------------------------
Hey Doug,
as I wrote in my previous post watchdog enabled. and it restarting the service every time it dies.
But as you can see in the attached image we're losing data because the winperf service crashing a lot and the graphs looks really bad.. :(
btw - cpu & memory consumption of the server is normal so I dfon't think the winperf crashes related to resources issue..

I'll appreciate any suggestion..

Thanks
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/70623#70623]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss 4.2 -> zenwinperf crashing"

To view the discussion, visit: http://community.zenoss.org/message/71191#71191

--------------------------------------------------------------
*Upgrading to 4.2.3 fixed that.*

But created another problem:
Problem with monitoring localhost: http://community.zenoss.org/thread/19072
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71191#71191]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]