Zenoss daemons crashing every hour

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Zenoss daemons crashing every hour"

To view the discussion, visit: http://community.zenoss.org/message/74524#74524

--------------------------------------------------------------
Zenoss 4.2.3 CentOS 5.4

In the last week the zenoss services (daemons) are crashing every hour (exactly, every hour! x:00), each time for 2-3 minutes.
I didn't change anything in zenoss settings, due to those crashes all our graphs are incomplete and fragmented.

I checked each daemon logs but couldn't find any indication for errors.

This daemons / processes crashing:
zenprocess, zenjmx, zenperfsnmp, zeneventlog, zenmodeler, zencommand, zenwinperf, zeneventd, zenhub, zenactiond, zenwin, zentrap, zenstatus, zensyslog, zenping

This is very critical for us, and I would really appreciate any troubleshooting suggests,
Is there any additional logs to check? specific log to enable debug?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74524#74524]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[hydruid]

hydruid [http://community.zenoss.org/people/hydruid] created the discussion

"Re: Zenoss daemons crashing every hour"

To view the discussion, visit: http://community.zenoss.org/message/74512#74512

--------------------------------------------------------------
How many devices are you monitoring and what do the resources look like on your zenoss server?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74512#74512]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss daemons crashing every hour"

To view the discussion, visit: http://community.zenoss.org/message/74525#74525

--------------------------------------------------------------
hey

187 devices
8GB RAM
2 CPUs
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74525#74525]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[hydruid]

hydruid [http://community.zenoss.org/people/hydruid] created the discussion

"Re: Zenoss daemons crashing every hour"

To view the discussion, visit: http://community.zenoss.org/message/74514#74514

--------------------------------------------------------------
run the "top" command and look to see if your CPU's are maxed out or not...
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74514#74514]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss daemons crashing every hour"

To view the discussion, visit: http://community.zenoss.org/message/74526#74526

--------------------------------------------------------------
hey http://community.zenoss.org/people/hydruid hydruid

already checked it, there's no high CPU or memory utilization.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74526#74526]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[hydruid]

hydruid [http://community.zenoss.org/people/hydruid] created the discussion

"Re: Zenoss daemons crashing every hour"

To view the discussion, visit: http://community.zenoss.org/message/74515#74515

--------------------------------------------------------------
check /var/log/messages or /var/log/syslog to see if there are any hints as
to what happened...make sure to look up from when it shows the zenoss
daemons died to see if anything else happened to cause it!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74515#74515]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[vitaly_il]

vitaly_il [http://community.zenoss.org/people/vitaly_il] created the discussion

"Re: Zenoss daemons crashing every hour"

To view the discussion, visit: http://community.zenoss.org/message/74529#74529

--------------------------------------------------------------
From Zenoss components logs it seems that they have problems with MySQL and Rabbitmq from time to time. Is there something interesting into Rabbitmq logs?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74529#74529]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[James M]

James M [http://community.zenoss.org/people/James] created the discussion

"Re: Zenoss daemons crashing every hour"

To view the discussion, visit: http://community.zenoss.org/message/74548#74548

--------------------------------------------------------------
Thanks for all your help guys!

I found out there was low disk space in the root partition that caused by the RabbitMQ temporary dump files (in /tmp)
In the RabbitMQ logs i found:

=INFO REPORT==== 1-Sep-2013::08:14:06 ===
Disk free limit set to 1000MB

=INFO REPORT==== 1-Sep-2013::08:14:06 ===
Disk free space insufficient. Free bytes:963620864 Limit:1000000000

There were no alerts on disk space (because it was momentary each time)

Once I increased the disk space in the root partition (/)  it seems to resolve the crashes.

Thanks again.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74548#74548]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[hydruid]

hydruid [http://community.zenoss.org/people/hydruid] created the discussion

"Re: Zenoss daemons crashing every hour"

To view the discussion, visit: http://community.zenoss.org/message/74542#74542

--------------------------------------------------------------
Glad to hear you got it resolved!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74542#74542]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]