Zenoss stopped executing commands

Discussion:

Marcin Zajączkowski

2012-03-16 13:41:35 UTC

Marcin ZajÄczkowski [http://community.zenoss.org/people/Szpak] created the discussion

"Zenoss stopped executing commands"

To view the discussion, visit: http://community.zenoss.org/message/65249#65249

--------------------------------------------------------------
Hi,

Our Zenoss instance stopped update stats at some point. It is a test environment and it was detected some time later, so it is not sure what causes that situation (never the less some administration work was doing on a server at that time). I dig in the logs and it seems that the execution of commands was stopped. Things not being a command (like zenjmx measures) seems to work anyway.

In zencommand.log there is:

15:21:45,264 INFO zen.CmdClient: command client finished collection for host1
15:21:46,314 INFO zen.CmdClient: command client finished collection for host2
15:21:59,028 WARNING zen.zencommand: Command timed out on device host3: '/usr/sbin/rabbitmqctl -q -n ***@host3 status 2>&1'
15:22:00,177 WARNING zen.zencommand: Command timed out on device host4: '/usr/sbin/rabbitmqctl -q -n ***@host4 status 2>&1'
15:22:30,707 INFO zen.CmdClient: command client finished collection for host2
15:22:41,378 INFO zen.zencommand: config: <Products.ZenRRD.zencommand.DeviceConfig instance at 0xb50090c>
15:28:14,537 INFO zen.zencommand: config: <Products.ZenRRD.zencommand.DeviceConfig instance at 0xb383acc>
15:28:18,037 INFO zen.zencommand: config: <Products.ZenRRD.zencommand.DeviceConfig instance at 0xb4cb68c>
(... 7 more)

The commands are called for a while (a few to several minutes) after manual restart or at midnight and at noon. Every time the last few lines are:
2012-03-16 13:41:10,479 INFO zen.zencommand: config: <Products.ZenRRD.zencommand.DeviceConfig instance at 0xbd7becc>

The process is still alive, so maybe it hangs on some infinite call? Unfortunately I don't know how to check it.

zenoss-3.2.0-1258.el5 on CentOS downloaded from zenoss.org.

Do you have any suggestions what could cause that behavior?

Regards
Marcin

P.S. There is also in zenmodeler.log occurring fro mtime to time error:
2012-03-15 11:59:50,234 WARNING zen.PythonClient: Error in zenoss.nmap.IpServiceMap: got stderr: 'Upon execvpe /opt/zenoss/libexec/nmap [\'/opt/zenoss/libexec/nmap\', \'-p\', \'1-1024;-sT;--open;-oG\', \'-\'
] in environment id 177177980\n:Traceback (most recent call last):\nÂ File "/opt/zenoss/lib/python/twisted/internet/process.py", line 394, in _fork\nÂ Â Â executable, args, environment)\nÂ File "/opt/zenoss/li
b/python/twisted/internet/process.py", line 440, in _execChild\nÂ Â Â os.execvpe(executable, args, environment)\nÂ File "/opt/zenoss/lib/python2.6/os.py", line 353, in execvpe\nÂ Â Â _execvpe(file, args, env)\n
File "/opt/zenoss/lib/python2.6/os.py", line 368, in _execvpe\nÂ Â Â func(file, *argrest)\nOSError: [Errno 2] No such file or directory\n'

but it started to show long before the problem with measurements.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65249#65249]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

Shane Scott

2012-03-16 15:22:02 UTC

Permalink

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: Zenoss stopped executing commands"

To view the discussion, visit: http://community.zenoss.org/message/65255#65255

--------------------------------------------------------------
Marcin:

To be honest it sounds like part of your installation has gone missing. You might have disk corruption. What I would do is install the same version of zenoss in a vm and copy the missing files back in. As for zencommand failing to colelct, it could be due to mising components or state pickles. ZenOSS hates state pickles. Check $ZENHOME/perf/ for associated .pickle files and rename them. Stop and start zencommand to initiate a fresh config fetch. Don't use zencommand restart as it might not really restart zencommand. Be sure zencommand has stopped and has no processes running before you start it back up.

Best,
--Shane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65255#65255]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

Marcin Zajączkowski

2012-03-16 16:26:58 UTC

Permalink

Marcin ZajÄczkowski [http://community.zenoss.org/people/Szpak] created the discussion

"Re: Zenoss stopped executing commands"

To view the discussion, visit: http://community.zenoss.org/message/65284#65284

--------------------------------------------------------------
Thanks for your reply, Shane!

I did "rpm -V zenoss" and in fact it seems there are some files missing. I will try to reinstall Zenoss.
I haven't found any .pickle files in mentioned directory.

Regards
Marcin
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65284#65284]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

Shane Scott

2012-03-16 18:21:51 UTC

Permalink