Jo Rhett
2012-08-27 19:01:12 UTC
Jo Rhett [http://community.zenoss.org/people/jorhett] created the discussion
"Help identify global locks in Zenoss which could prevent tests?"
To view the discussion, visit: http://community.zenoss.org/message/68180#68180
--------------------------------------------------------------
As described in this discussion: http://community.zenoss.org/thread/18241?tstart=0 http://community.zenoss.org/thread/18241?tstart=0 we are seeing periods every 30 hours or thereabouts (not precise nor exactly consistent) that all tests fail for anywhere from 30 seconds to 2 minutes. During the interval, every test returns "timed out". Every type of test fails, including snmp, local, direct query, custom-built -- you name it. Everything fails during that period.
Zenoss 3.2.1 on CentOS 5 x86_64
During the failures, every test returns a timeout. Even tests to localhost, and to services which always return success. Then the problem disappears and all tests return success.
The zenoss user is not limited for core size or number of processes at an OS level. No complaints are made in system logs of any resource issues. Neither zenoss nor system utilities like sar show any resource utilitization problem. IO is not higher than normal during the period, in fact it drops considerably.
This points at a global lock somewhere inside of Zenoss which prevents tests from running. Can someone identify what this lock might be? Where can I look, or what debugging can I enable to determine this?
Debugging of the events doesn't help. It shows the timeouts returning in great detail. The only information we learned from running the hub at higher debug was evident proof that the checks weren't actually being executed, as some of the services it claimed it couldn't talk to were local to itself.
--------------------------------------------------------------
Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68180#68180]
Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
"Help identify global locks in Zenoss which could prevent tests?"
To view the discussion, visit: http://community.zenoss.org/message/68180#68180
--------------------------------------------------------------
As described in this discussion: http://community.zenoss.org/thread/18241?tstart=0 http://community.zenoss.org/thread/18241?tstart=0 we are seeing periods every 30 hours or thereabouts (not precise nor exactly consistent) that all tests fail for anywhere from 30 seconds to 2 minutes. During the interval, every test returns "timed out". Every type of test fails, including snmp, local, direct query, custom-built -- you name it. Everything fails during that period.
Zenoss 3.2.1 on CentOS 5 x86_64
During the failures, every test returns a timeout. Even tests to localhost, and to services which always return success. Then the problem disappears and all tests return success.
The zenoss user is not limited for core size or number of processes at an OS level. No complaints are made in system logs of any resource issues. Neither zenoss nor system utilities like sar show any resource utilitization problem. IO is not higher than normal during the period, in fact it drops considerably.
This points at a global lock somewhere inside of Zenoss which prevents tests from running. Can someone identify what this lock might be? Where can I look, or what debugging can I enable to determine this?
Debugging of the events doesn't help. It shows the timeouts returning in great detail. The only information we learned from running the hub at higher debug was evident proof that the checks weren't actually being executed, as some of the services it claimed it couldn't talk to were local to itself.
--------------------------------------------------------------
Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68180#68180]
Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]