Unauthorized: You are not allowed to access 'zProperty' in this context - causing getDeviceCommands() exception

[Gerald Talton]

Gerald Talton [http://community.zenoss.org/people/gtalton] created the discussion

"Re: Unauthorized: You are not allowed to access 'zProperty' in this context - causing getDeviceCommands() exception"

To view the discussion, visit: http://community.zenoss.org/message/73282#73282

--------------------------------------------------------------
I am having this same exact problem .. have you fixed it? Are you still having this problem?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73282#73282]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[chitambira]

chitambira [http://community.zenoss.org/people/chitambira] created the discussion

"Re: Unauthorized: You are not allowed to access 'zProperty' in this context - causing getDeviceCommands() exception"

To view the discussion, visit: http://community.zenoss.org/message/73306#73306

--------------------------------------------------------------
lets see

grep -r "zOracleUser" /opt/zenoss/ZenPacks/
grep -r "zJmxAuthenticante" /opt/zenoss/ZenPacks/

also check threshold calculations and respond with the tales expressions that are being used

Cheers
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73306#73306]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Stephen Hille]

Stephen Hille [http://community.zenoss.org/people/shille] created the discussion

"Re: Unauthorized: You are not allowed to access 'zProperty' in this context - causing getDeviceCommands() exception"

To view the discussion, visit: http://community.zenoss.org/message/73331#73331

--------------------------------------------------------------
OK so I know which Zenpacks use these variables, in fact these two are not the only variables that fail when it starts to fail. There are more templates now using these variables than were in the original Zenpack, and the templates that call on these variables usually work using the context they are currently configured for, one can get the commands using the zendmd method of calling getDeviceCommands() for the devices that eventually fail to collect later on.
Although there were thresholds with complex tales expressions we have generally deleted them all while diagnosing this issue and well before the most recent occurence so that is not too relevant now.


*Sample outputs from the greps suggested:*

/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.RomanTykhonov.OracleDB-2.0-py2.6.egg/ZenPacks/RomanTykhonov/OracleDB/objects/objects.xml:${here/ZenPackManager/packs/ZenPacks/RomanTykhonov/OracleDB/objects/objects.xml:${here/ZenPackManager/packs/ZenPacks.RomanTykhonov.OracleDB/path}/libexec/oracle_status.sh " http://community.zenoss.org/mailto:${device/zOracleUser}/${device/zOraclePass}@${device/zOracleDBInstance ${device/zOracleUser}/${device/zOraclePass}@${device/zOracleDBInstance}" ${device/zOracleHome}

/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.RomanTykhonov.OracleDB-2.0-py2.6.egg/ZenPacks/RomanTykhonov/OracleDB/__init__.py:          ('zOracleUser','sys', 'string'),

/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.5.3-py2.6.egg/ZenPacks/zenoss/ZenJMX/datasources/JMXDataSource.py:    authenticate = '${here/zJmxAuthenticate}'

/usr/local/zenoss/zenoss/ZenPacks/ZenPacks.zenoss.ZenJMX-3.5.3-py2.6.egg/ZenPacks/zenoss/ZenJMX/__init__.py: ('zJmxAuthenticate', False, 'boolean'),



Here is a typical usage of the variables in a JMX datasource.  Note that the variable are typically referred to with "${here/", when is that not appropriate?
Loading Image... Loading Image...
What does not make sense to me is why zenhub is happy to traverse and find an instance of the required variables even if not instantiated at the device level for days on end and then suddenly start having context issues for what seems like any variable which is not.  zJmxAuthenticate for intance is typically set with a useful value at / so is not normally set down at the device level.  Remember all ZenJMX and zencommand data sources that need a zProperty from a higher level than the device get dropped from every collector when this problem starts happening.  I think we may be looking for one config or action that causes everything else to break or could it be a "try:" Python operation which fails to catch the exception eventually such as at line:
File "/usr/local/zenoss/zenoss/Products/ZenHub/services/CommandConfig.py", line 87, in getDeviceCommands
     try:
           threshs = getComponentCommands(dev, cache, cmds, dev.getDmd())
    except (SystemExit, KeyboardInterrupt), ex:
        log.exception("Unable to process device commands for %s -- skipping",
                          dev.id)

I notice that the Oracle templates from the RomanTykhonov Zenpack seem to all refer to their variables as "${device/" and sure enough we tend to fail on those variables like zOracleHome which are often literally the same for all devices so are not instantiated at the device level but instead at the device type level that the template is associated with which is /Server/Oracle as it comes from the Zenpack.

To add insult to injury, the system has been inexplicably stable with regards to this problem for 5 days, the longest period in months, though there have been myriad zenoss restarts in that time due to other needs that might have reset the clock/time bomb whatever that is.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73331#73331]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Stephen Hille]

Stephen Hille [http://community.zenoss.org/people/shille] created the discussion

"Re: Unauthorized: You are not allowed to access 'zProperty' in this context - causing getDeviceCommands() exception"

To view the discussion, visit: http://community.zenoss.org/message/73467#73467

--------------------------------------------------------------
By the way, we had another failure soon after this post and plenty since  also not covinced any more it matters whether the variable is present at the device level as we have failures with those as well.  It is as though the database does not trust zenhub or the collectors any more.   Restart and all is good again.

Any other debug ideas next time we have an event anyone?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73467#73467]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Stephen Hille]

Stephen Hille [http://community.zenoss.org/people/shille] created the discussion

"Re: Unauthorized: You are not allowed to access 'zProperty' in this context - causing getDeviceCommands() exception"

To view the discussion, visit: http://community.zenoss.org/message/73567#73567

--------------------------------------------------------------
During the last failure event after many zencommand and zenjmx data sources had spontaneously dropped from the usual schedule at multiple collectors I was able to recreate the zencommand "ERROR zen.hub: getDeviceCommands() exception for device device_name" stack trace above at will by running from the applicable collector:

zemcommand -v10 -d device_name

with debug output:

2013-06-10 19:41:26,724 DEBUG zen.zencommand: Starting PBDaemon initialization
2013-06-10 19:41:26,731 INFO zen.zencommand: Connecting to central_zenoss.mydomain.com:8789
2013-06-10 19:41:26,732 DEBUG zen.zencommand: Logging in as admin
2013-06-10 19:41:26,836 INFO zen.zencommand: Connected to ZenHub
2013-06-10 19:41:26,836 DEBUG zen.zencommand: Setting up initial services: EventService, CommandConfig
2013-06-10 19:41:26,837 DEBUG zen.zencommand: Chaining getInitialServices with d2
2013-06-10 19:41:26,865 DEBUG zen.zencommand: Loaded service EventService from zenhub
2013-06-10 19:41:26,865 DEBUG zen.zencommand: Loaded service CommandConfig from zenhub
2013-06-10 19:41:26,865 DEBUG zen.zencommand: Queueing event {'severity': 0, 'component': 'zencommand', 'agent': 'zencommand', 'summary': 'started', 'manager': 'collector_hostname.corporate.transgrid.local', 'device': 'collector_hostname', 'eventClass': '/App/Start', 'monitor': 'collector_hostname'}
2013-06-10 19:41:26,865 DEBUG zen.zencommand: Total of 1 queued events
2013-06-10 19:41:26,866 DEBUG zen.zencommand: pushEventsLoop : after rrdstatsGauge()
2013-06-10 19:41:26,866 DEBUG zen.zencommand: pushEventsLoop : currently sending time = 1370857286.866498 curTimeThreshold = 1370857346.865970
2013-06-10 19:41:26,866 DEBUG zen.zencommand: Calling connected.
2013-06-10 19:41:26,866 DEBUG zen.zencommand: Fetching configuration from zenhub
2013-06-10 19:41:26,920 DEBUG zen.zencommand: Updated configCycleInterval config to 360
2013-06-10 19:41:27,040 DEBUG zen.zencommand: Loading classes ['Products.ZenModel.MinMaxThreshold', 'ZenPacks.community.PointThreshold.thresholds.PointThreshold', 'ZenPacks.community.deviceAdvDetail.thresholds.StatusThreshold']
2013-06-10 19:41:27,112 DEBUG zen.thresholds: Updating threshold ('high event queue', ('collector_hostname', ''))
2013-06-10 19:41:27,112 DEBUG zen.thresholds: Updating threshold ('zenmodeler cycle time', ('collector_hostname', ''))
2013-06-10 19:41:27,112 DEBUG zen.thresholds: Updating threshold ('zenperfsnmp cycle time', ('collector_hostname', ''))
2013-06-10 19:41:27,112 DEBUG zen.thresholds: Updating threshold ('zenping cycle time', ('collector_hostname', ''))
2013-06-10 19:41:27,113 DEBUG zen.thresholds: Updating threshold ('zenprocess cycle time', ('collector_hostname', ''))
2013-06-10 19:41:27,152 DEBUG zen.zencommand: Sent a 'stop' event
2013-06-10 19:41:27,153 DEBUG zen.zencommand: stop() called when not running
2013-06-10 19:41:27,153 INFO zen.zencommand: Daemon zencommand shutting down
2013-06-10 19:41:27,153 DEBUG zen.zencommand: Removing service EventService
2013-06-10 19:41:27,153 DEBUG zen.zencommand: Removing service CommandConfig
2013-06-10 19:41:27,154 DEBUG zen.zencommand: Finished config fetch
2013-06-10 19:41:27,154 DEBUG zen.zencommand: stop() called when not running


The collector would evidently trigger the same sequence and stop short from zencommand's point of view at "Sent a 'stop' event"
where nomally when it is working we would get a list of thresholds related to the device_name and then the command with zProperties in the command expanded.

At the same time though I could execute the following zendmd script without any error from the central zenoss server:


from Products.ZenHub.services.CommandConfig import getDeviceCommands
cmds = getDeviceCommands(find("device_name"))
for c in cmds.commands:                            
   print c.command

And a nice list of the expected commands for device_name would output.

This tells me zenhub was behaving differenty for a locally connected client than a remote collector zencommand client.  Again, restarting zenhub was all that was required to get the remote zencommand to be able to fetch commands again.  Had to restart zencommand and zenjmx at the collectors to trigger them to load their complete schedule of commands/data sources after that and get to full operation though.

Why would zenhub start doing this after many hours of happy operation?  What else could I look for when the failure at zenhub is occuring to isolate the cause of this issue?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73567#73567]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]