Event Escalation based on Time

[Chaz Vidal]

Chaz Vidal [http://community.zenoss.org/people/chazzvid] created the discussion

"Event Escalation based on Time"

To view the discussion, visit: http://community.zenoss.org/message/69771#69771

--------------------------------------------------------------
Hi All,
I want to escalate an event if it gets thrown up more than once in a 5 minute interval.  e.g. the first event isn't necessarily a problem but a second event within 5 minutes would be and escalate accordingly.

I have played around with the event transform to escalate an event based on event counts ( https://gist.github.com/1486570 https://gist.github.com/1486570) and this seems to work for the most part.

However, because the first event sometimes happens overnight, we get false alarm when a second one appears sometime later and the first event hasn't aged yet.  I don't want to put event aging too quickly.  I tried creating an event transform that would be based on the firstTime attribute of an event but it looks like I can't reference this attribute in the transform.

Any pointers appreciated, thanks!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69771#69771]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Floyd Strimling]

This post might be inappropriate. Click to display it.

[Doug Syer]

Doug Syer [http://community.zenoss.org/people/dsyer%40nwnit.com] created the discussion

"Re: Event Escalation based on Time"

To view the discussion, visit: http://community.zenoss.org/message/69832#69832

--------------------------------------------------------------
The time fields arent there at transform time. 

You can deal with this a bunch of different ways all of which are a bit complicated but if this is something critical you are monitoring...honestly the best way is to either try to find a counter you can poll to set a threshold against that indicates the conditon state  or find another event that indicates that the condition has cleared and set it with a clear priority while making sure you account for the fields that need to match according to the clear mechanic.  If the event clears and you are not concerned with the number of individual failure instances but the total duration of any one then..

you can apply the event count transform transform against it knowing that the time is equal to polling cycle *count  Make sure though that  you do a check in the transform evt.severity = logic  so you dont turn your clear it to a warning, etc.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69832#69832]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Chaz Vidal]

Chaz Vidal [http://community.zenoss.org/people/chazzvid] created the discussion

"Re: Event Escalation based on Time"

To view the discussion, visit: http://community.zenoss.org/message/69805#69805

--------------------------------------------------------------
Thanks Doug,

I am already doing an escalation event based on count.  The first event comes in as a warning.  The problem is when I get the second event more than 5 minutes after the first event, say 30 minutes after the first event.  The transform will still escalate the 2nd event although it means that this 2nd event is likley a false alert.

I didnt' think that this would be something complicated --> If an event occurs twice within 5 minutes then escalate.

Anyway, thanks for the tips.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69805#69805]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Doug Syer]

Doug Syer [http://community.zenoss.org/people/dsyer%40nwnit.com] created the discussion

"Re: Event Escalation based on Time"

To view the discussion, visit: http://community.zenoss.org/message/69863#69863

--------------------------------------------------------------
I saw dpetzel posted this a few days ago, i thought it was pretty interesting way to potentially solve a problem like this but I havent tried it.

http://community.zenoss.org/message/69740#69740#69740 http://community.zenoss.org/message/69740#69740
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69863#69863]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Chaz Vidal]

Chaz Vidal [http://community.zenoss.org/people/chazzvid] created the discussion

"Re: Event Escalation based on Time"

To view the discussion, visit: http://community.zenoss.org/message/69843#69843

--------------------------------------------------------------
Yeah, the key part is getting the event's FirstTime attribute.  It doesn't look like this is straight forward in Zenoss which is a bit of shame really.

The only thing I can think of doing is lowering the event aging time so the first event doesnt stay open for very long.  This obviously affects the events globally and other events that i may want to work with in the future.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69843#69843]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Chaz Vidal]

Chaz Vidal [http://community.zenoss.org/people/chazzvid] created the discussion

"Re: Event Escalation based on Time"

To view the discussion, visit: http://community.zenoss.org/message/69804#69804

--------------------------------------------------------------
Thanks for responding Floyd.

I am using Enterprise 4.1.11.

Not really looking at using triggers here as I'm more concerned with escalating the event on the event console.  I could possibly use triggers but not sure how that logic will work.  I only want to escalate the event if the it repeats within 5 minutes.  Using existing trigger rules I can only send the event after 5 minutes if it is not closed before then which isn't what I am after.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69804#69804]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]