best practices for generating alerts

Discussion:

briffle

2012-04-24 13:27:38 UTC

briffle [http://community.zenoss.org/people/briffle] created the discussion

"best practices for generating alerts"

To view the discussion, visit: http://community.zenoss.org/message/66034#66034

--------------------------------------------------------------
I have been using zenoss for about a year now to monitor our systems. We are looking to expand its use to include many of the scripts we run on our servers, as part of our cron jobs.Â I guess I am not sure the best way to use Zenoss to alert for two very different scenario.. I have never done it inside a shell script (mostly bash, some ksh), and would love some examples.

We have some scripts that run ittermittently, and would love to use Zenoss to alert us if they fail (have an error).Â Right now, we send emails of the results, good or bad, which means hundreds of emails to ignore, and an occasional one to have to deal with.Â We want to get out of the habit of ignoring all the emails :) Â Actually, i would love to also send the good results too, so we know it ran, which leads to the next issue..

Secondly, we have some scripts that are hard to parse for errors.. in that case, it would be ideal to somehow have it let zenoss know it ran, and then, have zenoss throw an alert if it DOES NOT check in that day, but that is often only once per day.Â (so I guess the last 'event' would have to be > 24 hours)..

We just really haven't added our own 'custom' stuff to zenoss much, and would like to start!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/66034#66034]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

briffle

2012-04-24 13:33:13 UTC

Permalink

themactech

2012-04-24 15:24:05 UTC

Permalink

briffle

2012-04-24 15:56:14 UTC

Permalink

briffle [http://community.zenoss.org/people/briffle] created the discussion

"Re: best practices for generating alerts"

To view the discussion, visit: http://community.zenoss.org/message/66036#66036

--------------------------------------------------------------
Manuel, thanks, but from digging around, the command scripts you are referring to are run from zenoss.. (ie, running commands over SSH).. I have written several scripts and commands that reach out from the zenoss server, and parse the nagios style output.Â

However, I am trying to update existing scripts (sometimes manually run, sometimes cron jobs) on dozens of remote servers.Â I cannot run them from the zenoss machine, they must run locally (and originate) from the server.. Basically, i want to customize a script, running standalone on a remote machine, to generate alerts or actions on a remote zenoss server.

For example, at an exact time, we make a copy of our Oracle Database by calling a cron job.Â As soon as it is done, several other scripts are manually run (from the end of the cronjob).Â I would like the script to alert Zenoss that the database was copied successfully, or throw an alert if it was not.Â I also want the other scripts it starts up to do similar things.Â For security reasons, the Zenoss machine cannot do anything put passively monitor SNMP on some of my servers, in a read only community.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/66036#66036]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

themactech

2012-04-24 16:00:50 UTC

Permalink

themactech

2012-04-24 16:48:48 UTC

Permalink

themactech [http://community.zenoss.org/people/themactech] created the discussion

"Re: best practices for generating alerts"

To view the discussion, visit: http://community.zenoss.org/message/66046#66046

--------------------------------------------------------------
Also I have done setups where scripts are run locally on servers (like you are doing), but will write outcomes to a text file.

Zenoss then parses this text file for input of data, I have even included checking the modification date of the file to test if the local script failed to execute.

There are many ways to skin this cat, I am sure it can be done.Â For example, if you run you Oracle database copy script, then have it trigger say 3 scripts when it is done, you would have your scripts do this:

Your Oracle copy script output exits status in OracleCopy.txt, overwriting the file each time it executes

Script1 - 3 will do the save with files called Script1Output.txt, Script2Output.txt, Script3Output.txt, again overwriting the original.

Your command script on the Zenoss machine then does this for each of your files:

1- As the file NOT been updated in over 24 hours:

Â Â Â Â Yes: Scripts failed to update files, set exit status to 1 or 2 (if you want warning or crit alert)
Â Â Â Â No: File was updated properly on last script run, proceed
Â Â Â Â Â Â Â Â Â Is content of file within thresholds:
Â Â Â Â Â Â Â Â Â Â Â Â Â Â No: Script execution was not within acceptable bounds, set exit status to 1 or 2
Â Â Â Â Â Â Â Â Â Â Â Â Â Â Yes: Script execution was within bounds, set exit status to 0

This can be run from the Zenoss computer over SSH or could be run locally via the Extend function of Net-SNMP.

I might be off on my understanding of what you are trying to do however.

Manuel
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/66046#66046]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

briffle

2012-04-24 19:32:26 UTC

Permalink

briffle [http://community.zenoss.org/people/briffle] created the discussion

"Re: best practices for generating alerts"

To view the discussion, visit: http://community.zenoss.org/message/66038#66038

--------------------------------------------------------------
basically, I am hoping there is an easy way I can give to our developers (and my felow admins) to add a simple routine to send Zenoss alerts on a script when it fails.. For example, with a SQL command on our linux servers, I can send an email directly from the DB server from inside the script, since they were all setup once to allow emailing. I am hoping to avoid having to define and figure out OID's to use, or having to setup lots of steps in zenoss..

if there was a way to write a small function that says something along the lines of:

Â Â Â Â SendAlert <zenoss Server> <severity> "text of Alert"
or
Â Â Â Â SendAlert <zenoss Server> <DataSource> <number>

that would be a godsend.. the assigned people would then see the alerts, and route them to the correct group.Â (I would really kill for a REST API that let me set values, instead of just read.. (which is weird, since you can create new objects.. )
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/66038#66038]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

themactech

2012-04-24 19:49:53 UTC

Permalink

themactech [http://community.zenoss.org/people/themactech] created the discussion

"Re: best practices for generating alerts"

To view the discussion, visit: http://community.zenoss.org/message/66050#66050

--------------------------------------------------------------
There is a way to generate alerts via scripts, I have used it.Â Â I made a proof of concept test where events at one end would take all the event variables I needed, encoded them in an email, sent it out.

At the other end, it would read the email, parse out the event variables and recreate the event.Â The result was 2 identical event histories at both ends with no network link (other than an accessible email server) between both ends.

You just need to generate the variables yourself instead of pulling them from an actual zenoss event.

Let me find my notes or scripts for this...

Ok, from the shell, the command is:

zensendevent, it is documented in the Zenoss Administration guide in section 6.1.8.2

You could have your servers execute this remotely on the Zenoss server with the right variables, the event would be created in Zenoss and trigger alerts.Â Creating a serverity zero event will clear the first.

Manuel
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/66050#66050]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

dpetzel

2012-04-24 22:41:08 UTC

Permalink

dpetzel [http://community.zenoss.org/people/dpetzel] created the discussion

"Re: best practices for generating alerts"

To view the discussion, visit: http://community.zenoss.org/message/66039#66039

--------------------------------------------------------------
Will the JSON API not meet your needs? http://community.zenoss.org/docs/DOC-8584

add event(self, summary, device, component, severity, evclasskey, evclass)

Seems to be what your after?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/66039#66039]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]