How to monitor batch job outcome

[whitetr6]

whitetr6 [http://community.zenoss.org/people/whitetr6] created the discussion

"How to monitor batch job outcome"

To view the discussion, visit: http://community.zenoss.org/message/64968#64968

--------------------------------------------------------------
Hi, I fought with this unsuccessfully with other monitoring systems, and I wanted to see if anyone in the Zenoss community has an idea on how to accomplish it. We have a nightly shell script running on a CentOS Linux box, which does various tasks such as stopping a MySQL database instance, moving files, sending an email etc. After each of the critical steps, I would like to notify Zenoss that the step has completed successfully, or it has not.

In the other systems I tried (Nagios, Icinga), the solution was to use passive monitoring and use a tool called send_nsca to send an update to the monitoring server as each step completed. The server was supposed to then throw an alert if a success message was not received in a given amount of time (25 hours maximum from the last message).

This was sketchy at best, never reliable enough for us to depend on it.

How would you suggest I approach this problem in Zenoss?

Thanks for any suggestions!
Mark
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64968#64968]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[themactech]

themactech [http://community.zenoss.org/people/themactech] created the discussion

"Re: How to monitor batch job outcome"

To view the discussion, visit: http://community.zenoss.org/message/64984#64984

--------------------------------------------------------------
This would be a mix of command datasource and writing your script to share data.  Here is an example...

Say your script does 4 things, at each step, have it write the data to a text file, i.e.

#!/bin/bash

blah blah blah

echo $TheResult1 > /Destination/foder/of/your/choice/Result1.txt

blah blah blah

echo $TheResult2 > /Destination/foder/of/your/choice/Result2.txt

blah blah blah

echo $TheResult3 > /Destination/foder/of/your/choice/Result3.txt

blah blah blah

echo $TheResult4 > /Destination/foder/of/your/choice/Result4.txt

Then you do a command script in Zenoss that will check for 2 things:

1- Are your files (Result1-4.txt) containing data that should trigger an alert?

2- Are your files (Result1-4.txt) older than 25 hours, if so send alert saying your daily script failed to run

This would have zenoss keep a close eye on your script data output and whether or not the script ran.

I have done very similar stuff and it works well.

Manuel
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/64984#64984]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[whitetr6]

whitetr6 [http://community.zenoss.org/people/whitetr6] created the discussion

"Re: How to monitor batch job outcome"

To view the discussion, visit: http://community.zenoss.org/message/65000#65000

--------------------------------------------------------------
Manuel thanks for this explanation. I believe that will get me where I need to be. Much appreciated
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65000#65000]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[dpetzel]

dpetzel [http://community.zenoss.org/people/dpetzel] created the discussion

"Re: How to monitor batch job outcome"

To view the discussion, visit: http://community.zenoss.org/message/65010#65010

--------------------------------------------------------------
This is an interesting solution Manuel. I've halfed toyed with the idea before of "alerting on a missed event", this approach would certainly work pretty well.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65010#65010]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[themactech]

themactech [http://community.zenoss.org/people/themactech] created the discussion

"Re: How to monitor batch job outcome"

To view the discussion, visit: http://community.zenoss.org/message/65098#65098

--------------------------------------------------------------
I use the -newerct tag of the find command a lot to keep track of what has executed.  Even scripts that don't produce output.  In that case I will simply use a 'touch' command, i.e.

#!/bin/bash

<script useful steps>

touch /my/dest/folder/bogus.txt


Then my command script will do a file modified date check

TheCheck=`find /my/dest/folder/ -name bogus.txt -newerct '15 minute ago' | wc -l`

If $TheCheck = 0, file hasn't been touched for 15 minutes, trigger alert...

Manuel
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/65098#65098]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]