Zenoss status/ping flase alarm

[Chakravarthi PS]

Chakravarthi PS [http://community.zenoss.org/people/pschakravarthi] created the discussion

"Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/68402#68402

--------------------------------------------------------------
Hi,

I am a newbie to Zenoss.   I added few devices which were connected thorugh a router.
I am able to ping these devices from zenoss VMs and also from Zenoss GUI I am able to ping

However, in alerts it says /status/ping "Down"

How can i correct this ?

Regards
Chakri              
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68402#68402]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[cjet]

cjet [http://community.zenoss.org/people/cjet] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/68497#68497

--------------------------------------------------------------
Same here. But old Zenoss user since 2.x..

I installed a clean Zenoss Core 4.2 install, and did zenbatchload from my devices.txt, which I dumped from old server.
In old Zenoss 3.2.1 everything worked as charm. Now in the new one, from 105 monitored devices, randomly about 40 devices are in the state "/status/ping DOWN". I have tried to find relations within these, which could cause this, but can not find any.

I am able to ping all my devices, I am able to model devices, even if Zenoss states those are down..
If I delete these events, which are "falsely down", they all come back. Firewalls are configured ok, but I am still missing something.

So any help would be preciated. Is there any ping timeout values in this new version which should be checked?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68497#68497]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[cjet]

cjet [http://community.zenoss.org/people/cjet] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/68503#68503

--------------------------------------------------------------
I might have found a reason for these fake alarms?

I found "a common denominator" for all monitored sites that are having this problem. All of those sites have a Checkpoint firewall between the devices and our monitoring site. None of the sites not having this false alarm problem have a checkpoint firewall in between of devices and monitoring.

Since the Checkpoints are doing their own shitty "factory detailed higher level inspection" for packets, it makes me think... Could it be that ICMP packet zenping uses for status-ping, has changed since older version?

Where could I modify ping packet size and stuff like that?
I found this older discussion from forum: http://community.zenoss.org/thread/16093 http://community.zenoss.org/thread/16093
--> How do we do the same in newer zenoss?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68503#68503]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Gerd Leue]

Gerd Leue [http://community.zenoss.org/people/cable_hogue] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/68606#68606

--------------------------------------------------------------
Hi all,
I use VMWare ZenossCore 4.2.0 had the same problem until
i added the following in
/opt/zenoss/Products/ZenStatus/nmap/NmapPingTask.py
in section " @defer.inlineCallbacks" of class NmapPingTask(BaseTask) (just vi search  "args.extend" in file):

args.extend(["--data-length", "8"])

which adds 8 bytes (or other value of your choice) of data to the ICMPping.
stop and restart zenping daemon...

see tcpdump
before:
10:58:49.275172  I IP xx.xx.225.43 > xx.xx.4.12: ICMP echo request, id 43221, seq 0, *length 8*
10:58:50.826112  I IP xx.xx.225.43 > xx.xx.4.12: ICMP echo request, id 14226, seq 0, *length 8*

after:
11:00:57.889802  I IP xx.xx.225.43 > xx.xx.4.12: ICMP echo request, id 31088, seq 0, *length 16*
11:00:57.911061  O IP xx.xx.4.12 > xx.xx.225.43: ICMP echo reply, id 31088, seq 0, *length 16*
11:00:59.441738  I IP xx.xx.225.43 > xx.xx.4.12: ICMP echo request, id 49980, seq 0, *length 16*
11:00:59.467399  O IP xx.xx.4.12 > xx.xx.225.43: ICMP echo reply, id 49980, seq 0, *length 16*

packets were not dropped anymore --> everything's green ;)

HTH

have fun
Gerd
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68606#68606]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[cjet]

cjet [http://community.zenoss.org/people/cjet] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/68605#68605

--------------------------------------------------------------
Seems to be working. Thanks. B-)

ps. This should be working like this straight out-of-the-box. Worked fine in 3.2.1.. Maybe one for a next bug fix?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68605#68605]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jplouis]

jplouis [http://community.zenoss.org/people/jplouis] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/68611#68611

--------------------------------------------------------------
4.2 started using nmap to do pings and traceroutes of devices in batch. Please open a ticket so that the datasize can be addressed.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/68611#68611]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Daniel Burge]

Daniel Burge [http://community.zenoss.org/people/dburge] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/69970#69970

--------------------------------------------------------------
Hi,

I was having this same problem. I found this thread and worked through it, however changing the packet size didn't make any difference for me. After a lot of testing I made a different change to the nmap code which fixed the problem.

Here are my notes on how I fixed it.

Fixing issue with device down alerts, even though ping appears to work correctly.

If you ping a device through the ping command on the device, or thourgh the ping command when logged into ZenOSSit will show that you can ping the device just fine. However, ZenOSS reports that the device is down.

The problem is that ZenOSS uses nmap behind the scenes in the ZenPing daemon to do the pings, whereas

the ZenOSS interface uses something else, maybe regular ping, or maybe they execute nmap differently there.

In my case, on my Virtuosso based CentOS virtual server, nmap was picking the wrong ethernet interface (venet0) to base it's source address off of, which was resulting in the wrong source address in the ICMP packets.

I tested this on a physical CentOS server as well, and nmap behaved correctly and picked the correct interface (eth0), and the icmp packets contained the correct source address.

For some background information from the nmap author: http://nmap.org/nmap_doc.html http://nmap.org/nmap_doc.html
"nmap tries to detect your primary interface and uses that address.  You can also use -S to specify it directly, but you shouldn't have to"

In my case, nmap is not detecting the correct source address. So I think this issue that I found is limited in scope to situations where nmap has this issue.

nmap has options which allow you to override the source address if needed.

By default ZenOSS doesn't include the source information to NMap, so nmap is called similar to this:

# nmap -sn -PE -n --privileged --send-ip --initial-rtt-timeout 2 --min-rtt-timeout 2 --max-retries 0 -v -d --packet-trace 8.8.8.8

The ICMP packets look like this:

SENT (0.0280s) ICMP 127.0.0.1 > 8.8.8.8 Echo request (type=8/code=0) ttl=55 id=16996 iplen=28

Notice the 127.0.0.1 as the source address.

I found that the destination device never receives this packet by monitoring the icmp traffic on the source and destination using tcpdump.

Watch the icmp traffic like this:

# tcpdump icmp

To fix it nmap must be run like this, to include the source IP using the nmap "spoof" option, even though you aren't really spoofing, you are just specifying your actual source information.

# nmap -S YOURIPADDRESS -sn -PE -n --privileged --send-ip --initial-rtt-timeout 2 --min-rtt-timeout 2 --max-retries 0 -v -d --packet-trace 8.8.8.8

The ICMP packets then look like this:
SENT (0.0400s) ICMP YOURIPADDRESS > 8.8.8.8 Echo request (type=8/code=0) ttl=53 id=40229 iplen=28
RCVD (0.0960s) ICMP 8.8.8.8 > YOURIPADDRESS Echo reply (type=0/code=0) ttl=57 id=36350 iplen=28

I couldn't figure out a way to fix it without having to hard code the source IP or Domain into the ZenOSS nmap code. Maybe someone else can.

To modify the Zenoss nmap code to send the requests with your corrected source address:

# vi /opt/zenoss/Products/ZenStatus/nmap/NmapPingTask.py

search for the python _executeNmapCmd function

add this line to the args:

    args.extend(["-S", "YOURIPADDRESSORDOMAIN"])



Then restart the "zenping" daemon.


hope that helps,

daniel
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/69970#69970]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[gedv]

gedv [http://community.zenoss.org/people/gedv] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/71106#71106

--------------------------------------------------------------
That worked for me, but only locally. The servers in the internet reports the same issue.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71106#71106]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Daniel Burge]

Daniel Burge [http://community.zenoss.org/people/dburge] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/71108#71108

--------------------------------------------------------------
Here's a thought. If you are running ZenOss on your local network, it might be that the ICMP packets are making it out of your network just fine, but the ICMP response packats are being blocked by a firewall on the way back into your network. You could monitor traffic on the server out on the internet to see if it's seeing the ICMP packet coming in.

daniel
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71108#71108]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[gedv]

gedv [http://community.zenoss.org/people/gedv] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/71158#71158

--------------------------------------------------------------
Daniel, I was monitoring with tcpdump the packets sent by zenping, this is the output:
17:01:05.332540 IP [zenoss-ip] > [my-desteny]: ICMP echo request, id 31573, seq 0, length 8

I think that the length 8 should be 64, as normal ICMP ping. But I can't change the length following this thread

http://community.zenoss.org/thread/16093 http://community.zenoss.org/thread/16093

Also, I was trying to run zenping by ping instead nmap, but still doing it by nmap

Many thanks.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71158#71158]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Philip Warren]

Philip Warren [http://community.zenoss.org/people/pwarren] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/71153#71153

--------------------------------------------------------------
The latest development release has fixed the problem with this change: http://jira.zenoss.com/jira/browse/ZEN-3439 http://jira.zenoss.com/jira/browse/ZEN-3439.

I am working on getting this backported to the 4.2.x branch so it should be available in the next maintenance release of Zenoss 4.2.

Thanks,

Philip
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71153#71153]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Philip Warren]

Philip Warren [http://community.zenoss.org/people/pwarren] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/71175#71175

--------------------------------------------------------------
A patch against the 4.2.3 release is available in this JIRA issue: http://jira.zenoss.com/jira/browse/ZEN-5104 http://jira.zenoss.com/jira/browse/ZEN-5104. Follow the instructions in the JIRA issue to apply the patch and set the data-length option.

This fix will be available in the next maintenance release of Zenoss.

Thanks,

Philip
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/71175#71175]

Start a new discussion in zenoss-users by email
[discussions-community-forums-zenoss--***@community.zenoss.org] -or- at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[cjet]

cjet [http://community.zenoss.org/people/cjet] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/73989#73989

--------------------------------------------------------------
I doubt that it has been fixed "in the next maintenance release".

After upgrading to 4.2.4, we are back fixing this issue again.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73989#73989]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[Jaajog]

Jaajog [http://community.zenoss.org/people/Jaajog] created the discussion

"Re: Zenoss status/ping flase alarm"

To view the discussion, visit: http://community.zenoss.org/message/74280#74280

--------------------------------------------------------------
After upgrading to 4.2.4 yesterday I get PING DOWN on several servers behind small Checkpoint boxes.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74280#74280]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]