Zenoss Events not displaying whole Syslog Message

Discussion:

heinds

2013-04-22 08:35:16 UTC

heinds [http://community.zenoss.org/people/heinds] created the discussion

"Zenoss Events not displaying whole Syslog Message"

To view the discussion, visit: http://community.zenoss.org/message/72931#72931

--------------------------------------------------------------
Hi Guys,

I have a fairly simple question.

When looking in /opt/zenoss/log/ i see the following message that zensyslog received from my server:

Apr 20 10:20:28 192.168.99.2Â alertÂ : 1/1/1025: alarm_mgr: 01: 1:07:02 Minor ONU Down
Line 1/1/7/2/gpononu CAUSE: Dying Gasp received

BUT....

When it goes through to "events" is comes through like this:

| agent | zensyslog |
| component | 10:20:28 |
| dedupid | 192.168.99.2|10:20:28|/Unknown|5|alertÂ : 1/1/1025: alarm_mgr: 01: 1:07:02 Minor ONU Down |

In other words, the section "Line 1/1/1/2/gpononu CAUSE: Dying Gasp received" is not being fed through to "events" and i'm not receiving that fital part of the message.

How can I change this that the second line also comes throgh and forms part of the message?

Thank you!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/72931#72931]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

heinds

2013-04-23 17:41:13 UTC

Permalink

nilie

2013-04-24 17:53:07 UTC

Permalink

heinds

2013-04-24 20:37:28 UTC

Permalink

nilie

2013-04-25 15:15:06 UTC

Permalink

heinds

2013-04-26 16:19:12 UTC

Permalink

nilie

2013-04-26 18:11:55 UTC

Permalink

nilie [http://community.zenoss.org/people/nilie] created the discussion

"Re: Zenoss Events not displaying whole Syslog Message"

To view the discussion, visit: http://community.zenoss.org/message/72993#72993

--------------------------------------------------------------
When receiving a syslog message, Zenoss server is doing its best to parse it hoping it will conform to syslog standards format. It starts by looking for a time stamp, the host IP address or name, facility and severity and will parse what's left trying to find some distinctive paterns that will help in classifying the event. According to standards, a syslog message is one single ASCII character string with some structure added to it and this is where the problem might occur: different vendors might construct different structures because the IETF standard doesn't cover the message part in detail, only the header is standardized.
If your equipment (by the way can you tell us what kind of equipment is this) is sending some info on a separate line, it seems this second line lacks the standard syslog header so Zenoss can't parse it at all. Other possibility would be for Zenoss to detect a character that will falsely signal the end of the message and this will cause the rest of the message to be discarded. In order for me to figure out what happens there are two things that I need :
* the setup of syslog on your server. Is Zenoss receiving syslog messages directly by listening on port UDP/514 or you are using some technique to relay syslog messages to Zenoss ?
* can you capture one or more of these messages on your server using tcpdump or Wireshark and post some screenshots here ?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/72993#72993]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

heinds

2013-04-29 12:17:35 UTC

Permalink

heinds [http://community.zenoss.org/people/heinds] created the discussion

"Re: Zenoss Events not displaying whole Syslog Message"

To view the discussion, visit: http://community.zenoss.org/message/73003#73003

--------------------------------------------------------------
Hi Nilie,

Thank you for your feedback.

To answer your questions:

The equipment that is sending the syslog messages is a Zhone MXK.

http://www.zhone.com/products/msan/MXK/ http://www.zhone.com/products/msan/MXK/

The server is sending syslog messages directly to the zenoss server. No relaying..

The Zenoss server is listining on UDP/514, please see below..

[***@localhost ~]# lsof -i -P | grep 514
pythonÂ Â Â Â 3159Â Â Â zenossÂ Â 12uÂ IPv4Â 13481Â Â Â Â Â 0t0Â UDP *:514

Here is the messages that I can see on the MXK/Server logs:

APR 28 13:36:39: alertÂ : 1/2/1025: alarm_mgr: 01: 2:04:05 Minor ONU Down
Line 1/2/4/5/gpononu CAUSE: Dying Gasp received

APR 28 13:36:53: alertÂ : 1/2/1025: alarm_mgr: 01: 2:04:02 Minor ONU Down
Line 1/2/4/2/gpononu CAUSE: Dying Gasp received

APR 28 13:37:21: alertÂ : 1/2/1025: alarm_mgr: 01: 2:04:05 Minor ONU Up
Line 1/2/4/5/gpononu CAUSE: active

APR 28 13:59:00: alertÂ : 1/2/1025: alarm_mgr: 01: 2:04:02 Minor ONU Up
Line 1/2/4/2/gpononu CAUSE: active

APR 28 15:28:03: alertÂ : 1/2/1025: alarm_mgr: 01: 2:04:02 Minor ONU Down
Line 1/2/4/2/gpononu CAUSE: Dying Gasp received

APR 28 15:29:41: alertÂ : 1/2/1025: alarm_mgr: 01: 2:04:02 Minor ONU Up
Line 1/2/4/2/gpononu CAUSE: active

APR 29 04:59:39: alertÂ : 1/2/1025: alarm_mgr: 01: 2:07 MinorÂ Â Unassigned ONU serial number 10 for OLT7 found
APR 29 05:03:09: alertÂ : 1/2/1025: alarm_mgr: 01: 2:07 MinorÂ Â Unassigned ONU serial number 10 for OLT7 no longer present

I see exactly the same messages in origsyslog.log in $ZENHOME/log.

I personanly dont think it's a lenth issue because the longer messages still comes through as long as it's on one line.

It looks to me that zenoss only ever exepts the first line as the message, is there a way to manuaaly tell zenoss to include the second line as well?

Regards
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73003#73003]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

nilie

2013-04-29 20:19:29 UTC

Permalink

jmp242

2013-04-30 12:35:54 UTC

Permalink

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: Zenoss Events not displaying whole Syslog Message"

To view the discussion, visit: http://community.zenoss.org/message/73025#73025

--------------------------------------------------------------
I'm not sure this is entirely true - it is open source after all. I've seen other forum threads munge syslog in 2 ways:
1. edit the regex in the Zenoss code (probably not a great idea unless you hire a guru / consultant)
2. front end Zenoss with a different syslog daemon that can "fix" the bad syslog - but for this particular issue I can't actually recommend one as I haven't seen this before. Either way may well require coding on something though.

The take away is unless you're pretty experianced with coding or can hire someone who is - you probably can't fix this.

--
James Pulver
ZCA Member
LEPP Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73025#73025]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]