Discussion:
phantom event cannot be cleared
Benjamin Schwarcz
2013-10-21 10:27:17 UTC
Permalink
Benjamin Schwarcz [http://community.zenoss.org/people/toxicgarden] created the discussion

"phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75014#75014

--------------------------------------------------------------
Hey,

i got one strange issue using Zenoss Core 4: since about one week, there is one event that always shows up, even as the problem doesn't exist anymore. If i want to open event details, it just says "ServiceResponseError: not found". There is no event with the given event_id in the databases and i can't even clear the event. It just keeps showing up, even when i delete the affected device. Is there some kind of reindexing i can do to get rid of this event?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75014#75014]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Benjamin Schwarcz
2013-10-21 11:31:37 UTC
Permalink
Benjamin Schwarcz [http://community.zenoss.org/people/toxicgarden] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75015#75015

--------------------------------------------------------------
additional info: this is the event, grabbed from zendmd. Any chance to delete it from there?

{'count': 241,
'first_seen_time': 1381847619966L,
'last_seen_time': 1381919721126L,
'occurrence': [{'actor': {'element_identifier': u'192.168.7.126',
                           'element_title': u'Klapp Prod Conn Backup',
                           'element_type_id': 1,
                           'element_uuid': u'add4b44e-085f-4308-be79-899a15ec6d8b'},
                 'agent': u'zencommand',
                 'created_time': 1381919721126L,
                 'details': [{'name': u'manager',
                              'value': [u'h0002-zenossp01']},
                             {'name': u'performanceData',
                              'value': [u'time=0.004867s;;;0.000000 size=1424B;;;0']},
                             {'name': u'zenoss.device.ip_address',
                              'value': [u'192.168.7.126']},
                             {'name': u'zenoss.device.production_state',
                              'value': [u'1000']},
                             {'name': u'zenoss.device.priority',
                              'value': [u'3']},
                             {'name': u'zenoss.device.device_class',
                              'value': [u'/Klapp/Connector/Prod']}],
                 'event_class': u'/Status/Web',
                 'event_key': u'KS',
                 'fingerprint': u'192.168.7.126||/Status/Web|KS|5',
                 'message': u'HTTP WARNING: HTTP/1.1 404 Not Found - 1424 bytes in 0.005 second response time ',
                 'monitor': u'localhost',
                 'severity': 5,
                 'summary': u'HTTP WARNING: HTTP/1.1 404 Not Found - 1424 bytes in 0.005 second response time ',
                 'tags': [{'type': u'zenoss.device.device_class',
                           'uuid': [u'37d143de-7c78-4ceb-83f6-fd58fb705eb1',
                                    u'e935643d-5954-4b48-9a98-77d42c91109d',
                                    u'83465291-b9c6-4645-8de8-c51be320cb10']},
                          {'type': u'zenoss.event.event_class',
                           'uuid': [u'27a8c329-eb37-4b24-a4ad-69ff79af8659',
                                    u'c73b4280-058f-40d7-a33e-fa3460f58a94']}]}],
'status': 0,
'status_change_time': 1381847619966L,
'update_time': 1381919724698L,
'uuid': u'005056a9-6696-b296-11e3-35a6c98a1617'}
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75015#75015]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
hydruid
2013-10-21 13:08:32 UTC
Permalink
hydruid [http://community.zenoss.org/people/hydruid] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75006#75006

--------------------------------------------------------------
Take a look at this post: http://community.zenoss.org/message/69522#69522 http://community.zenoss.org/message/69522
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75006#75006]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
jonas.andersson
2013-10-21 13:18:38 UTC
Permalink
jonas.andersson [http://community.zenoss.org/people/jonas.andersson] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75007#75007

--------------------------------------------------------------
unsubscribe zenoss-user
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75007#75007]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Benjamin Schwarcz
2013-10-21 13:48:07 UTC
Permalink
Benjamin Schwarcz [http://community.zenoss.org/people/toxicgarden] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75016#75016

--------------------------------------------------------------
already tried that, sadly it didn't work. But it seems like the event isn't even deletable in zendmd. Here's my weird history.

Find out that the event is still there and active in any way:

[quote]
filter_zep = zep.createEventFilter(uuid='005056a9-6696-b296-11e3-35a6c98a1617')
...     pprint(evt)
...
{'count': 241,
'first_seen_time': 1381847619966L,
'last_seen_time': 1381919721126L,
'occurrence': [{'actor': {'element_identifier': u'192.168.7.126',
                           'element_title': u'Klapp Prod Conn Backup',
                           'element_type_id': 1,
                           'element_uuid': u'add4b44e-085f-4308-be79-899a15ec6d8b'},
                 'agent': u'zencommand',
                 'created_time': 1381919721126L,
                 'details': [{'name': u'manager',
                              'value': [u'h0002-zenossp01']},
                             {'name': u'performanceData',
                              'value': [u'time=0.004867s;;;0.000000 size=1424B;;;0']},
                             {'name': u'zenoss.device.ip_address',
                              'value': [u'192.168.7.126']},
                             {'name': u'zenoss.device.production_state',
                              'value': [u'1000']},
                             {'name': u'zenoss.device.priority',
                              'value': [u'3']},
                             {'name': u'zenoss.device.device_class',
                              'value': [u'/Klapp/Connector/Prod']}],
                 'event_class': u'/Status/Web',
                 'event_key': u'KS',
                 'fingerprint': u'192.168.7.126||/Status/Web|KS|5',
                 'message': u'HTTP WARNING: HTTP/1.1 404 Not Found - 1424 bytes in 0.005 second response time ',
                 'monitor': u'localhost',
                 'severity': 5,
                 'summary': u'HTTP WARNING: HTTP/1.1 404 Not Found - 1424 bytes in 0.005 second response time ',
                 'tags': [{'type': u'zenoss.device.device_class',
                           'uuid': [u'37d143de-7c78-4ceb-83f6-fd58fb705eb1',
                                    u'e935643d-5954-4b48-9a98-77d42c91109d',
                                    u'83465291-b9c6-4645-8de8-c51be320cb10']},
                          {'type': u'zenoss.event.event_class',
                           'uuid': [u'27a8c329-eb37-4b24-a4ad-69ff79af8659',
                                    u'c73b4280-058f-40d7-a33e-fa3460f58a94']}]}],
'status': 0,
'status_change_time': 1381847619966L,
'update_time': 1381919724698L,
'uuid': u'005056a9-6696-b296-11e3-35a6c98a1617'}
[/quote]

Try to get details on this specific one. Now he seems to be gone!?

[quote]
zep.getEventSummary('005056a9-6696-b296-11e3-35a6c98a1617')
Traceback (most recent call last):
  File "<console>", line 1, in <module>
  File "/opt/zenoss/Products/Zuul/facades/zepfacade.py", line 404, in getEventSummary
    response, content = self.client.getEventSummary(uuid)
  File "/opt/zenoss/lib/python/zenoss/protocols/services/zep.py", line 131, in getEventSummary
    return self.client.get('%s' % uuid)
  File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 181, in get
    return self._request(path, method='GET', params=params, headers=headers)
  File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 160, in _request
    response, content = self._executeRequest(request)
  File "/opt/zenoss/lib/python/zenoss/protocols/services/__init__.py", line 154, in _executeRequest
    raise ServiceResponseError(response.reason, response.status, request, response, content)
ServiceResponseError: Not Found
[/quote]


it's also not possible to just delete the event via  dmd.Events.manage_deleteEvents. It still shows up.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75016#75016]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Benjamin Schwarcz
2013-10-24 14:34:58 UTC
Permalink
Benjamin Schwarcz [http://community.zenoss.org/people/toxicgarden] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75055#75055

--------------------------------------------------------------
*le bump*
no ideas? I was going to present Zenoss as our new monitoring solution next week and would really like to get rid of this one critical event. :\
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75055#75055]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
jcurry
2013-10-25 12:54:12 UTC
Permalink
jcurry [http://community.zenoss.org/people/jcurry] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75080#75080

--------------------------------------------------------------
Have you seen Ryan's wiki hint at http://wiki.zenoss.org/ZenDMD_Tip_-_Manipulate_Events http://wiki.zenoss.org/ZenDMD_Tip_-_Manipulate_Events .

My experience with digging around with accessing events with the zep.getEventSummariesGenerator and zep.getEventSummary is that they are subtly different.  I cannot really give you chapter-and-verse on how / why they are different.  I do know that they are generator functions and once you have used the generated value (like to print it out for debugging), then it is "no longer there" to use in other calls.

I would try Ryan's code with the zep.getEventSummariesGenerator and feed the reult directly into a
zep.closeEventSummaries
.

Cheers,
Jane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75080#75080]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Benjamin Schwarcz
2013-10-29 09:54:48 UTC
Permalink
Benjamin Schwarcz [http://community.zenoss.org/people/toxicgarden] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75109#75109

--------------------------------------------------------------
i tried closing it via this event, but sadly, it didn't work.
zep.closeEventSummaries(eventFilter=filter_zep,userName='admin')
({'status': '200', 'content-length': '4', 'server': 'Jetty(7.5.3.v20111011)', 'x-protobuf-fullname': 'org.zenoss.protobufs.zep.EventSummaryUpdateResponse', 'date': 'Tue, 29 Oct 2013 09:51:20 GMT', 'content-type': 'application/x-protobuf'}, {'updated': 0, 'total': 1})

so the event seems to be recognized ("total: 1"), but not updated. I was desperate enough to search for the uuid in the whole /opt/zenoss directory, but it seems like there's no file or database-entry related to it. :\
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75109#75109]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Benjamin Schwarcz
2013-10-29 11:50:53 UTC
Permalink
Benjamin Schwarcz [http://community.zenoss.org/people/toxicgarden] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75112#75112

--------------------------------------------------------------
okay so i solved it the hard way:
rm -rf /opt/zenoss/var/zeneventserver/*
zeneventserver restart


now the phantom event is gone
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75112#75112]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
jcurry
2013-10-29 12:29:30 UTC
Permalink
jcurry [http://community.zenoss.org/people/jcurry] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75113#75113

--------------------------------------------------------------
Wow - I wonder what that actually did?  Removed all the indexes by the looks of it.  Did they get recreated again when you restarted zeneventserver?

What led you to doing this??

If it worked without breaking anything else then I am delighted - and so will someone else be when they end up in the same situation.  I wonder if you have tickled an occasional bug here?

Cheers,
Jane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75113#75113]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Benjamin Schwarcz
2013-10-29 12:34:18 UTC
Permalink
Benjamin Schwarcz [http://community.zenoss.org/people/toxicgarden] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75114#75114

--------------------------------------------------------------
i was searching around about clearing queues and got to this page: http://www.nickyeates.com/technology/zenoss/useful_commands_zenoss4 http://www.nickyeates.com/technology/zenoss/useful_commands_zenoss4

Key point there was "Reset your ZEP instance to zero". I didn't want to lose all Objects and such, so i skipped the point where it says "force-recreate the database" and just deleted the files mentioned there. (to be honest: i moved them to /tmp just for the case... ;) )

As i wasn't sure about just restarting the zeneventserver, i just restarted zenoss as a whole and all events, as well as the objects, were still there. So no harm done to the working systems.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75114#75114]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
jcurry
2013-10-29 15:46:30 UTC
Permalink
jcurry [http://community.zenoss.org/people/jcurry] created the discussion

"Re: phantom event cannot be cleared"

To view the discussion, visit: http://community.zenoss.org/message/75119#75119

--------------------------------------------------------------
Sounds like this is the method that Zenoss currently suggest for deleting "stuck" events, although they would recommend copying to something like $ZENHOME/var/zeneventserver.bak and recreating the $ZENHOME/var/zeneventserver directory.

Many thanks - I have certainly learnt something from this.

Cheers,
Jane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/75119#75119]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]
Loading...