No events on 4.2.3 / CentOS 6.3

Discussion:

szheng

2013-05-02 10:54:15 UTC

szheng [http://community.zenoss.org/people/szheng] created the discussion

"No events on 4.2.3 / CentOS 6.3"

To view the discussion, visit: http://community.zenoss.org/message/73048#73048

--------------------------------------------------------------
Hello,

I have installed the zenoss core 4.2.3 on centos 6.3 with RPM package. The installation wentÂ smoothly with no errors. But i just found that totally no events logged on the UI , rabbitmq and mysql . Also no error or warning in my zenoss log files. Could any body tell me how toÂ resolve the issue? Any advice will be appreciated.

[***@aaa]# rabbitmqctl list_queues -p /zenoss
Listing queues ...
celeryÂ Â Â Â Â Â Â Â Â 0
zenoss.queues.zep.modelchangeÂ Â Â Â Â Â Â Â Â 0
zenoss.queues.zep.migrated.summaryÂ Â Â Â Â Â Â Â Â 0
zenoss.queues.zep.heartbeatsÂ Â Â Â Â Â Â Â Â 0
zenoss.queues.zep.zeneventsÂ Â Â Â Â Â Â Â Â 0
inf017.tyo02.ett.celeryd.pidboxÂ Â Â Â Â Â Â Â Â 0
zenoss.queues.zep.migrated.archiveÂ Â Â Â Â Â Â Â Â 0
...done.

[***@aaa]# rabbitmqctlÂ list_user_permissions -p /zenoss zenoss
Listing permissions for user "zenoss" ...
/zenossÂ Â Â Â Â Â Â Â Â .*Â Â Â Â Â Â Â Â Â .*Â Â Â Â Â Â Â Â Â .*
...done.

I am using Percona Mysql server:
Percona-Server-server-55-5.5.30-rel30.1.465.rhel6.x86_64
Percona-Server-client-55-5.5.30-rel30.1.465.rhel6.x86_64
Percona-Server-shared-55-5.5.30-rel30.1.465.rhel6.x86_64
Percona-Server-shared-compat-5.5.30-rel30.1.465.rhel6.x86_64

Oracle JRE 1.7:
[***@aaa# rpm -qa|grep jre
jre-1.7.0_21-fcs.i586

Lots of MySQL temporary table in my /tmp directory, i don't know what is it :
[***@aaa]# ls -al
total 157732
drwxrwxrwt. 10 rootÂ Â rootÂ Â Â Â Â Â 12288 MayÂ 2 19:48 .
dr-xr-xr-x. 24 rootÂ Â rootÂ Â Â Â Â Â Â 4096 Apr 26 10:35 ..
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â 8658 MayÂ 2 19:48 #sql5bbf_7c6_1128.frm
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â 1704 MayÂ 2 19:48 #sql5bbf_7c6_1128.MYD
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â 1024 MayÂ 2 19:48 #sql5bbf_7c6_1128.MYI
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â 8632 MayÂ 2 19:48 #sql5bbf_7c6_1129.frm
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â Â Â Â 0 MayÂ 2 19:48 #sql5bbf_7c6_1129.MYD
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â 1024 MayÂ 2 19:48 #sql5bbf_7c6_1129.MYI
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â 8658 MayÂ 2 11:18 #sql5bbf_7c7_0.frm
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â Â Â Â 0 MayÂ 2 11:18 #sql5bbf_7c7_0.MYD
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â 1024 MayÂ 2 11:18 #sql5bbf_7c7_0.MYI
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â 8632 MayÂ 2 11:18 #sql5bbf_7c7_1.frm
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â Â Â Â 0 MayÂ 2 11:18 #sql5bbf_7c7_1.MYD
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â 1024 MayÂ 2 11:18 #sql5bbf_7c7_1.MYI
-rw-rw----Â Â 1 mysqlÂ mysqlÂ Â Â Â Â Â 8658 MayÂ 2 17:59 #sql5bbf_7c8_206.frm

Thanks!
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73048#73048]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

jmp242

2013-05-02 11:59:49 UTC

Permalink

jmp242 [http://community.zenoss.org/people/jmp242] created the discussion

"Re: No events on 4.2.3 / CentOS 6.3"

To view the discussion, visit: http://community.zenoss.org/message/73049#73049

--------------------------------------------------------------
I'm not sure I understand your question. Do you mean there is no Events tab in the UI? Are there some events you've generated that don't show up? What are you monitoring that you think would generate an event?

--
James Pulver
ZCA Member
LEPP Computer Group
Cornell University
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73049#73049]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

szheng

2013-05-07 05:39:32 UTC

Permalink

Ryan Matte

2013-05-07 17:21:57 UTC

Permalink

Ryan Matte [http://community.zenoss.org/people/rmatte] created the discussion

"Re: No events on 4.2.3 / CentOS 6.3"

To view the discussion, visit: http://community.zenoss.org/message/73087#73087

--------------------------------------------------------------
Did you change the hostname of that server after you installed rabbitmq?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73087#73087]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

Andrew Kirch

2013-05-07 17:24:01 UTC

Permalink

Andrew Kirch [http://community.zenoss.org/people/akirch] created the discussion

"Re: No events on 4.2.3 / CentOS 6.3"

To view the discussion, visit: http://community.zenoss.org/message/73088#73088

--------------------------------------------------------------
Event or trigger/notification?Â Does it show up in the events console?Â There is a new event system in 4.x which requires triggers/notifications be set up.

Please, shut down a server, or generate a test event with 'zensendevent' , does it show up in the events console?Â If not what is in zenactiond.log?
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73088#73088]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

szheng

2013-05-08 10:03:12 UTC

Permalink

szheng [http://community.zenoss.org/people/szheng] created the discussion

"Re: No events on 4.2.3 / CentOS 6.3"

To view the discussion, visit: http://community.zenoss.org/message/73100#73100

--------------------------------------------------------------
Ryan,
I didn't change the hostname after i installed the rabbitmq.

Andrew,
No,it show nothing in my events console.
I just generated a event by 'zensendevnet' command , but still can't find any event in my events console. The logs like below:

test command:Â /opt/zenoss/bin/zensendevent -v -i 10.1.4.1 -s Info -c /Status/Ping --auth=admin:zenossÂ "ip 10.1.4.2 is down"

zenhub:
2013-05-08 18:41:52,914 DEBUG zen.ZenHub: worklist has 1 items
2013-05-08 18:41:52,915 DEBUG zen.ZenHub: get candidate workers for sendEvents...
2013-05-08 18:41:52,915 DEBUG zen.ZenHub: candidate workers are [0, 1]
2013-05-08 18:41:52,915 DEBUG zen.ZenHub: Giving sendEvents to worker 0, (localhost:Products.ZenHub.services.EventService.sendEvents)
2013-05-08 18:41:52,919 DEBUG zen.ZenHub: worker 0, work localhost:Products.ZenHub.services.EventService.sendEvents finished in 0.00407791137695
2013-05-08 18:42:02,830 DEBUG zen.Events: ===============Â incoming eventÂ ===============
2013-05-08 18:42:02,831 DEBUG zen.queuepublisher: About to publish this event to the raw event queue:uuid: "8986278c-b7c3-11e2-a7bf-003048d46f86"
created_time: 1368006122831
event_class: "/Status/Ping"
actor {
Â element_type_id: DEVICE
Â element_identifier: "myhost.int"
}
summary: "ip 10.1.4.2 is down"
severity: SEVERITY_CRITICAL
event_key: ""
monitor: "localhost"
details {
Â name: "zenoss.device.ip_address"
Â value: "10.1.4.1"
}
first_seen_time: 1368006122831
, with this routing key: zenoss.zenevent.status.ping
2013-05-08 18:42:02,834 DEBUG zen.zenoss.protocols.amqp: Connecting to RabbitMQ...
2013-05-08 18:42:02,835 DEBUG zen.protocols: Creating exchange: zenoss.zenevents.raw
2013-05-08 18:42:02,835 DEBUG zen.zenoss.protocols.amqp: Publishing with routing key zenoss.zenevent.status.ping to exchange zenoss.zenevents.raw
2013-05-08 18:42:02,836 INFO zen.ZenHub.twisted: None - - [08/May/2013:09:42:02 +0000] "POST /RPC2 HTTP/1.1" 200 162 "-" "xmlrpclib.py/1.0.1 (by www.pythonware.com)"

no log in zenactiond.log or other log files

still nothing in the rabbitmq queues and the zenoss_zep database:

[***@myhost log]# rabbitmqctl list_queues -p /zenoss
Listing queues ...
celeryÂ Â Â Â Â Â Â Â Â 0
zenoss.queues.zep.modelchangeÂ Â Â Â Â Â Â Â Â 0
zenoss.queues.zep.migrated.summaryÂ Â Â Â Â Â Â Â Â 0
zenoss.queues.zep.heartbeatsÂ Â Â Â Â Â Â Â Â 0
zenoss.queues.zep.zeneventsÂ Â Â Â Â Â Â Â Â 0
inf017.tyo02.ett.celeryd.pidboxÂ Â Â Â Â Â Â Â Â 0
zenoss.queues.zep.migrated.archiveÂ Â Â Â Â Â Â Â Â 0
...done.

Some mysql errors was found in mysql.log like below. The errors happen every morning :

130508Â 7:02:30 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_2d through its index translation table
130508Â 7:41:08 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_2f through its index translation table
130508Â 7:43:49 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_31 through its index translation table
130508Â 7:48:05 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_33 through its index translation table
130508Â 7:52:20 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_35 through its index translation table
130508Â 7:56:30 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_37 through its index translation table
130508Â 7:57:49 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_39 through its index translation table
130508Â 7:59:25 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_3b through its index translation table
130508Â 8:00:55 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_3d through its index translation table
130508Â 8:02:16 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_3f through its index translation table
130508Â 8:05:12 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_41 through its index translation table
130508Â 8:08:19 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_43 through its index translation table
130508Â 8:13:06 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_45 through its index translation table
130508Â 8:17:31 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_47 through its index translation table
130508Â 8:21:51 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_49 through its index translation table
130508Â 8:26:04 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_4b through its index translation table
130508Â 8:30:32 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_4d through its index translation table
130508Â 8:31:13 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_4f through its index translation table
130508Â 8:38:03 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_51 through its index translation table
130508Â 8:43:04 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_53 through its index translation table
130508Â 8:48:12 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_55 through its index translation table
130508Â 8:53:29 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_57 through its index translation table
130508Â 8:55:28 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_59 through its index translation table
130508Â 9:01:02 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_5b through its index translation table
130508Â 9:06:26 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_5d through its index translation table
130508Â 9:10:18 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_5f through its index translation table
130508Â 9:15:39 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_61 through its index translation table
130508Â 9:19:25 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_63 through its index translation table
130508Â 9:20:21 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_65 through its index translation table
130508Â 9:26:48 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_67 through its index translation table
130508Â 9:57:42 [Warning] InnoDB could not find index DelInx1 key no 0 for table tmp/#sql5bbf_3e8a_69 through its index translation table

Thanks.

Sheng
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73100#73100]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

Shane Scott

2013-05-10 20:50:40 UTC

Permalink

Shane Scott [http://community.zenoss.org/people/hackman238] created the discussion

"Re: No events on 4.2.3 / CentOS 6.3"

To view the discussion, visit: http://community.zenoss.org/message/73150#73150

--------------------------------------------------------------
Sszheng:

Your zenoss-zep database is toast. Make sure your server has enough RAM and disk space that the kernel won't OOM the mysql service and that the mysql service won't run out of space. You'll probably want to drop zenoss-zep then manually su root; $ZENHOME=/opt/zenoss; $ZENHOME/bin/zenoss_init_pre to rebuild it.

Best,
--Shane Scott (Hackman238)
http://www.linkedin.com/in/shanewilliamscott/ http://www.linkedin.com/in/shanewilliamscott/
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/73150#73150]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

szheng

2013-05-15 02:25:57 UTC

Permalink

Shane Scott

2013-05-15 13:00:13 UTC

Permalink