Zenhub zopectl won't start...rabbitmq connection_closed_abruptly

[kenhen93]

kenhen93 [http://community.zenoss.org/people/kenhen93] created the discussion

"Zenhub zopectl won't start...rabbitmq connection_closed_abruptly"

To view the discussion, visit: http://community.zenoss.org/message/74504#74504

--------------------------------------------------------------
Hi,

I am having a problem with Zenoss 4.2.4 running on Red Hat Enterprise Linux Server release 6.4    2.6.32-358.el6.x86_64. I have searched the zenoss community and internet pretty extensively.

When I start zenoss zenhub and zope will not start. Everything points to rabbitmq but I am not getting any good information from the logging of rabbitmq.

+/var/log/rabbitmq/***@localhost.log+
=WARNING REPORT==== 27-Aug-2013::09:04:23 ===
closing AMQP connection <0.278.0> (127.0.0.1:36093 -> 127.0.0.1:5672):
connection_closed_abruptly

This is the message I get whenever I try to start zenhub.

Here are some other logs I have been looking at:

+# rabbitmqctl status+
Status of node ***@localhost ...
[{pid,47180},
{running_applications,[{rabbit,"RabbitMQ","2.8.7"},
                        {os_mon,"CPO  CXC 138 46","2.2.7"},
                        {sasl,"SASL  CXC 138 11","2.1.10"},
                        {mnesia,"MNESIA  CXC 138 12","4.5"},
                        {stdlib,"ERTS  CXC 138 10","1.17.5"},
                        {kernel,"ERTS  CXC 138 10","2.14.5"}]},
{os,{unix,linux}},
{erlang_version,"Erlang R14B04 (erts-5.8.5) [source] [64-bit] [smp:24:24] [rq:24] [async-threads:30] [kernel-poll:true]\n"},
{memory,[{total,31029512},
          {processes,11371088},
          {processes_used,11366544},
          {system,19658424},
          {atom,1139905},
          {atom_used,1138863},
          {binary,1053232},
          {code,11368014},
          {ets,2881632}]},
{vm_memory_high_watermark,0.4},
{vm_memory_limit,60839985152},
{disk_free_limit,1000000000},
{disk_free,1029694262272},
{file_descriptors,[{total_limit,924},
                    {total_used,18},
                    {sockets_limit,829},
                    {sockets_used,8}]},
{processes,[{limit,1048576},{used,236}]},
{run_queue,0},
{uptime,1845}]
...done.

+/etc/rabbitmq/rabbitmq-env.conf+
RABBITMQ_NODENAME=***@localhost
RABBITMQ_MNESIA_BASE=/opt/zenoss/rabbitmq
RABBITMQ_MNESIA_DIR=/opt/zenoss/rabbitmq/mnesia/rabbit\@localhost

+#rabbitmqctl list_vhosts+
Listing vhosts ...
/zenoss
/
...done.

+# rabbitmqctl list_users+
Listing users ...
guest   [administrator]
zenoss  []
...done.

# rabbitmqctl list_queues -p /zenoss
Listing queues ...
celery  0
zenoss.queues.zep.modelchange   0
zenoss.queues.zep.signal        0
zenoss.queues.zep.migrated.summary      0
zenoss.queues.zep.rawevents     0
zenoss.queues.zep.heartbeats    0
m-prd-zen2.advertising.aol.com.celeryd.pidbox   0
zenoss.queues.zep.zenevents     0
zenoss.queues.zep.migrated.archive      0
...done.

+# rabbitmqctl list_user_permissions zenoss+
Listing permissions for user "zenoss" ...
/zenoss .*      .*      .*
...done.

+/opt/zenoss/log/zenhub.log+
2013-08-26 15:44:18,225 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,224 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused
2013-08-26 15:44:18,225 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,224 CRITICAL zen.Events: Unable to publish event to <Products.ZenMessaging.queuemessaging.publisher.EventPublisher object at 0x5380c50>: Could not connect to RabbitMQ: [111] Connection refused
2013-08-26 15:44:18,250 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,250 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused
2013-08-26 15:44:18,251 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,250 CRITICAL zen.Events: Unable to publish event to <Products.ZenMessaging.queuemessaging.publisher.EventPublisher object at 0x5380c50>: Could not connect to RabbitMQ: [111] Connection refused
2013-08-26 15:44:18,271 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,270 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused
2013-08-26 15:44:18,271 INFO zen.ZenHub: Worker (25825) reports 2013-08-26 15:44:18,271 CRITICAL zen.Events: Unable to publish event to <Products.ZenMessaging.queuemessaging.publisher.EventPublisher object at 0x5380c50>: Could not connect to RabbitMQ: [111] Connection refused

+/opt/zenoss/log/event.log+
2013-08-27T09:05:40 ERROR ZODB.Connection Couldn't load state for 0x01d101
Traceback (most recent call last):
  File "/opt/zenoss/lib/python/ZODB/Connection.py", line 860, in setstate
    self._setstate(obj)
  File "/opt/zenoss/lib/python/ZODB/Connection.py", line 914, in _setstate
    self._reader.setGhostState(obj, p)
  File "/opt/zenoss/lib/python/ZODB/serialize.py", line 613, in setGhostState
    obj.__setstate__(state)
TypeError: __setstate__() takes exactly 2 arguments (1 given)

+/opt/zenoss/log/zenactiond.log+
2013-08-27 09:04:49,695 INFO zen.zenoss.protocols.amqp: RabbitMQ connection was closed: [Errno 104] Connection reset by peer

+/opt/zenoss/etc/global.conf+
amqphost localhost
amqpport 5672
amqpvhost /zenoss
amqpuser zenoss
amqppassword zenoss
amqpusessl 0
amqpadminport 55672
amqpadminusessl 0

This did work initially and i don't believe anything has changed so i don't know why it is not working all of the sudden. Thanks for any help!

Ken
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74504#74504]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jcurry]

jcurry [http://community.zenoss.org/people/jcurry] created the discussion

"Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly"

To view the discussion, visit: http://community.zenoss.org/message/74551#74551

--------------------------------------------------------------
Could this be my usual bete noir of disks filling up?
Cheers,
Jane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74551#74551]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[kenhen93]

kenhen93 [http://community.zenoss.org/people/kenhen93] created the discussion

"Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly"

To view the discussion, visit: http://community.zenoss.org/message/74562#74562

--------------------------------------------------------------
Hi Jane,

Thanks for the reply. I wish it were that easy of an answer but disks look fine.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74562#74562]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[kenhen93]

kenhen93 [http://community.zenoss.org/people/kenhen93] created the discussion

"Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly"

To view the discussion, visit: http://community.zenoss.org/message/74564#74564

--------------------------------------------------------------
Can anyone explain how zenhub uses rabbitmq? How many connections does it initiate? Does it open and close them?


On a working version of zenoss I see in the rabbitmq log

=INFO REPORT==== 4-Sep-2013::16:00:01 ===
accepting AMQP connection <0.615.0> (127.0.0.1:50411 -> 127.0.0.1:5672)

=INFO REPORT==== 4-Sep-2013::16:00:01 ===
closing AMQP connection <0.615.0> (127.0.0.1:50411 -> 127.0.0.1:5672)

on my non-working version I see

=INFO REPORT==== 4-Sep-2013::15:49:48 ===
accepting AMQP connection <0.294.0> (127.0.0.1:43124 -> 127.0.0.1:5672)

=WARNING REPORT==== 4-Sep-2013::15:49:48 ===
closing AMQP connection <0.294.0> (127.0.0.1:43124 -> 127.0.0.1:5672):
connection_closed_abruptly
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74564#74564]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jcurry]

jcurry [http://community.zenoss.org/people/jcurry] created the discussion

"Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly"

To view the discussion, visit: http://community.zenoss.org/message/74568#74568

--------------------------------------------------------------
I have just seen the same thing on a 4.2.4 system.  All requests for rabbitmqctl info got a response like:

[***@zen42 rabbitmq]# rabbitmqctl -p /zenoss list_queues
Listing queues ...
Error: unable to connect to node ***@zen42: nodedown

DIAGNOSTICS
===========

nodes in question: [***@zen42]

hosts, their running nodes and ports:
- zen42: [{rabbitmqctl5921,51984}]

current node details:
- node name: ***@zen42
- home dir: /var/lib/rabbitmq
- cookie hash: WxE1AhHRrbuaQLykIGgbaQ==

Tried stopping rabbit and it said it wasn't running!


[***@zen42 rabbitmq]# /etc/init.d/rabbitmq-server stop
Stopping rabbitmq-server: RabbitMQ is not running
rabbitmq-server.
[***@zen42 rabbitmq]# ps -ef | grep rabbit
rabbitmq  2194     1  0 09:57 ?        00:00:00 /usr/lib64/erlang/erts-5.8.5/bin/epmd -daemon
root      6469  4889  1 10:22 pts/0    00:00:00 grep rabbit

Manually killed this rabbitmq process and restarted rabbit with:

service rabbitmq-server start
Starting rabbitmq-server: SUCCESS
rabbitmq-server.

ps -ef shows LOTS more rabbit processes.
rabbitmqctl commands work.
zenhub.log is now clean of messages to do with rabbit.
Everything seems OK.

Strange thing was that zenoss status was reporting all daemons as running (whereas I think you said zenhub and zopectl wouldn't start?).

It looks like rabbit had only half come up and forcing a complete restart got things going again.

I would be very interested to know if this also cures the problem in your environment.

Cheers,
Jane


BTW - I don't have your +/etc/rabbitmq/rabbitmq-env.conf .+ Is this something you created??+
+
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74568#74568]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[kenhen93]

kenhen93 [http://community.zenoss.org/people/kenhen93] created the discussion

"Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly"

To view the discussion, visit: http://community.zenoss.org/message/74570#74570

--------------------------------------------------------------
Hi Jane,

Yes I have noticed how when stopping rabbitmq '/usr/lib64/erlang/erts-5.8.5/bin/epmd -daemon' it still running. If I kill that process and start rabbitmq back up that process starts running again. I don't understand why Rabbitmq init script would just leave that process running. I tried this and it did not help with what is going on.

Yes I created rabbitmq-env.conf so that I could force the mnesia location and hostname. I wanted the mnesia location to be located on a SAN drive that is mounted on my server and force the hostname to be localhost instead of the actual hostname.
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74570#74570]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[jcurry]

jcurry [http://community.zenoss.org/people/jcurry] created the discussion

"Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly"

To view the discussion, visit: http://community.zenoss.org/message/74571#74571

--------------------------------------------------------------
"and force the hostname to be localhost instead of the actual hostname"....

Does this mean that the hostname has potentially changes as far as rabbit is concerned??

Have you seen this append - http://community.zenoss.org/message/71139#71139 http://community.zenoss.org/message/71139#71139 ?

It still sounds like rabbit is actively refusing to respond to zenhub so it has to be some sort of authentication mis-match.

Do you now have lots of processes if you do ps -ef / grep rabbit ???

Cheers,
Jane
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74571#74571]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[kenhen93]

kenhen93 [http://community.zenoss.org/people/kenhen93] created the discussion

"Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly"

To view the discussion, visit: http://community.zenoss.org/message/74572#74572

--------------------------------------------------------------
ok let me explain my setup better

I am actually running zenoss on a 2 node veritas cluster. The zenoss filesystem, MySQL DB, rabbitmq mensia is all on a SAN that can fail back and forth between both nodes. This is why I use the rabbitmq-env.conf to make the mnesia location on the SAN and also force the hostname to be localhost (as both nodes are localhost).

None of the Apps (Zenoss, MySQL, RabbitMQ) have any idea that this is a cluster.

Zenoss works great on Node 1 but when I failover to node 2 I get the problem I reported in the first post where zenhub cannot connect to rabbitmq. This worked originally on the original failover test but will not now. I cannot find a reason why. Everything is the same on both nodes. The erlang cookie shouldn't matter because rabbitmq in not in a cluster mode. The erland cookie is also on on the SAN so it is passed back and forth when rabbitmq is failover.

I didn't really want to get into this as I don't want this to get too confusing.

On a failover to node 2, rabbitmq starts

ps -ef | grep rabbitmq
rabbitmq 28171     1  0 Sep04 ?        00:00:00 /usr/lib64/erlang/erts-5.8.5/bin/epmd -daemon
root     41777     1  0 10:50 ?        00:00:00 runuser rabbitmq --session-command /usr/sbin/rabbitmq-server
rabbitmq 41781 41777  0 10:50 ?        00:00:00 /bin/sh /usr/sbin/rabbitmq-server
rabbitmq 41791 41781  3 10:50 ?        00:00:02 /usr/lib64/erlang/erts-5.8.5/bin/beam.smp -W w -K true -A30 -P 1048576 -- -root /usr/lib64/erlang -progname erl -- -home /opt/zenoss/rabbitmq -- -noshell -noinput -sname ***@localhost -boot /opt/zenoss/rabbitmq/***@localhost-plugins-expand/rabbit -kernel inet_default_connect_options [{nodelay,true}] -sasl errlog_type error -sasl sasl_error_logger false -rabbit error_logger {file,"/var/log/rabbitmq/***@localhost.log"} -rabbit sasl_error_logger {file,"/var/log/rabbitmq/***@localhost-sasl.log"} -os_mon start_cpu_sup false -os_mon start_disksup false -os_mon start_memsup false -mnesia dir "/opt/zenoss/rabbitmq/mnesia/***@localhost"
rabbitmq 41971 41791  0 10:50 ?        00:00:00 inet_gethost 4
rabbitmq 41972 41971  0 10:50 ?        00:00:00 inet_gethost 4
root     45458 40092  0 10:51 pts/0    00:00:00 grep rabbitmq

rabbitmqctl list_connections
Listing connections ...
zenoss  127.0.0.1       43314   running
zenoss  127.0.0.1       43288   running
zenoss  127.0.0.1       43301   running
zenoss  127.0.0.1       43284   running
zenoss  127.0.0.1       43277   running
zenoss  127.0.0.1       43313   running
zenoss  127.0.0.1       43297   running
...done.

service zenoss status
Daemon: zeneventserver program running; pid=43636
Daemon: zopectl not running
Daemon: zenrrdcached program running; pid=43835
Daemon: zenhub not running
Daemon: zenjobs program running; pid=45012
Daemon: zeneventd program running; pid=45063
Daemon: zenping program running; pid=45153
Daemon: zensyslog program running; pid=45205
Daemon: zenstatus program running; pid=45242
Daemon: zenactiond program running; pid=45280
Daemon: zentrap program running; pid=45315
Daemon: zenmodeler program running; pid=45368
Daemon: zenrender program running; pid=45398
Daemon: zenperfsnmp program running; pid=45438
Daemon: zencommand program running; pid=45487
Daemon: zenprocess program running; pid=45517
Daemon: zenmail program running; pid=45558
Daemon: zredis program running; pid=45561
Daemon: zenjmx program running; pid=45604
Daemon: zenwinperf program running; pid=45650
Daemon: zenwin program running; pid=45693
Daemon: zeneventlog program running; pid=45746

zenhub start
(/opt/zenoss/log/zenhub.log)
2013-09-04 15:48:27,231 INFO zen.ZenHub: Worker (28552) reports 2013-09-04 15:48:27,230 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused
2013-09-04 15:48:27,231 INFO zen.ZenHub: Worker (28552) reports 2013-09-04 15:48:27,231 CRITICAL zen.Events: Unable to publish event to <Products.ZenMessaging.queuemessaging.publisher.EventPublisher object at 0x5eb77d0>: Could not connect to RabbitMQ: [111] Connection refused
2013-09-04 15:48:27,246 INFO zen.ZenHub: Worker (28552) reports 2013-09-04 15:48:27,245 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused
2013-09-04 15:48:27,246 INFO zen.ZenHub: Worker (28552) reports 2013-09-04 15:48:27,245 CRITICAL zen.Events: Unable to publish event to <Products.ZenMessaging.queuemessaging.publisher.EventPublisher object at 0x5eb77d0>: Could not connect to RabbitMQ: [111] Connection refused
2013-09-04 15:48:27,267 INFO zen.ZenHub: Worker (28552) reports 2013-09-04 15:48:27,267 CRITICAL zen.zenoss.protocols.amqp: Could not use exchange $RawZenEvents: Could not connect to RabbitMQ: [111] Connection refused

/var/log/rabbitmq/***@localhost.log
=INFO REPORT==== 5-Sep-2013::10:53:08 ===
accepting AMQP connection <0.350.0> (127.0.0.1:43405 -> 127.0.0.1:5672)

=WARNING REPORT==== 5-Sep-2013::10:53:09 ===
closing AMQP connection <0.350.0> (127.0.0.1:43405 -> 127.0.0.1:5672):
connection_closed_abruptly

free
             total       used       free     shared    buffers     cached
Mem:     148535120    4199564  144335556          0     349992    1838108
-/+ buffers/cache:    2011464  146523656
Swap:      4194288          0    4194288

df
Filesystem           1K-blocks      Used Available Use% Mounted on
/dev/mapper/vg0-root   1032088    377348    602312  39% /
tmpfs                 74267560         0  74267560   0% /dev/shm
/dev/sde1               148742     33391    107671  24% /boot
/dev/mapper/vg0-home   1032088     34176    945484   4% /home
/dev/mapper/vg0-opt    8256952    845196   6992328  11% /opt
/dev/mapper/vg0-srv  558356472    202408 530014544   1% /srv
/dev/mapper/vg0-usr    4128448   1602120   2316616  41% /usr
/dev/mapper/vg0-var    4128448    855076   3063660  22% /var
none                   1048576    238772    809804  23% /tmp
tmpfs                        4         0         4   0% /dev/vx
/dev/vx/dsk/zenmasterDG/zenoss-vol01
                     1073708032   1369014 1005317891   1% /opt/zenoss
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74572#74572]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]

[kenhen93]

kenhen93 [http://community.zenoss.org/people/kenhen93] created the discussion

"Re: Zenhub zopectl won't start...rabbitmq connection_closed_abruptly"

To view the discussion, visit: http://community.zenoss.org/message/74609#74609

--------------------------------------------------------------
This was fixed by a reboot. I hate to say that
--------------------------------------------------------------

Reply to this message by replying to this email -or- go to the discussion on Zenoss Community
[http://community.zenoss.org/message/74609#74609]

Start a new discussion in zenoss-users at Zenoss Community
[http://community.zenoss.org/choose-container!input.jspa?contentType=1&containerType=14&container=2003]